HERLMES: Heterogeneous Ensemble with Reinforcement Learning Mechanisms for Evolutionary Search

This paper proposes HERLMES (Heterogeneous Ensemble with Reinforcement Learning Mechanisms for Evolutionary Search), an ensemble optimization framework that couples evolutionary search with a dimension-aware Q-learning agent using experience replay to adaptively select among five complementary operators. The method introduces a compact 7-dimensional state representation and a novel Success-History Fusion Averaging (SHFA) operator, which extends the Success-History Intelligent Optimizer (SHIO) via time-decaying perturbations and triple-elite averaging. The framework integrates Particle Swarm Optimization (PSO), Covariance Matrix Adaptation Evolution Strategy (CMA-ES), Interior Point Method (IPM), Sequential Quadratic Programming (SQP), and SHFA. Experiments on CEC 2017 (30 functions) at D = 10, 30, 50, 100 (51 runs per function) show that HERLMES ranks first by Friedman test across all tested dimensionalities. Ablation studies comprising eight variants confirm the necessity of the ensemble structure, local search components, and dimension-aware reinforcement learning (RL). The source code of HERLMES is publicly available for download at https://github.com/microhard1999/CODES.