Stochastic optimal well control in subsurface reservoirs using reinforcement learning

Atish Dixit, Ahmed H. Elsheikh

Research output: Contribution to journalArticlepeer-review

8 Citations (Scopus)
27 Downloads (Pure)


We present a case study of model-free reinforcement learning (RL) framework to solve stochastic optimal control for a predefined parameter uncertainty distribution and partially observable system. We focus on robust optimal well control problem which is a subject of intensive research activities in the field of subsurface reservoir management. For this problem, the system is partially observed since the data is only available at well locations. Furthermore, the model parameters are highly uncertain due to sparsity of available field data. In principle, RL algorithms are capable of learning optimal action policies – a map from states to actions – to maximize a numerical reward signal. In deep RL, this mapping from state to action is parameterized using a deep neural network. In the RL formulation of the robust optimal well control problem, the states are represented by saturation and pressure values at well locations while the actions represent the valve openings controlling the flow through wells. The numerical reward refers to the total sweep efficiency and the uncertain model parameter is the subsurface permeability field. The model parameter uncertainties are handled by introducing a domain randomization scheme that exploits cluster analysis on its uncertainty distribution. We present numerical results using two state-of-the-art RL algorithms, proximal policy optimization (PPO) and advantage actor–critic (A2C), on two subsurface flow test cases representing two distinct uncertainty distributions of permeability field. The results were benchmarked against optimization results obtained using differential evolution algorithm. Furthermore, we demonstrate the robustness of the proposed use of RL by evaluating the learned control policy on unseen samples drawn from the parameter uncertainty distribution that were not used during the training process.
Original languageEnglish
Article number105106
JournalEngineering Applications of Artificial Intelligence
Early online date4 Jul 2022
Publication statusPublished - Sept 2022


  • Artificial intelligence in reservoir management
  • Optimal control for partially observable system
  • Reinforcement learning
  • Stochastic optimal control
  • Subsurface flow control

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Artificial Intelligence
  • Electrical and Electronic Engineering


Dive into the research topics of 'Stochastic optimal well control in subsurface reservoirs using reinforcement learning'. Together they form a unique fingerprint.

Cite this