From Simulation to Reality: Deep Reinforcement Learning for Autonomous Underwater Vehicle Docking

Vibhav Bharti; Sümer Tunçay; Ignacio Carlucho; Maria Koskinopoulou; Yvan R. Petillot

doi:10.1109/oceans58557.2025.11104520

From Simulation to Reality: Deep Reinforcement Learning for Autonomous Underwater Vehicle Docking

Vibhav Bharti, Sümer Tunçay, Ignacio Carlucho, Maria Koskinopoulou, Yvan R. Petillot

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Deep Reinforcement Learning (DRL) offers significant potential for real-world robotics applications, yet sim-to-real transfer remains a major challenge. In this work, we propose the use of the Twin-Delayed Deep Deterministic Policy Gradient (TD3) algorithm to efficiently train a docking policy. We tailor our reward function to perform smooth docking. Additionally, we employ a 'training wheels' approach that initially fills the replay buffer with PID controller demonstrations to accelerate learning. The resulting model is applied to visual servoing and docking tasks, utilizing AprilTag markers for localization. Real experiments validate our approach.

Original language	English
Title of host publication	OCEANS 2025 Brest
Publisher	IEEE
ISBN (Electronic)	9798331537470
ISBN (Print)	9798331537487
DOIs	https://doi.org/10.1109/oceans58557.2025.11104520
Publication status	Published - 11 Aug 2025
Event	OCEANS 2025 Brest Conference - Le Quartz, Brest, France Duration: 16 Jun 2025 → 19 Jun 2025 https://brest25.oceansconference.org/

Conference

Conference	OCEANS 2025 Brest Conference
Country/Territory	France
City	Brest
Period	16/06/25 → 19/06/25
Internet address	https://brest25.oceansconference.org/

Keywords

Training
Location awareness
Robust control
Autonomous underwater vehicles
PI control
Wheels
Deep reinforcement learning
Visual servoing
Robustness
Vehicle dynamics

ASJC Scopus subject areas

Oceanography
Ocean Engineering

Access to Document

10.1109/oceans58557.2025.11104520

Cite this

@inproceedings{4f01977a3c714240a71cdc57515f39a6,

title = "From Simulation to Reality: Deep Reinforcement Learning for Autonomous Underwater Vehicle Docking",

abstract = "Deep Reinforcement Learning (DRL) offers significant potential for real-world robotics applications, yet sim-to-real transfer remains a major challenge. In this work, we propose the use of the Twin-Delayed Deep Deterministic Policy Gradient (TD3) algorithm to efficiently train a docking policy. We tailor our reward function to perform smooth docking. Additionally, we employ a 'training wheels' approach that initially fills the replay buffer with PID controller demonstrations to accelerate learning. The resulting model is applied to visual servoing and docking tasks, utilizing AprilTag markers for localization. Real experiments validate our approach.",

keywords = "Training, Location awareness, Robust control, Autonomous underwater vehicles, PI control, Wheels, Deep reinforcement learning, Visual servoing, Robustness, Vehicle dynamics",

author = "Vibhav Bharti and S{\"u}mer Tun{\c c}ay and Ignacio Carlucho and Maria Koskinopoulou and Petillot, \{Yvan R.\}",

year = "2025",

month = aug,

day = "11",

doi = "10.1109/oceans58557.2025.11104520",

language = "English",

isbn = "9798331537487",

booktitle = "OCEANS 2025 Brest",

publisher = "IEEE",

address = "United States",

note = "OCEANS 2025 Brest Conference ; Conference date: 16-06-2025 Through 19-06-2025",

url = "https://brest25.oceansconference.org/",

}

TY - GEN

T1 - From Simulation to Reality: Deep Reinforcement Learning for Autonomous Underwater Vehicle Docking

AU - Bharti, Vibhav

AU - Tunçay, Sümer

AU - Carlucho, Ignacio

AU - Koskinopoulou, Maria

AU - Petillot, Yvan R.

PY - 2025/8/11

Y1 - 2025/8/11

N2 - Deep Reinforcement Learning (DRL) offers significant potential for real-world robotics applications, yet sim-to-real transfer remains a major challenge. In this work, we propose the use of the Twin-Delayed Deep Deterministic Policy Gradient (TD3) algorithm to efficiently train a docking policy. We tailor our reward function to perform smooth docking. Additionally, we employ a 'training wheels' approach that initially fills the replay buffer with PID controller demonstrations to accelerate learning. The resulting model is applied to visual servoing and docking tasks, utilizing AprilTag markers for localization. Real experiments validate our approach.

AB - Deep Reinforcement Learning (DRL) offers significant potential for real-world robotics applications, yet sim-to-real transfer remains a major challenge. In this work, we propose the use of the Twin-Delayed Deep Deterministic Policy Gradient (TD3) algorithm to efficiently train a docking policy. We tailor our reward function to perform smooth docking. Additionally, we employ a 'training wheels' approach that initially fills the replay buffer with PID controller demonstrations to accelerate learning. The resulting model is applied to visual servoing and docking tasks, utilizing AprilTag markers for localization. Real experiments validate our approach.

KW - Training

KW - Location awareness

KW - Robust control

KW - Autonomous underwater vehicles

KW - PI control

KW - Wheels

KW - Deep reinforcement learning

KW - Visual servoing

KW - Robustness

KW - Vehicle dynamics

UR - https://www.scopus.com/pages/publications/105015035963

U2 - 10.1109/oceans58557.2025.11104520

DO - 10.1109/oceans58557.2025.11104520

M3 - Conference contribution

SN - 9798331537487

BT - OCEANS 2025 Brest

PB - IEEE

T2 - OCEANS 2025 Brest Conference

Y2 - 16 June 2025 through 19 June 2025

ER -

From Simulation to Reality: Deep Reinforcement Learning for Autonomous Underwater Vehicle Docking

Abstract

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this