Reinforcement learning for autonomous underwater vehicles (AUVs): navigating challenges in dynamic and energy-constrained environments

Mohab M. Eweda, Karim ElNaggar

Abstract


Autonomous Underwater Vehicles (AUVs) are essential for underwater exploration, inspection, and environmental surveillance. Nevertheless, navigation, obstacle avoidance, and energy efficiency are greatly hindered by the ever-changing underwater environments. Reinforcement Learning (RL) has arisen as a revolutionary method for tackling these challenges. This paper examines significant progress in reinforcement learning algorithms, emphasizing their application in the training of autonomous underwater vehicles in both simulated and real-world environments. The review synthesizes findings from multiple studies, identifies gaps in existing research, and highlights the potential of algorithms such as Deep Deterministic Policy Gradient (DDPG) for continuous control tasks. This review offers an extensive examination of current methodologies, their constraints, and avenues for future investigation.

 

Received: 02 November 2024

Accepted: 22 December 2024

Published: 24 December 2024


Keywords


autonomous underwater vehicles; AUVs; reinforcement learning; RL; navigation; obstacle avoidance; energy efficiency;

Full Text:

PDF

References


T. P. Lillicrap et al., “Continuous control with deep reinforcement learning,” arXiv Cornell University, 2015, doi:10.48550/arxiv.1509.02971.

P. Bhopale, F. Kazi, and N. Singh, “Reinforcement Learning Based Obstacle Avoidance for Autonomous Underwater Vehicle,” Journal of Marine Science and Application, vol. 18, no. 2, pp. 228–238, Jun. 2019, doi: 10.1007/s11804-019-00089-3.

Y. Jiang, K. Zhang, M. Zhao, and H. Qin, “Adaptive meta-reinforcement learning for AUVs 3D guidance and control under unknown ocean currents,” Ocean Engineering, vol. 309, p. 118498, Oct. 2024, doi: 10.1016/j.oceaneng.2024.118498.

J. Wen, A. Wang, J. Zhu, F. Xia, Z. Peng, and W. Zhang, “Adaptive energy-efficient reinforcement learning for AUV 3D motion planning in complex underwater environments,” Ocean Engineering, vol. 312, p. 119111, Nov. 2024, doi: 10.1016/j.oceaneng.2024.119111.

R. K. Lea, R. Allen, and S. L. Merry, “A comparative study of control techniques for an underwater flight vehicle,” Int J Syst Sci, vol. 30, no. 9, pp. 947–964, Jan. 1999, doi: 10.1080/002077299291831.

R. S. Sutton and A. G. Barto, Reinforcement Learning, Second Edition An Introduction. 2017.

Y. Sun, X. Ran, G. Zhang, X. Wang, and H. Xu, “AUV path following controlled by modified Deep Deterministic Policy Gradient,” Ocean Engineering, vol. 210, p. 107360, Aug. 2020, doi: 10.1016/j.oceaneng.2020.107360.

M. Rahmati, M. Nadeem, V. Sadhu, and D. Pompili, “UW-MARL,” in Proceedings of the International Conference on Underwater Networks & Systems, New York, NY, USA: ACM, Oct. 2019, pp. 1–5. doi: 10.1145/3366486.3366533.

N. Anh Vien, N. Hoang Viet, S. Lee, and T. Chung, “Heuristic Search Based Exploration in Reinforcement Learning,” in Computational and Ambient Intelligence, Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 110–118. doi: 10.1007/978-3-540-73007-1_14.

B. Hadi, A. Khosravi, and P. Sarhadi, “Deep reinforcement learning for adaptive path planning and control of an autonomous underwater vehicle,” Applied Ocean Research, vol. 129, p. 103326, Dec. 2022, doi: 10.1016/j.apor.2022.103326.

S. Levine, C. Finn, T. Darrell, and P. Abbeel, “End-to-end training of deep visuomotor policies,” 2016.

A. Ramezani Dooraki and D.-J. Lee, “A Multi-Objective Reinforcement Learning Based Controller for Autonomous Navigation in Challenging Environments,” Machines, vol. 10, no. 7, p. 500, Jun. 2022, doi: 10.3390/machines10070500.

K. Sivamayil, E. Rajasekar, B. Aljafari, S. Nikolovski, S. Vairavasundaram, and I. Vairavasundaram, “A Systematic Study on Reinforcement Learning Based Applications,” Energies (Basel), vol. 16, no. 3, p. 1512, Feb. 2023, doi: 10.3390/en16031512.

Z. Wang et al., “Toward Communication Optimization for Future Underwater Networking: A Survey of Reinforcement Learning-Based Approaches,” IEEE Communications Surveys & Tutorials, pp. 1–1, 2024, doi: 10.1109/COMST.2024.3505850.




DOI: http://dx.doi.org/10.21622/RIMC.2024.01.2.1145

Refbacks

  • There are currently no refbacks.


Copyright (c) 2024 Mohab M. Eweda, Karim ElNaggar

Robotics : Integration, Manufacturing and Control

E-ISSN: 3009-7967

P-ISSN: 3009-6987

 

Published by:

Academy Publishing Center (APC)

Arab Academy for Science, Technology and Maritime Transport (AASTMT)

Alexandria, Egypt

rimc@aast.edu