Movatterモバイル変換

[0]ホーム
URL:

Skip to main content
Springer Nature Link
Log in

E-GCDT: advanced reinforcement learning with GAN-enhanced data for continuous excavation system

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

The automation of excavator operations entails the development and implementation of systems that allow excavators to execute tasks autonomously, thereby significantly reducing the need for human intervention. By integrating advanced sensors and artificial intelligence algorithms, these systems aim to increase operational efficiency, safety, and precision in construction and mining. However, previously developed methods have two weaknesses. First, existing automated excavator systems struggle with adapting to diverse and complex environmental conditions and with precision in control mechanisms. Second, operating an excavator involves multiple, repeated decisions that need to be modeled, planned, and executed in real time. However, there is a significant lack of comprehensive datasets that reflect real-world excavation operations to support this process. In this paper, we present an innovative system named E-GCDT. This system integrates the DoppelGANger module, which generates action time series by emulating human-mined trajectories through a generative adversarial mechanism and replays them in a simulation environment, ultimately expanding the dataset to 155 continuous mining trajectories. Furthermore, E-GCDT integrates terrain features into the decision-making process with the contrastive language-image pre-training model (CLIP), in which the decision transformer optimizes trajectory planning for efficient and accurate continuous excavation tasks. E-GCDT uniquely integrates advanced data augmentation and terrain awareness, developing an advanced Markov decision framework (DT) for continuous excavation tasks. The experimental results of a bulldozer verify that the efficiency of E-GCDT surpasses human efficiency. This system sets a new standard for continuous autonomous mining, and this study provides a new perspective on the application of reinforcement learning in industrial environments.

This is a preview of subscription content,log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Algorithm 1
Algorithm 2
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data Availability

Data are available on request to the authors. For more information, please contact us at 1025034345@sjtu.edu.cn

References

  1. Hemami A, Hassani F (2009) An overview of autonomous loading of bulk material. In: 26th International symposium on automation and robotics in construction, pp 405–411. International Association for Automation and Robotics in Construction (IAARC …

  2. Safety O, Administration H, et al (2015) Excavation: Hazard recognition in trenching and shoring. OSHA Technical Manual (Chapter 2). US Department of Labor Occupational Safety and Health Administration. Retrieved on 10, November 2017

  3. Lew J, Abraham D, Wirahadikusumah R, Irizarry J, Arboleda C (2002) Excavation and trenching safety: existing standards and challenges. Implement. Safety Health Construct, Sites, CIB

  4. Akinosho T, Oyedele LO, Bilal M, Ajayi A, Delgado M, AkinadéOO Ahmed AA (2020) Deep learning in the construction industry: A review of present status and future innovations. J Build Eng 32:101827.https://doi.org/10.1016/J.JOBE.2020.101827

  5. Zhang L, Zhao J, Long P, Wang L, Qian L, Lu F, Song X, Manocha D (2021) An autonomous excavator system for material loading tasks. Sci Robot 6(55):3164

    Article MATH  Google Scholar 

  6. Afanuh S, Gillen M, Lentz T (2011) Preventing worker deaths from trench cave-ins

  7. Bauerle T, Dugdale Z, Poplin G (2018) Mineworker fatigue: A review of what we know and future decisions. Min Engg 70(3):33

    Google Scholar 

  8. Dadhich S, Bodin U, Andersson U (2016) Key challenges in automation of earth-moving machines. Autom Constr 68:212–222

    Article MATH  Google Scholar 

  9. François-Lavet V, Henderson P, Islam R, Bellemare MG, Pineau J, et al (2018) An introduction to deep reinforcement learning. Foundations and Trends® in Machine Learning 11(3-4):219–354

  10. Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O (2017) Proximal policy optimization algorithms.arXiv:1707.06347

  11. Liu Y, Wang Y, Zhou Z (2024) Simulation of coherent excavator operations in earthmoving tasks based on reinforcement learning. Buildings.https://doi.org/10.3390/buildings14103270

  12. Casas N (2017) Deep deterministic policy gradient for urban traffic light control.arXiv:1703.09035

  13. Azulay O, Shapiro A (2021) Wheel loader scooping controller using deep reinforcement learning. IEEE Access 9:24145–24154.https://doi.org/10.1109/ACCESS.2021.3056625

    Article MATH  Google Scholar 

  14. Paduraru C, Mankowitz D, Dulac-Arnold G, Li J, Levine N, Gowal S, Hester T (2021) Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Mach Learn 110:2419–2468.https://doi.org/10.1007/s10994-021-05961-4

    Article MathSciNet MATH  Google Scholar 

  15. Hodel BJ (2018) Learning to operate an excavator via policy optimization. Procedia Comput Sci 140:376–382

    Article MATH  Google Scholar 

  16. Prudencio RF, Maximo M, Colombini E (2022) A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE transactions on neural networks and learning systems PP.https://doi.org/10.1109/TNNLS.2023.3250269

  17. Bansal MA, Sharma DR, Kathuria DM (2022) A systematic review on data scarcity problem in deep learning: solution and applications. ACM Comput Surv (CSUR) 54(10s):1–29

    Article MATH  Google Scholar 

  18. Babu A, Kirchner F (2021) Terrain adaption controller for a walking excavator robot using deep reinforcement learning. 2021 20th International Conference on Advanced Robotics (ICAR), pp 64–70.https://doi.org/10.1109/ICAR53236.2021.9659399

  19. Babu A, Danter L, Willenbrock P, Natarajan S, Kuehn D, Kirchner F (2022) Arter: a walking excavator robot for autonomous and remote operations. at - Automatisierungstechnik 70, 876–887.https://doi.org/10.1515/auto-2022-0056

  20. Chen L, Lu K, Rajeswaran A, Lee K, Grover A, Laskin M, Abbeel P, Srinivas A, Mordatch I (2021) Decision transformer: Reinforcement learning via sequence modeling. Adv Neural Inf Process Syst 34:15084–15097

    Google Scholar 

  21. Gui J, Sun Z, Wen Y, Tao D, Ye J (2021) A review on generative adversarial networks: Algorithms, theory, and applications. IEEE Trans Knowl Data Eng 35(4):3313–3332

    Article MATH  Google Scholar 

  22. Radford A, Kim J.W, Hallacy C, Ramesh A, Goh G, Agarwal S, Sastry G, Askell A, Mishkin P, Clark J, et al (2021) Learning transferable visual models from natural language supervision. In: International conference on machine learning, pp 8748–8763. PMLR

  23. Lin Z, Jain A, Wang C, Fanti G, Sekar V (2020) Using gans for sharing networked time series data: Challenges, initial promise, and open questions. In: Proceedings of the ACM internet measurement conference, pp 464–483

  24. Stentz A, Bares J, Singh S, Rowe P (1999) A robotic excavator for autonomous truck loading. Auto Robot 7:175–186

    Article  Google Scholar 

  25. Singh S, Cannon H (1998) Multi-resolution planning for earthmoving. In: Proceedings. 1998 IEEE international conference on robotics and automation (Cat. No. 98CH36146), vol 1, pp 121–126. IEEE

  26. Chang PH, Lee S-J (2002) A straight-line motion tracking control of hydraulic excavator system. Mechatronics 12(1):119–138

    Article MATH  Google Scholar 

  27. Schmidt D, Proetzsch M, Berns K (2010) Simulation and control of an autonomous bucket excavator for landscaping tasks. In: 2010 IEEE international conference on robotics and automation, pp 5108–5113. IEEE

  28. Quanchen Z, Tao Z, Runpu W, Jinxin M, Meicong L, Chen C, Changyu H (2018) From automation to intelligence: Survey of research on vulnerability discovery techniques. J Tsinghua Univ (Sci Technol) 58(12):1079–1094

    Google Scholar 

  29. Afshar RR, Zhang Y, Vanschoren J, Kaymak U (2022) Automated reinforcement learning: An overview.arXiv:2201.05000

  30. Dadhich S, Bodin U, Sandin F, Andersson U (2016) Machine learning approach to automatic bucket loading. In: 2016 24th Mediterranean conference on control and automation (MED), pp 1260–1265. IEEE

  31. Fukui R, Niho T, Nakao M, Uetake M (2017) Imitation-based control of automated ore excavator: improvement of autonomous excavation database quality using clustering and association analysis processes. Adv Robot 31(11):595–606

    Article  Google Scholar 

  32. Kurinov I, Orzechowski G, Hämäläinen P, Mikkola A (2020) Automated excavator based on reinforcement learning and multibody system dynamics. IEEE access 8:213998–214006

    Article MATH  Google Scholar 

  33. Osa T, Aizawa M (2022) Deep reinforcement learning with adversarial training for automated excavation using depth images. IEEE Access 10:4523–4535

    Article MATH  Google Scholar 

  34. Kalashnikov D, Irpan A, Pastor P, Ibarz J, Herzog A, Jang E, Quillen D, Holly E, Kalakrishnan M, Vanhoucke V, et al (2018) Scalable deep reinforcement learning for vision-based robotic manipulation. In: Conference on robot learning, pp 651–673. PMLR

  35. Xu J, Yoon H-S (2016) A review on mechanical and hydraulic system modeling of excavator manipulator system. J Constr Eng 1:9409370

    Google Scholar 

  36. SY750H | Large Excavator. (2024)https://www.sanyglobal.com/product/excavator/large_excavator/115/847/, Last accessed on 2024-3-30

  37. SKT90S (Automatic) Diesel Off-highway Mining Truck (2024)https://www.sanyglobal.com/product/truck/off-highway_mining_truck/75/445/, Last accessed on 2024-3-30

  38. Bernardes E, Viollet S (2022) Quaternion to euler angles conversion: A direct, general and computationally efficient method. Plos one 17(11):0276302

    Article MATH  Google Scholar 

  39. Van Erven T, Harremos P (2014) Rényi divergence and kullback-leibler divergence. IEEE Trans Inf Theory 60(7):3797–3820

    Article MATH  Google Scholar 

  40. Yin H, Vahdat A, Alvarez J.M, Mallya A, Kautz J, Molchanov P (2022) A-vit: Adaptive tokens for efficient vision transformer. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 10809–10818

  41. Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I et al (2019) Language models are unsupervised multitask learners. OpenAI blog 1(8):9

    Google Scholar 

  42. Kumar A, Zhou A, Tucker G, Levine S (2020) Conservative q-learning for offline reinforcement learning. Adv Neural Inf Process Syst 33:1179–1191

  43. Torabi F, Warnell G, Stone P (2018) Behavioral cloning from observation.arXiv:1805.01954

  44. Hout MC, Papesh MH, Goldinger SD (2013) Multidimensional scaling. Wiley Interdiscip Rev: Cogn Sci 4(1):93–103

    Article MATH  Google Scholar 

  45. Anh DT, Thanh LH (2015) An efficient implementation of k-means clustering for time series data with dtw distance. Int J Bus Intell Data Mining 10(3):213–232

Download references

Acknowledgements

This research is supported by the National Key R&D Program of China (2022YFB3402001), and the National Natural Science Foundation of China (Grant Nos. 52475270, 52375254). We would also like to extend our sincere thanks to Sany Company for their support and contribution to this research.

Author information

Authors and Affiliations

  1. School of Mechanical Engineering, Shanghai Jiao Tong University, 201100, Shanghai, China

    Qianyou Zhao, Duidi Wu, Yihao Lei, Lingyu Wang, Jin Qi & Jie Hu

  2. School of Design, Shanghai Jiao Tong University, 201100, Shanghai, China

    Jie Hu

  3. Sany heavy machinery Co.ltd, Kunshan, 215300, Jiangsu, China

    Le Gao

Authors
  1. Qianyou Zhao

    You can also search for this author inPubMed Google Scholar

  2. Le Gao

    You can also search for this author inPubMed Google Scholar

  3. Duidi Wu

    You can also search for this author inPubMed Google Scholar

  4. Yihao Lei

    You can also search for this author inPubMed Google Scholar

  5. Lingyu Wang

    You can also search for this author inPubMed Google Scholar

  6. Jin Qi

    You can also search for this author inPubMed Google Scholar

  7. Jie Hu

    You can also search for this author inPubMed Google Scholar

Contributions

Conceptualization: Qianyou Zhao; Methodology: Qianyou Zhao, Duidi Wu and Yihao Lei; Formal analysis and investigation: Qianyou Zhao; Writing - original draft preparation: Qianyou Zhao; Writing - review and editing: Lingyu Wang, Jin Qi and Jie Hu; Funding acquisition: Le Gao and Jie Hu; Resources: Le Gao; Supervision: Jie Hu.

Corresponding authors

Correspondence toJin Qi orJie Hu.

Ethics declarations

Competing Interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Ethical Approval

This study did not involve any human participants, human data, or animals, and therefore did not require any ethical approval or informed consent.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Access this article

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Advertisement

[8]ページ先頭
©2009-2025 Movatter.jp