Movatterモバイル変換

[0]ホーム
URL:

Skip to main content

Advertisement

Springer Nature Link
Log in

A Model-Based Reinforcement Learning Protocol for Routing in Vehicular Ad hoc Network

  • Published:
Wireless Personal Communications Aims and scope Submit manuscript

Abstract

Todays by equipping vehicles with wireless technologies, Vehicular Ad Hoc Network (VANET) has been emerged. This type of network can be utilized in many fields such as emergency, safety or entertainment. It is also considered as a main component of intelligent transportation system. However, due to the nodes velocity (vehicles velocity), varying density, obstacles and lack of fixed infrastructure, finding and maintaining a route between nodes are always challenging in VANET. Any routing protocol can be effective only if the nodes can learn and adapt themselves with such a dynamic environment. One way to achieve this adaptation is using machine learning techniques. In this paper we try to reach this goal by applying Multi-Agent Reinforcement Learning (MARL) that enables agents to solve routing optimization problems in a distributed way. Although model-free Reinforcement Learning (RL) schemes are introduced for this purpose, such techniques learn using a trial and error scheme in a real environment so they cannot reach an optimal policy in a short time. To deal with such a problem, we have proposed a mode-based RL based routing scheme. We have also developed a Fuzzy Logic (FL) system to evaluate the quality of links between neighbor nodes based on parameters such as velocity and connection quality. Outputs of this fuzzy system have been used to form the state transition model, needed in MARL. Results of evaluations have shown that our approach can improve some routing metrics like delivery ratio, end-to-end delay and traffic overhead.

This is a preview of subscription content,log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

References

  1. Tong, W., Hussain, A., Bo, W. X., & Maharjan, S. (2019). Artificial intelligence for vehicle-to-everything: A survey.IEEE Access,7, 10823–10843.https://doi.org/10.1109/ACCESS.2019.2891073

    Article  Google Scholar 

  2. Nazib, R. A., & Moh, S. (2021). Reinforcement learning-based routing protocols for vehicular Ad Hoc networks: A comparative survey.IEEE Access,9, 27552–27587.https://doi.org/10.1109/ACCESS.2021.3058388

    Article  Google Scholar 

  3. Srivastava, A., Prakash, A., & Tripathi, R. (2020). Location based routing protocols in VANET: Issues and existing solutions.Vehicular Communications,23, 100231.https://doi.org/10.1016/j.vehcom.2020.100231

    Article  Google Scholar 

  4. Awang, A., Husain, K., Kamel, N., & Aïssa, S. (2017). Routing in vehicular Ad-hoc networks: A survey on single- and cross-layer design techniques, and perspectives.IEEE Access,5, 9497–9517.https://doi.org/10.1109/ACCESS.2017.2692240

    Article  Google Scholar 

  5. Gao, H., Liu, C., Li, Y., & Yang, X. (2021). V2VR: Reliable hybrid-network-oriented V2V data transmission and routing considering RSUs and connectivity probability.IEEE Transactions on Intelligent Transportation Systems,22(6), 3533–3546.https://doi.org/10.1109/TITS.2020.2983835

    Article  Google Scholar 

  6. Liu, J., Wan, J., Wang, Q., Deng, P., Zhou, K., & Qiao, Y. (2016). A survey on position-based routing for vehicular ad hoc networks.Telecommunication Systems,62(1), 15–30.https://doi.org/10.1007/s11235-015-9979-7

    Article  Google Scholar 

  7. Kaur, R., & Rana, D. S. B. (2015). Overview on routing protocols in VANET.International Research Journal of Engineering and Technology, 02(03), 1333–1337.

  8. Katsaros, K., Dianati, M., Tafazolli, R., & Kernchen, R. CLWPR—A novel cross-layer optimized position based routing protocol for VANETs. In2011 IEEE Vehicular Networking Conference (VNC), 14–16 Nov. 2011 2011 (pp. 139–146). doi:https://doi.org/10.1109/VNC.2011.6117135.

  9. Chettibi, S., & Chikhi, S. (2010). A survey of reinforcement learning based routing protocols for Mobile Ad-Hoc networks. In (Vol. 162, pp. 1–13).

  10. Mohandas, G., Silas, S., & Sam, S. (2013). Survey on routing protocols on mobile adhoc networks. In: 2013 International mutli-conference on automation, computing, communication, control and compressed sensing (iMac4s), 22–23 March 2013 (pp. 514–517).https://doi.org/10.1109/iMac4s.2013.6526467.

  11. Wang, W., Xie, F., & Chatterjee, M. (2009). Small-scale and large-scale routing in vehicular Ad Hoc networks.IEEE Transactions on Vehicular Technology,58(9), 5200–5213.https://doi.org/10.1109/TVT.2009.2025652

    Article  Google Scholar 

  12. Perkins, C. E., & Royer, E. M. (1999). Ad-hoc on-demand distance vector routing. In: Proceedings WMCSA'99. Second IEEE workshop on mobile computing systems and applications, 25–26 Feb. 1999 (pp. 90–100).https://doi.org/10.1109/MCSA.1999.749281.

  13. Li, F., & Wang, Y. (2007). Routing in vehicular ad hoc networks: A survey.IEEE Vehicular Technology Magazine,2(2), 12–22.https://doi.org/10.1109/MVT.2007.912927

    Article  Google Scholar 

  14. Karagiannis, G., Altintas, O., Ekici, E., Heijenk, G., Jarupan, B., Lin, K., et al. (2011). Vehicular networking: A survey and tutorial on requirements, architectures, challenges, standards and solutions.IEEE Communications Surveys and Tutorials,13(4), 584–616.https://doi.org/10.1109/SURV.2011.061411.00019

    Article  Google Scholar 

  15. Blum, J. J., Eskandarian, A., & Hoffman, L. J. (2004). Challenges of intervehicle ad hoc networks.IEEE Transactions on Intelligent Transportation Systems,5(4), 347–351.https://doi.org/10.1109/TITS.2004.838218

    Article  Google Scholar 

  16. Wang, S. Y., Lin, C. C., Hwang, Y. W., Tao, K. C., & Chou, C. L. (2005).A practical routing protocol for vehicle-formed mobile ad hoc networks on the roads. In: Paper presented at the 2005 IEEE intelligent transportation systems conference, Oct. 2005.

  17. Karp, B., & Kung, H. T. (2000).GPSR: greedy perimeter stateless routing for wireless networks. In: Paper presented at the Proceedings of the 6th annual international conference on mobile computing and networking, Boston, Massachusetts, USA.

  18. Salkuyeh, M. A., & Abolhassani, B. (2016). An adaptive multipath geographic routing for video transmission in urban VANETs.IEEE Transactions on Intelligent Transportation Systems,17(10), 2822–2831.https://doi.org/10.1109/TITS.2016.2529178

    Article  Google Scholar 

  19. Huang, C., & Lin, S. (2014). Timer-based greedy forwarding algorithm in vehicular ad hoc networks.IET Intelligent Transport Systems,8(4), 333–344.https://doi.org/10.1049/iet-its.2013.0014

    Article  Google Scholar 

  20. Abuashour, A., & Kadoch, M. (2017). Performance improvement of cluster-based routing protocol in VANET.IEEE Access,5, 15354–15371.https://doi.org/10.1109/ACCESS.2017.2733380

    Article  Google Scholar 

  21. Bitam, S., Mellouk, A., & Zeadally, S. (2015). Bio-inspired routing algorithms survey for vehicular Ad Hoc networks.IEEE Communications Surveys and Tutorials,17(2), 843–867.https://doi.org/10.1109/COMST.2014.2371828

    Article  Google Scholar 

  22. Eiza, M. H., Owens, T., Ni, Q., & Shi, Q. (2015). Situation-aware QoS routing algorithm for vehicular Ad Hoc networks.IEEE Transactions on Vehicular Technology,64(12), 5520–5535.https://doi.org/10.1109/TVT.2015.2485305

    Article  Google Scholar 

  23. Li, G., Boukhatem, L., & Wu, J. (2017). Adaptive quality-of-service-based routing for vehicular Ad Hoc networks with ant colony optimization.IEEE Transactions on Vehicular Technology,66(4), 3249–3264.https://doi.org/10.1109/TVT.2016.2586382

    Article  Google Scholar 

  24. Sun, Y., Lin, Y., & Tang, Y. (2019) A reinforcement learning-based routing protocol. In: VANETs. In Q. Liang, J. Mu, M. Jia, W. Wang, X. Feng, & B. Zhang (Eds.),Communications, signal processing, and systems, Singapore, 2019 (pp. 2493–2500), Springer

  25. Wu, C., Ji, Y., Liu, F., Ohzahata, S., & Kato, T. (2015). Toward practical and intelligent routing in vehicular Ad Hoc networks.IEEE Transactions on Vehicular Technology,64(12), 5503–5519.https://doi.org/10.1109/TVT.2015.2481464

    Article  Google Scholar 

  26. Zhang, X., Zhang, X., & Gu, C. (2017). A micro-artificial bee colony based multicast routing in vehicular ad hoc networks.Ad Hoc Networks,58, 213–221.https://doi.org/10.1016/j.adhoc.2016.06.009

    Article  Google Scholar 

  27. Yang, X., Zhang, W., Lu, H., & Zhao, L. (2020). V2V routing in VANET based on heuristic Q-learning.International Journal of Computers communications and Control; Vol 15 No 5 (2020): International Journal of Computers Communications and Control (October).https://doi.org/10.15837/ijccc.2020.5.3928.

  28. Yao, L., Wang, J., Wang, X., Chen, A., & Wang, Y. (2018). V2X Routing in a VANET based on the hidden Markov model.IEEE Transactions on Intelligent Transportation Systems,19(3), 889–899.https://doi.org/10.1109/TITS.2017.2706756

    Article  Google Scholar 

  29. Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996). Reinforcement Learning: A Survey.Journal of Artificial Intelligence Research,4, 237–285.

    Article  Google Scholar 

  30. Watkins, C. (1989).Learning from delayed rewards. Student thesis, dissertation,King’s College, Cambridge, U.K.

  31. Zadeh, L. A. (1965). Fuzzy sets.Information and Control,8(3), 338–353.https://doi.org/10.1016/S0019-9958(65)90241-X

    Article MathSciNet MATH  Google Scholar 

  32. Cintula, P., Fermüller, Christian G. & Noguera, C. (2017). Fuzzy logic.Stanford encyclopedia of philosophy.

  33. Mamdani, E. H. A. S. A. (1975). An experiment in linguistic synthesis with a fuzzy logic controller.International Journal of Man-Machine Studies,7(1), 1–13.

    Article  Google Scholar 

  34. Siddique, M. (2009 ).Fuzzy decision making using max-min method and minimization of regret method (MMR) Blekinge Institute of Technology

  35. Bellman, R. (1957).Dynamic Programming. Princeton, NJ: (Vol. Press): Princeton Univ.

  36. OMNeT++ Community, OMNeT++ Network Simulator, . [Online]. Available:http://www.omnetpp.org/.

  37. SUMO Simulation of Urban Mobility. [Online]. Available:www.dlr.de/ts/sumo/en/.

  38. Lochert, C., Hartenstein, H., Tian, J., Fussler, H., Hermann, D., & Mauve, M. (2003). A routing strategy for vehicular ad hoc networks in city environments. InIEEE IV2003 Intelligent Vehicles Symposium. Proceedings (Cat. No.03TH8683), 9–11 June 2003(pp. 156–161). doi:https://doi.org/10.1109/IVS.2003.1212901.

  39. Jerbi, M., Senouci, S., Meraihi, R., & Ghamri-Doudane, Y. (2007). An improved vehicular Ad Hoc routing protocol for city environments. In2007 IEEE International Conference on Communications, 24–28 June 2007 (pp. 3972–3979).https://doi.org/10.1109/ICC.2007.654.

Download references

Funding

No funds, grants, or other support was received.

Author information

Authors and Affiliations

  1. Department of Electrical, Computer and IT Engineering, Qazvin Branch, Islamic Azad University, Qazvin, Iran

    Omid Jafarzadeh, Mehdi Dehghan & Hadi Sargolzaey

  2. Faculty of Computer Engineering, K. N. Toosi University of Technology, Tehran, Iran

    Mohammad Mehdi Esnaashari

Authors
  1. Omid Jafarzadeh

    You can also search for this author inPubMed Google Scholar

  2. Mehdi Dehghan

    You can also search for this author inPubMed Google Scholar

  3. Hadi Sargolzaey

    You can also search for this author inPubMed Google Scholar

  4. Mohammad Mehdi Esnaashari

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence toOmid Jafarzadeh.

Ethics declarations

Conflict of interest

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jafarzadeh, O., Dehghan, M., Sargolzaey, H.et al. A Model-Based Reinforcement Learning Protocol for Routing in Vehicular Ad hoc Network.Wireless Pers Commun123, 975–1001 (2022). https://doi.org/10.1007/s11277-021-09166-9

Download citation

Keywords

Access this article

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Advertisement

[8]ページ先頭
©2009-2025 Movatter.jp