Your browser does not support JavaScript!
http://iet.metastore.ingenta.com
1887

access icon free Value-based deep reinforcement learning for adaptive isolated intersection signal control

© The Institution of Engineering and Technology
Received 26/01/2018, Accepted 08/06/2018, Revised 26/04/2018, Published 26/07/2018

Under efficiency improvement of road networks by utilizing advanced traffic signal control methods, intelligent transportation systems intend to characterize a smart city. Recently, due to significant progress in artificial intelligence, machine learning-based framework of adaptive traffic signal control has been highly concentrated. In particular, deep Q-learning neural network is a model-free technique and can be applied to optimal action selection problems. However, setting variable green time is a key mechanism to reflect traffic fluctuations such that time steps need not be fixed intervals in reinforcement learning framework. In this study, the authors proposed a dynamic discount factor embedded in the iterative Bellman equation to prevent from a biased estimation of action-value function due to the effects of inconstant time step interval. Moreover, action is added to the input layer of the neural network in the training process, and the output layer is the estimated action-value for the denoted action. Then, the trained neural network can be used to generate action that leads to an optimal estimated value within a finite set as the agents' policy. The preliminary results show that the trained agent outperforms a fixed timing plan in all testing cases with reducing system total delay by 20%..

Inspec keywords: iterative methods; learning (artificial intelligence); neural nets; road traffic; dynamic programming; traffic engineering computing

Other keywords: road network efficiency improvement; advanced traffic signal control methods; intelligent transportation systems; traffic arrival rates; variable green time; VISSIM software; modern city; biased action-value function estimation; dynamic discount factor; traffic arrival patterns; artificial intelligence; iterative Bellman equation; traffic fluctuations; optimal discrete-time action selection problems; smart city; adaptive isolated intersection signal control; value-based deep reinforcement learning; model-free technique; machine learning-based framework; deep Q-learning neural network

Subjects: Neural computing techniques; Interpolation and function approximation (numerical analysis); Optimisation techniques; Knowledge engineering techniques; Traffic engineering computing

References

    1. 1)
      • 13. Wiering, M.: ‘Multi-agent reinforcement learning for traffic light control’. ICML, Stanford, CA, USA, 2000, pp. 1151–1158.
    2. 2)
      • 10. Krizhevsky, A., Sutskever, I., Hinton, G.: ‘Imagenet classification with deep convolutional neural networks’, Adv. Neural Inf. Process. Syst., 2012, 25, pp. 1106–1114.
    3. 3)
      • 14. Bingham, E.: ‘Reinforcement learning in neurofuzzy traffic signal control’, Eur. J. Oper. Res., 2001, 131, (2), pp. 232–241.
    4. 4)
      • 17. van der Pol, E.: ‘Deep reinforcement learning for coordination in traffic light control’, Master's thesis, University of Amsterdam, August 2016.
    5. 5)
      • 12. Abdulhai, B., Pringle, R., Karakoulas, G.J.: ‘Reinforcement learning for true adaptive traffic signal control’, J. Transp. Eng., 2003, 129, (3), pp. 278–285.
    6. 6)
      • 18. Li, L., Lv, Y., Wang, F.-Y.: ‘Traffic signal timing via deep reinforcement learning’, IEEE/CAA J. Autom. Sinica, 2016, 3, (3), pp. 247–254.
    7. 7)
      • 16. Genders, W., Razavi, S.: ‘Using a deep reinforcement learning agent for traffic signal control’, Available at: https://arxiv.org/abs/1611.01142, November 2016.
    8. 8)
      • 7. Watkins, C., Dayan, P.: Q-learning, Machine Learning, 1992, 8, (3-4), pp. 279–292.
    9. 9)
      • 5. Abdulhai, B., Kattan, L.: ‘Reinforcement learning: introduction to theory and potential for transport applications’, Can. J. Civ. Eng., 2003, 30, pp. 981–991.
    10. 10)
      • 15. Mnih, V., Kavukcuoglu, K., Silver, D., et al: ‘Playing atari with deep reinforcement learning’, arXiv preprint arXiv:1312.5602, 2013.
    11. 11)
      • 11. Hinton, G.E., Salakhutdinov, R.R.: ‘Reducing the dimensionality of data with neural networks’, Science, 2006, 313, pp. 504–507.
    12. 12)
      • 2. Kaelbling, L.P., Littman, M.L., Moore, A.W.: ‘Reinforcement learning: a survey’, J. Artif. Intell. Res., 1996, 4, pp. 237–285.
    13. 13)
      • 19. Lin, L.-J.: ‘Self-improving reactive agents based on reinforcement learning, planning and teaching’, Mach. Learn., 1992, 8, (3–4), pp. 293–321.
    14. 14)
      • 22. Webster, F.V.: ‘Traffic Signal Settings’. Road Research Technical Paper No. 39. London: Great Britain Road Research Laboratory, 1958.
    15. 15)
      • 21. Glorot, X., Bengio, Y.: ‘Understanding the difficulty of training deep feedforward neural networks’. Proc. of the Int. Conf. on Artificial Intelligence and Statistics. Society for Artificial Intelligence and Statistics, 2010, vol. 9, pp. 249–256.
    16. 16)
      • 6. Watkins, C.J.C.H.: ‘Learning from delayed rewards’, Ph.D. thesis, Cambridge University, 1989.
    17. 17)
      • 9. Bengio, Y.: ‘Learning deep architectures for AI’, Found. Trends Mach. Learn., 2009, 2, pp. 1–127.
    18. 18)
      • 20. Kingma, D.P., Ba, J.: ‘Adam: a method for stochastic optimization’, arXiv:1412.6980 [cs.LG], December 2014.
    19. 19)
      • 4. Abdulhai, B., Kattan, L.: ‘Reinforcement learning: Introduction to theory and potential for transport applications1’, Can. J. Civ. Eng., 2003, 30, pp. 981–991.
    20. 20)
      • 8. Mnih, V., et al: ‘Human-level control through deep reinforcement learning’, Nature, 2015, 518, (7540), pp. 529–533.
    21. 21)
      • 3. Bertsekas, D.P.: ‘Dynamic programming and optimal control’ (Athena Scientific, Nashua, NH, USA, 2007, 3rd edn.), vol. 2.
    22. 22)
      • 1. Sutton, R.S., Barto, A.G.: ‘Reinforcement learning: an introduction’ (The MIT Press, Cambridge, Massachusetts, USA/London, England, 1998).
http://iet.metastore.ingenta.com/content/journals/10.1049/iet-its.2018.5170
Loading

Related content

content/journals/10.1049/iet-its.2018.5170
pub_keyword,iet_inspecKeyword,pub_concept
6
6
Loading
Print this page
This is a required field
Please enter a valid email address