© The Institution of Engineering and Technology
This study presents the adaptive traffic signal control algorithm in a distributed traffic network system. The proposed algorithm is based on a micro-simulation model and a reinforcement learning method, namely approximate dynamic programming (ADP). By considering traffic environment in discrete time, the microscopic traffic dynamic model is built. In particular, the authors explore a vehicle-following model using cellular automata theory. This vehicle-following model theoretically contributes to traffic network loading environment in an accessible way. To make the network coordinated, tunable state with weights of queue length and vehicles on lane is considered. The intersection can share information with each other in this state representation and make a joint action for intersection coordination. Moreover, the traffic signal control algorithm based on ADP method performs quite well in different performance measures witnessed by simulations. By comparing with other control methods, experimental results present that the proposed algorithm could be a potential candidate in an application of traffic network control system.
References
-
-
1)
-
8. Florian, M., Mahut, M., Tremblay, N.: ‘Application of a simulation-based dynamic traffic assignment model’, Eur. J. Oper. Res., 2008, 189, (3), pp. 1381–1392 (doi: 10.1016/j.ejor.2006.07.054).
-
2)
-
13. El-Tantawy, S., Abdulhai, B., Abdelgawad, H.: ‘Multiagent reinforcement learning for integrated network of adaptive traffic signal controllers (MARLIN-ATSC): methodology and large-scale application on downtown Toronto’, IEEE Trans. Intell. Transp. Syst., 2013, 14, (3), pp. 1140–1150 (doi: 10.1109/TITS.2013.2255286).
-
3)
-
19. El-Tantawy, S., Abdulhai, B.: ‘Multi-agent reinforcement learning for integrated network of adaptive traffic signal controllers (MARLIN-ATSC)’. Proc. of IEEE Conf. on Intelligent Transportation Systems, 2012, pp. 319–326.
-
4)
-
20. Kuyer, L., Whiteson, S., Bakker, B., et al: ‘Multiagent reinforcement learning for urban traffic control using coordination graphs’, in Walter, D., et al (Ed.): ‘Machine learning and knowledge discovery in databases’ (Springer, 2008), pp. 656–671.
-
5)
-
25. Li, T., Zhao, D.B., Yi, J.Q.: ‘Adaptive dynamic programming for multi-intersections traffic signal intelligent control’. Proc. of IEEE Conf. on Intelligent Transportation Systems, 2008, pp. 286–291.
-
6)
-
23. Cai, C., Wong, C.K., Heydecker, B.G.: ‘Adaptive traffic signal control using approximate dynamic programming’, Transp. Res. C, 2009, 17, (5), pp. 456–474 (doi: 10.1016/j.trc.2009.04.005).
-
7)
-
26. Arel, I., Liu, C., Urbanik, T., et al: ‘Reinforcement learning-based multi-agent system for network traffic signal control’, IET Intell. Transp. Syst., 2010, 4, (2), pp. 128–135 (doi: 10.1049/iet-its.2009.0070).
-
8)
-
16. France, J., Ghorbani, A.A.: ‘A multiagent system for optimizing urban traffic’. Int. IEEE/WIC Conf. on Intelligent Agent Technology, 2003, pp. 411–414.
-
9)
-
17. Yang, Z.S., Chen, X., Tang, Y.S., et al: ‘Intelligent cooperation control of urban traffic networks’. Proc. of Int. Conf. on Machine Learning and Cybernetics, 2005, vol. 3, pp. 1482–1486.
-
10)
-
1. Esser, J., Schreckenberg, M.: ‘Microscopic simulation of urban traffic based on cellular automata’, Int. J. Mod. Phys. C., 1997, 8, (5), pp. 1025–1036 (doi: 10.1142/S0129183197000904).
-
11)
-
3. Nagel, K., Schreckenberg, M.: ‘A cellular automaton model for freeway traffic’, J. Phys. I, 1992, 2, (12), pp. 2221–2229.
-
12)
-
28. Prashanth, L., Bhatnagar, S.: ‘Reinforcement learning with function approximation for traffic signal control’, IEEE Trans. Intell. Transp. Syst., 2011, 12, (2), pp. 412–421 (doi: 10.1109/TITS.2010.2091408).
-
13)
-
11. Barto, A.G.: ‘Reinforcement learning: an introduction’ (MIT Press, Cambridge, 1998).
-
14)
-
14. Bazzan, A.L.: ‘A distributed approach for coordination of traffic signal agents’, Auton. Agents Multi-Agent, 2005, 10, (1), pp. 131–164 (doi: 10.1007/s10458-004-6975-9).
-
15)
-
24. Le, T., Cai, C., Walsh, T.: ‘Adaptive signal–vehicle cooperative controlling system’. Proc. of IEEE Conf. on Intelligent Transportation Systems, 2011, pp. 236–241.
-
16)
-
15. Khamis, M.A., Gomaa, W.: ‘Adaptive multi-objective reinforcement learning with hybrid exploration for traffic signal control based on cooperative multi-agent framework’, Eng. Appl. Artif. Intell., 2014, 29, pp. 134–151 (doi: 10.1016/j.engappai.2014.01.007).
-
17)
-
22. Powell, W.B.: ‘Approximate dynamic programming: solving the curses of dimensionality’ (John Wiley & Sons, 2007, 2nd edn.).
-
18)
-
29. Lämmer, S., Helbing, D.: ‘Self-control of traffic lights and vehicle flows in urban road networks’, J. Stat. Mech., Theory Exp., 2008, 4, pp. 1–36.
-
19)
-
2. Maerivoet, S., De Moor, B.: ‘Cellular automata models of road traffic’, Phys. Rep., 2005, 419, (1), pp. 1–64 (doi: 10.1016/j.physrep.2005.08.005).
-
20)
-
31. Kok, J.R., Vlassis, N.: ‘Collaborative multiagent reinforcement learning by payoff propagation’, J. Mach. Learn. Res., 2006, 7, pp. 1789–1828.
-
21)
-
21. Medina, J.C., Benekohal, R.F.: ‘Traffic signal control using reinforcement learning and the max-plus algorithm as a coordinating strategy’. Proc. of IEEE Conf. on Intelligent Transportation Systems, 2012, pp. 596–601.
-
22)
-
10. Wiering, M., Vreeken, J., Van Veenen, J., et al: ‘Simulation and optimization of traffic in a city’. IEEE Intelligent Vehicles Symp., 2004, pp. 453–458.
-
23)
-
30. Yin, B., Dridi, M., El Moudni, A.: ‘Traffic control model and algorithm based on decomposition of MDP’. Proc. of Int. Conf. on Control, Decision and Information Technologies, 2014, pp. 225–230.
-
24)
-
21. Sumalee, A., Zhong, R.X., Pan, T.L., Szeto, W.Y.: ‘Stochastic cell transmission model (SCTM): a stochastic dynamic traffic model for traffic state surveillance and assignment’, Transp. Res. B, 2011, 45, (3), pp. 507–533 (doi: 10.1016/j.trb.2010.09.006).
-
25)
-
18. Lighthill, M.J., Whitham, G.B.: ‘On kinematic waves (II): a theory of traffic flow on long crowded roads’, Proc. R. Soc. Lond. A, Math. Phys. Sci., 1955, 229, (1178), pp. 317–345 (doi: 10.1098/rspa.1955.0089).
-
26)
-
7. Tonguz, O.K., Viriyasitavat, W., Bai, F.: ‘Modeling urban traffic: a cellular automata approach’, IEEE Commun. Mag., 2009, 47, (5), pp. 142–150 (doi: 10.1109/MCOM.2009.4939290).
-
27)
-
9. Wiering, M.: ‘Multi-agent reinforcement learning for traffic light control’. Int. Conf. on Machine Learning, 2000, pp. 1151–1158.
-
28)
-
12. Balaji, P., German, X., Srinivasan, D.: ‘Urban traffic signal control using reinforcement learning agents’, IET Intell. Transp. Syst., 2010, 4, (3), pp. 177–188 (doi: 10.1049/iet-its.2009.0096).
-
29)
-
32. Cools, S.B., Gershenson, C., D'Hooghe, B.: ‘Self-organizing traffic lights: a realistic simulation’, in Mikhail, P., et al (Ed.): ‘Advances in applied self-organizing systems’ (Springer, 2008), pp. 41–50.
-
30)
-
18. Bazzan, A.L., de Oliveira, D., da Silva, B.C.: ‘Learning in groups of traffic signals’, Eng. Appl. Artif. Intell., 2010, 23, (4), pp. 560–568 (doi: 10.1016/j.engappai.2009.11.009).
-
31)
-
6. Abouaissa, H., Fliess, M., Join, C.: ‘Fast parametric estimation for macroscopic traffic flow model’. 17th IFAC World Congress, Seoul, South Korea, 2008.
-
32)
-
27. Box, S., Waterson, B.: ‘An automated signalized junction controller that learns strategies by temporal difference reinforcement learning’, Eng. Appl. Artif. Intell., 2013, 26, (1), pp. 652–659 (doi: 10.1016/j.engappai.2012.02.013).
http://iet.metastore.ingenta.com/content/journals/10.1049/iet-its.2015.0108
Related content
content/journals/10.1049/iet-its.2015.0108
pub_keyword,iet_inspecKeyword,pub_concept
6
6