CLC number: TU621
On-line Access: 2017-11-06
Received: 2016-09-30
Revision Accepted: 2017-03-28
Crosschecked: 2017-10-11
Cited: 0
Clicked: 5319
Qian Zhu, Qing-feng Wang. Real-time energy management controller design for a hybrid excavator using reinforcement learning[J]. Journal of Zhejiang University Science A, 2017, 18(11): 855-870.
@article{title="Real-time energy management controller design for a hybrid excavator using reinforcement learning",
author="Qian Zhu, Qing-feng Wang",
journal="Journal of Zhejiang University Science A",
volume="18",
number="11",
pages="855-870",
year="2017",
publisher="Zhejiang University Press & Springer",
doi="10.1631/jzus.A1600650"
}
%0 Journal Article
%T Real-time energy management controller design for a hybrid excavator using reinforcement learning
%A Qian Zhu
%A Qing-feng Wang
%J Journal of Zhejiang University SCIENCE A
%V 18
%N 11
%P 855-870
%@ 1673-565X
%D 2017
%I Zhejiang University Press & Springer
%DOI 10.1631/jzus.A1600650
TY - JOUR
T1 - Real-time energy management controller design for a hybrid excavator using reinforcement learning
A1 - Qian Zhu
A1 - Qing-feng Wang
J0 - Journal of Zhejiang University Science A
VL - 18
IS - 11
SP - 855
EP - 870
%@ 1673-565X
Y1 - 2017
PB - Zhejiang University Press & Springer
ER -
DOI - 10.1631/jzus.A1600650
Abstract: Real-time energy management of a hybrid excavator is addressed using reinforcement learning (RL). Due to the computational complexity and need for a priori knowledge of the load cycles, a traditional optimal control method, like dynamic programming (DP), is not feasible for real-time control. Real-time controllers derived from traditional optimal control methods compute the solutions either in a cycle-dependent manner or far away from the optimal. An RL-based energy management controller is proposed to solve this problem. The simulation and experimental results demonstrate that the RL controller has a better performance than the widely used thermostat and equivalent consumption minimization strategy (ECMS) controllers. It also shows that the RL controller is cycle-independent. pontryagin’;s minimum principle (PMP) is used to obtain the analytical solution of the energy management problem, and this can help to reduce the iteration time in the design process.
This paper presents an interesting topic on the energy management problem of a hybrid excavator. Four different control strategy have been implemented on a simulator over the standard digging cycle and then validated through a dedicated experimental activity on the hybrid prototype.
[1]Borhan, H.A., Vahidi, A., Phillips, A.M., et al., 2009. Predictive energy management of a power-split hybrid electric vehicle. American Control Conference, p.3970-3976.
[2]Brahma, A., Guezennec, Y., Rizzoni, G., 2000. Optimal energy management in series hybrid electric vehicles. American Control Conference, p.60-64.
[3]Busoniu, L., Babuska, R., Schutter, D., et al., 2010. Reinforcement Learning and Dynamic Programming Using Function Approximators. CRC Press, Boca Raton, USA, p.101-113.
[4]Chan, C.C., 2007. The state of the art of electric, hybrid, and fuel cell vehicles. Proceedings of the IEEE, 95(4):704-718.
[5]Ehsani, M., Gao, Y., Miller, J.M., 2007. Hybrid electric vehicles: architecture and motor drives. Proceedings of the IEEE, 95(4):719-728.
[6]Geering, H.P., 2007. Optimal Control with Engineering Applications. Springer, New York, USA, p.3-5.
[7]Gosavi, A., 2003. Simulation-based Optimization. Springer, New York, USA, p.197-211.
[8]Jalil, N., Kheir, N.A., Salman, M., 1997. A rule-based energy management strategy for a series hybrid vehicle. American Control Conference, p.689-693.
[9]Johri, R., Filipi, Z., 2009. Low-cost pathway to ultra efficient city car: series hydraulic hybrid system with optimized supervisory control. SAE International Journal of Engines, 2(2):505-520.
[10]Kim, H., Choi, J., Yi, K., 2012. Development of supervisory control strategy for optimized fuel consumption of the compound hybrid excavator. Proceedings of the Institution of Mechanical Engineers, Part D: Journal of Automobile Engineering, 226(12):1652-1666.
[11]Kim, H., Yoo, S., Cho, S., et al., 2016. Hybrid control algorithm for fuel consumption of a compound hybrid excavator. Automation in Construction, 68:1-10.
[12]Kim, N., Rousseau, A., 2012. Sufficient conditions of optimal control based on Pontryagin’s minimum principle for use in hybrid electric vehicles. Proceedings of the Institution of Mechanical Engineers, Part D: Journal of Automobile Engineering, 226(9):1160-1170.
[13]Kim, N., Cha, S., Peng, H., 2011. Optimal control of hybrid electric vehicles based on Pontryagin’s minimum principle. IEEE Transactions on Control Systems Technology, 19(5):1279-1287.
[14]Lin, C.C., Peng, H., Grizzle, J.W., et al., 2003. Power management strategy for a parallel hybrid electric truck. IEEE Transactions on Control Systems Technology, 11(6):839-849.
[15]Lin, T., Wang, Q., Hu, B., et al., 2010. Development of hybrid powered hydraulic construction machinery. Automation in Construction, 19(1):11-19.
[16]Lin, X., Pan, S.X., Wang, D.Y., 2008. Dynamic simulation and optimal control strategy for a parallel hybrid hydraulic excavator. Journal of Zhejiang University-SCIENCE A, 9(5):624-632.
[17]Musardo, C., Rizzoni, G., Guezennec, Y., et al., 2005. A-ECMS: an adaptive algorithm for hybrid electric vehicle energy management. European Journal of Control, 11(4-5):509-524.
[18]Salman, M., Schouten, N.J., Kheir, N.A., 2000. Control strategies for parallel hybrid vehicles. American Control Conference, p.524-528.
[19]Salmasi, F.R., 2007. Control strategies for hybrid electric vehicles: evolution, classification, comparison, and future trends. IEEE Transactions on Vehicular Technology, 56(5):2393-2404.
[20]Schouten, N.J., Salman, M.A., Kheir, N.A., 2002. Fuzzy logic control for parallel hybrid vehicles. IEEE Transactions on Control Systems Technology, 10(3):460-468.
[21]Sciarretta, A., Guzzella, L., 2007. Control of hybrid electric vehicles. IEEE Control Systems, 27(2):60-70.
[22]Sciarretta, A., Back, M., Guzzella, L., 2004. Optimal control of parallel hybrid electric vehicles. IEEE Transactions on Control Systems Technology, 12(3):352-363.
[23]Serrao, L., Onori, S., Rizzoni, G., 2009. ECMS as a realization of Pontryagin’s minimum principle for HEV control. American Control Conference, p.3964-3969.
[24]Serrao, L., Onori, S., Rizzoni, G., 2011. A comparative analysis of energy management strategies for hybrid electric vehicles. Journal of Dynamic Systems, Measurement, and Control, 133(3):031012.
[25]Watkins, C., Dayan, P., 1992. Q-learning. Machine Learning, 8(3-4):279-292.
[26]Xiao, Q., Wang, Q., Zhang, Y., 2008. Control strategies of power system in hybrid hydraulic excavator. Automation in Construction, 17(4):361-367.
[27]Zhang, S., Xiong, R., 2015. Adaptive energy management of a plug-in hybrid electric vehicle based on driving pattern recognition and dynamic programming. Applied Energy, 155:68-78.
[28]Zhu, Q., Wang, Q., Chen, Q., 2016. Component sizing for compound hybrid excavators. Proceedings of the Institution of Mechanical Engineers, Part D: Journal of Automobile Engineering, 230(7):969-982.
Open peer comments: Debate/Discuss/Question/Opinion
<1>