Full Text:   <825>

Summary:  <272>

CLC number: TU621

On-line Access: 2017-11-06

Received: 2016-09-30

Revision Accepted: 2017-03-28

Crosschecked: 2017-10-11

Cited: 0

Clicked: 2000

Citations:  Bibtex RefMan EndNote GB/T7714


Qian Zhu


-   Go to

Article info.
Open peer comments

Journal of Zhejiang University SCIENCE A 2017 Vol.18 No.11 P.855-870


Real-time energy management controller design for a hybrid excavator using reinforcement learning

Author(s):  Qian Zhu, Qing-feng Wang

Affiliation(s):  State Key Laboratory of Fluid Power and Mechatronic Systems, Zhejiang University, Hangzhou 310027, China

Corresponding email(s):   11125025@zju.edu.cn

Key Words:  Energy management, Real time, Hybrid excavator, Reinforcement learning (RL), Pontryagin’, s minimum principle (PMP)

Qian Zhu, Qing-feng Wang. Real-time energy management controller design for a hybrid excavator using reinforcement learning[J]. Journal of Zhejiang University Science A, 2017, 18(11): 855-870.

@article{title="Real-time energy management controller design for a hybrid excavator using reinforcement learning",
author="Qian Zhu, Qing-feng Wang",
journal="Journal of Zhejiang University Science A",
publisher="Zhejiang University Press & Springer",

%0 Journal Article
%T Real-time energy management controller design for a hybrid excavator using reinforcement learning
%A Qian Zhu
%A Qing-feng Wang
%J Journal of Zhejiang University SCIENCE A
%V 18
%N 11
%P 855-870
%@ 1673-565X
%D 2017
%I Zhejiang University Press & Springer
%DOI 10.1631/jzus.A1600650

T1 - Real-time energy management controller design for a hybrid excavator using reinforcement learning
A1 - Qian Zhu
A1 - Qing-feng Wang
J0 - Journal of Zhejiang University Science A
VL - 18
IS - 11
SP - 855
EP - 870
%@ 1673-565X
Y1 - 2017
PB - Zhejiang University Press & Springer
ER -
DOI - 10.1631/jzus.A1600650

Real-time energy management of a hybrid excavator is addressed using reinforcement learning (RL). Due to the computational complexity and need for a priori knowledge of the load cycles, a traditional optimal control method, like dynamic programming (DP), is not feasible for real-time control. Real-time controllers derived from traditional optimal control methods compute the solutions either in a cycle-dependent manner or far away from the optimal. An RL-based energy management controller is proposed to solve this problem. The simulation and experimental results demonstrate that the RL controller has a better performance than the widely used thermostat and equivalent consumption minimization strategy (ECMS) controllers. It also shows that the RL controller is cycle-independent. pontryagin’;s minimum principle (PMP) is used to obtain the analytical solution of the energy management problem, and this can help to reduce the iteration time in the design process.

This paper presents an interesting topic on the energy management problem of a hybrid excavator. Four different control strategy have been implemented on a simulator over the standard digging cycle and then validated through a dedicated experimental activity on the hybrid prototype.


创新点:1. 通过强化学习算法,设计时间无关的实时能量管理控制器;2. 通过极大值原理求得最优能量管理问题的解析解,并用来辅助实时能量管理控制器设计。
方法:1. 建立负载的马尔科夫模型,运用强化学习算法,得到实时能量管理控制器;2. 运用极大值原理,求得最优能量管理问题的解析解,并将其作为初始能量管理策略;3. 通过仿真模拟和实验研究,验证所设计的实时能量控制器的性能。
结论:1. 基于强化学习的能量管理控制器是一个可以在线应用的与时间无关的实时能量管理控制器; 2. 基于强化学习的能量管理控制器优于广泛使用的恒温控制器和等效消耗最小化策略控制器;3. 基于强化学习的能量管理控制器由于其闭环特性可适用于不同类型的作业工况。


Darkslateblue:Affiliate; Royal Blue:Author; Turquoise:Article


[1]Borhan, H.A., Vahidi, A., Phillips, A.M., et al., 2009. Predictive energy management of a power-split hybrid electric vehicle. American Control Conference, p.3970-3976.

[2]Brahma, A., Guezennec, Y., Rizzoni, G., 2000. Optimal energy management in series hybrid electric vehicles. American Control Conference, p.60-64.

[3]Busoniu, L., Babuska, R., Schutter, D., et al., 2010. Reinforcement Learning and Dynamic Programming Using Function Approximators. CRC Press, Boca Raton, USA, p.101-113.

[4]Chan, C.C., 2007. The state of the art of electric, hybrid, and fuel cell vehicles. Proceedings of the IEEE, 95(4):704-718.

[5]Ehsani, M., Gao, Y., Miller, J.M., 2007. Hybrid electric vehicles: architecture and motor drives. Proceedings of the IEEE, 95(4):719-728.

[6]Geering, H.P., 2007. Optimal Control with Engineering Applications. Springer, New York, USA, p.3-5.

[7]Gosavi, A., 2003. Simulation-based Optimization. Springer, New York, USA, p.197-211.

[8]Jalil, N., Kheir, N.A., Salman, M., 1997. A rule-based energy management strategy for a series hybrid vehicle. American Control Conference, p.689-693.

[9]Johri, R., Filipi, Z., 2009. Low-cost pathway to ultra efficient city car: series hydraulic hybrid system with optimized supervisory control. SAE International Journal of Engines, 2(2):505-520.

[10]Kim, H., Choi, J., Yi, K., 2012. Development of supervisory control strategy for optimized fuel consumption of the compound hybrid excavator. Proceedings of the Institution of Mechanical Engineers, Part D: Journal of Automobile Engineering, 226(12):1652-1666.

[11]Kim, H., Yoo, S., Cho, S., et al., 2016. Hybrid control algorithm for fuel consumption of a compound hybrid excavator. Automation in Construction, 68:1-10.

[12]Kim, N., Rousseau, A., 2012. Sufficient conditions of optimal control based on Pontryagin’s minimum principle for use in hybrid electric vehicles. Proceedings of the Institution of Mechanical Engineers, Part D: Journal of Automobile Engineering, 226(9):1160-1170.

[13]Kim, N., Cha, S., Peng, H., 2011. Optimal control of hybrid electric vehicles based on Pontryagin’s minimum principle. IEEE Transactions on Control Systems Technology, 19(5):1279-1287.

[14]Lin, C.C., Peng, H., Grizzle, J.W., et al., 2003. Power management strategy for a parallel hybrid electric truck. IEEE Transactions on Control Systems Technology, 11(6):839-849.

[15]Lin, T., Wang, Q., Hu, B., et al., 2010. Development of hybrid powered hydraulic construction machinery. Automation in Construction, 19(1):11-19.

[16]Lin, X., Pan, S.X., Wang, D.Y., 2008. Dynamic simulation and optimal control strategy for a parallel hybrid hydraulic excavator. Journal of Zhejiang University-SCIENCE A, 9(5):624-632.

[17]Musardo, C., Rizzoni, G., Guezennec, Y., et al., 2005. A-ECMS: an adaptive algorithm for hybrid electric vehicle energy management. European Journal of Control, 11(4-5):509-524.

[18]Salman, M., Schouten, N.J., Kheir, N.A., 2000. Control strategies for parallel hybrid vehicles. American Control Conference, p.524-528.

[19]Salmasi, F.R., 2007. Control strategies for hybrid electric vehicles: evolution, classification, comparison, and future trends. IEEE Transactions on Vehicular Technology, 56(5):2393-2404.

[20]Schouten, N.J., Salman, M.A., Kheir, N.A., 2002. Fuzzy logic control for parallel hybrid vehicles. IEEE Transactions on Control Systems Technology, 10(3):460-468.

[21]Sciarretta, A., Guzzella, L., 2007. Control of hybrid electric vehicles. IEEE Control Systems, 27(2):60-70.

[22]Sciarretta, A., Back, M., Guzzella, L., 2004. Optimal control of parallel hybrid electric vehicles. IEEE Transactions on Control Systems Technology, 12(3):352-363.

[23]Serrao, L., Onori, S., Rizzoni, G., 2009. ECMS as a realization of Pontryagin’s minimum principle for HEV control. American Control Conference, p.3964-3969.

[24]Serrao, L., Onori, S., Rizzoni, G., 2011. A comparative analysis of energy management strategies for hybrid electric vehicles. Journal of Dynamic Systems, Measurement, and Control, 133(3):031012.

[25]Watkins, C., Dayan, P., 1992. Q-learning. Machine Learning, 8(3-4):279-292.

[26]Xiao, Q., Wang, Q., Zhang, Y., 2008. Control strategies of power system in hybrid hydraulic excavator. Automation in Construction, 17(4):361-367.

[27]Zhang, S., Xiong, R., 2015. Adaptive energy management of a plug-in hybrid electric vehicle based on driving pattern recognition and dynamic programming. Applied Energy, 155:68-78.

[28]Zhu, Q., Wang, Q., Chen, Q., 2016. Component sizing for compound hybrid excavators. Proceedings of the Institution of Mechanical Engineers, Part D: Journal of Automobile Engineering, 230(7):969-982.

Open peer comments: Debate/Discuss/Question/Opinion


Please provide your name, email address and a comment

Journal of Zhejiang University-SCIENCE, 38 Zheda Road, Hangzhou 310027, China
Tel: +86-571-87952783; E-mail: cjzhang@zju.edu.cn
Copyright © 2000 - Journal of Zhejiang University-SCIENCE