Adaptive Control with Approximated Policy Search Approach
DOI:
https://doi.org/10.5614/itbj.eng.sci.2010.42.1.2Abstract
Most of existing adaptive control schemes are designed to minimize error between plant state and goal state despite the fact that executing actions that are predicted to result in smaller errors only can mislead to non-goal states. We develop an adaptive control scheme that involves manipulating a controller of a general type to improve its performance as measured by an evaluation function. The developed method is closely related to a theory of Reinforcement Learning (RL) but imposes a practical assumption made for faster learning. We assume that a value function of RL can be approximated by a function of Euclidean distance from a goal state and an action executed at the state. And, we propose to use it for the gradient search as an evaluation function. Simulation results provided through application of the proposed scheme to a pole -balancing problem using a linear state feedback controller and fuzzy controller verify the scheme's efficacy.
Downloads
References
Sutton, R.S. & Barto, A.G., Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press, 1998.
?strm, K.J. & Witternmark, B., Adaptive Control. Addison-Wesley Publishing Company, 1989.
Liang, C. & Su, J., A New Approach to The Design of Fuzzy Sliding Mode Controller, Fuzzy Sets and Systems, 139, pp. 111-124, 2003.
Lin, C.-K., A Reinforcement Learning Adaptive Fuzzy Controller for Robots, Fuzzy Sets and System, 137, pp. 339-352, 2003.
Nazaruddin, Y.Y., Naba, A. & Liong, T.H., Modified Adaptive Fuzzy Control System Using Universal Supervisory Controller, in Proc. of SCI/ISAS 2000, IX, Orlando, USA, July 2000.
Oh, S.K., Pedrycz, W., Rho, S.B. & Ahn, T.C., Parameters Estimation of Fuzzy Controller and Its Application to Inverted Pendulum, Engineering Applications of Artificial Intelligence, 17, pp. 37-60, 2004.
Park, J., Park, G., Kim, S. & Moon, C., Direct Adaptive Self-Structuring Fuzzy Controller for Nonaffine Nonlinear System, Fuzzy Sets and Systems, 153, pp. 429-445, 2005.
Wang, L.-X., A Course in Fuzzy Systems and Control. New Jersey: Prentice-Hall International, Inc., 1997.
Liuzzo, S., Marino, R. & Tomei, P., Adaptive Learning Control of Linear Systems by Output Error Feedback, Automatica, 43, pp. 669-676, 2007.
Marino, R., Tomeia, P. & C. Verrelli, An Adaptive Tracking Control from Current Measurements for Induction Motors with Uncertain Load Torque and Rotor Resistance, Automatica, 44, pp. 2593-2599, 2008.
Krstic, M., On Using Least-Squares Updates without Regressor Filtering in Identification And Adaptive Control of Nonlinear Systems," Automatica, 45, pp. 731-735, 2009.
Naba, A. & Miyashita, K., Tuning Fuzzy Controller Using Approximated Evaluation Function, in Proc. of the 4th IEEE International Workshop WSTST05, Muroran, Japan, May 2005, pp. 113-122.
"", Gradient-Based Tuning of Fuzzy Controller with Approximated Evaluation Function, in Proc. of Eleventh International Fuzzy Systems Association (IFSA) World Congress, Beijing, China, July 2005, pp. 671-676.
"", FCAPS: Fuzzy controller with approximated policy search approach, Journal of Adv. Comput. Intelligence and Intelligent Informatics, 1(1), pp. 84-92, 2006.
Baird, L. & Moore, A., Gradient Descent for General Reinforcement Learning, Advances in Neural Information Processing Systems, 11, 1999.
Sutton, R.S., McAllester, D., Singh, S. & Mansour, Y., Policy Gradient Methods for Reinforcement Learning with Function Approximation, Advances in Neural Information Processing System, 12, pp. 1057-1063, 2000.
Berenji, H.R. & Khedkar, P., Learning and Tuning Fuzzy Logic Controllers Through Reinforcement, IEEE Trans. Neural Networks, 3(5), pp. 724-740, 1992.
"", Using Fuzzy Logic for Performance Evaluation in Reinforcement Learning, International Journal of Approximate Reasoning, 18, pp. 131-144, 1998.
Barto, A.G., Sutton, R.S. & Anderson, C.W., Neuronlike Adaptive Elements That Can Solve Difficult Learning Control Problems, IEEE Trans. Syst., Man, Cybern., 13(5), pp. 834-846, 1983.
Santamaria, J.C., Sutton, R.R. & Ram, A., Experiment with Reinforcement Learning in Problems with Continuous State and Action Spaces, Adaptive Behavior, 6(2), pp. 163-218, 1998.
Ogata, K., Modern Control Engineering, Englewood Cliffs New Jersey: Prentice Hall, 1997.
.png){kind=logo}
