Adaptive Control with Approximated Policy Search Approach

Agus Naba


Most of existing adaptive control  schemes are designed to minimize error  between  plant  state  and  goal  state  despite  the  fact  that  executing  actions that are predicted to result in smaller errors only can mislead  to non-goal states. We develop an adaptive control scheme that involves manipulating a controller of  a  general  type  to  improve  its  performance  as  measured  by  an  evaluation function. The developed method is closely related  to a theory of Reinforcement Learning (RL) but imposes a practical assumption made for faster learning. We assume  that  a  value  function  of  RL  can  be  approximated  by  a  function  of Euclidean distance from a goal state and an action executed at the state. And, we propose  to  use  it  for  the  gradient  search  as  an  evaluation  function.  Simulation results provided through application of the proposed scheme to a pole -balancing problem using a linear  state feedback controller and fuzzy controller verify the scheme’s efficacy.

Full Text:



Sutton, R.S. & Barto, A.G., Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press, 1998.

Åström, K.J. & Witternmark, B., Adaptive Control. Addison-Wesley Publishing Company, 1989.

Liang, C. & Su, J., A New Approach to The Design of Fuzzy Sliding Mode Controller, Fuzzy Sets and Systems, 139, pp. 111–124, 2003.

Lin, C.-K., A Reinforcement Learning Adaptive Fuzzy Controller for Robots, Fuzzy Sets and System, 137, pp. 339–352, 2003.

Nazaruddin, Y.Y., Naba, A. & Liong, T.H., Modified Adaptive Fuzzy Control System Using Universal Supervisory Controller, in Proc. of SCI/ISAS 2000, IX, Orlando, USA, July 2000.

Oh, S.K., Pedrycz, W., Rho, S.B. & Ahn, T.C., Parameters Estimation of Fuzzy Controller and Its Application to Inverted Pendulum, Engineering Applications of Artificial Intelligence, 17, pp. 37–60, 2004.

Park, J., Park, G., Kim, S. & Moon, C., Direct Adaptive Self-Structuring Fuzzy Controller for Nonaffine Nonlinear System, Fuzzy Sets and Systems, 153, pp. 429–445, 2005.

Wang, L.-X., A Course in Fuzzy Systems and Control. New Jersey: Prentice-Hall International, Inc., 1997.

Liuzzo, S., Marino, R. & Tomei, P., Adaptive Learning Control of Linear Systems by Output Error Feedback, Automatica, 43, pp. 669–676, 2007.

Marino, R., Tomeia, P. & C. Verrelli, An Adaptive Tracking Control from Current Measurements for Induction Motors with Uncertain Load Torque and Rotor Resistance, Automatica, 44, pp. 2593–2599, 2008.

Krstic, M., On Using Least-Squares Updates without Regressor Filtering in Identification And Adaptive Control of Nonlinear Systems,” Automatica, 45, pp. 731–735, 2009.

Naba, A. & Miyashita, K., Tuning Fuzzy Controller Using Approximated Evaluation Function, in Proc. of the 4th IEEE International Workshop WSTST05, Muroran, Japan, May 2005, pp. 113–122.

——, Gradient-Based Tuning of Fuzzy Controller with Approximated Evaluation Function, in Proc. of Eleventh International Fuzzy Systems Association (IFSA) World Congress, Beijing, China, July 2005, pp. 671–676.

——, FCAPS: Fuzzy controller with approximated policy search approach, Journal of Adv. Comput. Intelligence and Intelligent Informatics, 1(1), pp. 84–92, 2006.

Baird, L. & Moore, A., Gradient Descent for General Reinforcement Learning, Advances in Neural Information Processing Systems, 11, 1999.

Sutton, R.S., McAllester, D., Singh, S. & Mansour, Y., Policy Gradient Methods for Reinforcement Learning with Function Approximation, Advances in Neural Information Processing System, 12, pp. 1057–1063, 2000.

Berenji, H.R. & Khedkar, P., Learning and Tuning Fuzzy Logic Controllers Through Reinforcement, IEEE Trans. Neural Networks, 3(5), pp. 724–740, 1992.

——, Using Fuzzy Logic for Performance Evaluation in Reinforcement Learning, International Journal of Approximate Reasoning, 18, pp. 131–144, 1998.

Barto, A.G., Sutton, R.S. & Anderson, C.W., Neuronlike Adaptive Elements That Can Solve Difficult Learning Control Problems, IEEE Trans. Syst., Man, Cybern., 13(5), pp. 834–846, 1983.

Santamaria, J.C., Sutton, R.R. & Ram, A., Experiment with Reinforcement Learning in Problems with Continuous State and Action Spaces, Adaptive Behavior, 6(2), pp. 163–218, 1998.

Ogata, K., Modern Control Engineering, Englewood Cliffs New Jersey: Prentice Hall, 1997.



  • There are currently no refbacks.