My program of research is an interdisciplinary study of mechanisms and algorithms permitting systems—both natural and artificial—to improve performance with experience, that is, to learn. I direct the Autonomous Learning Laboratory in the Department of Computer Science, which focuses on learning in both machines and animals. We are a highly interdisciplinary lab, interacting with researchers in psychology, neuroscience, control engineering, operations research, and robotics. We are best known for pioneering work in Reinforcement Learning. This is a framework for learning to maximize reward over time while interacting with a dynamic environment. We also work on neural models of animal motor learning, maintaining close contact with the laboratory of Prof. John Moore, and on the development of motor control abilities by infants in collaboration with Profs. Neil Berthier and Rachel Keen. Biological control systems demonstrate an amazing ability to deal with complex and ever-changing bodies and environments.

Andrew G. Barto Computational and Neural Models of Motor Learning; Reinforcement Learning The objective of one aspect of our research is to use mathematical and computer models to refine and test hypotheses about how the cerebellum, motor cortex, and basal ganglia function together in learning to support motor activities. The resulting models help us to better understand how the brain performs control, and can also provide inspiration for the design of new robotic control techniques. Our current work focuses on the cerebellum’s role in learning to perform accurate and smooth arm movements. We have constructed a model of the cerebellum that learns how to control a simulated arm with biologically realistic sensorimotor loop delays and training signals. The role of the cerebellum is to incorporate goal position, delayed sensory afferents, and motor efference copy signals to produce an appropriate muscle bursting pattern that brings the arm to the target position quickly and accurately. In a closely related project, we bring modern knowledge from developmental psychology about how infants learn elementary motor skills together with expertise in mathematical and computer modeling of adaptive systems and motor control. Our objectives include evaluating influential existing models of motor learning in light of developmental data, extracting principal areas of agreement and disagreement, conducting behavioral experiments with human infants designed to illuminate these issues, and developing new models of motor development.

We are also working with robotics researchers here at UMass to study ways of organizing purposeful, coordinated action for complex robotic systems operating in unstructured environments. This work emphasizes adaptive skill acquisition through the activation of combinations of reusable feedback control laws. Here, reinforcement learning techniques are employed to synthesize behavior on-line. This research is performed in collaboration with Prof. Roderic Grupen of the Laboratory for Perceptual Robotics. The Autonomous Learning Laboratory also continues to study basic problems in computational reinforcement learning. For example, in collaboration with Dr. Richard Sutton of AT&T Research, we are developing new approaches to learning and planning which allow systems to learn and plan at multiple temporal scales. We have developed a mathematical theory that defines what models are suitable for planning and a method by which a reinforcement learning system can learn multiple time scale models and use them as the basis for hierarchical learning and planning. Our future objectives include demonstrating the effectiveness of this approach in a number of tasks, and examining its relationship to control theory and to behavioral and neural models of learning.   For additional information, please visit my Computer Science Homepage.

Representative Publications:

J. Houk, A. Barto, A. Fagg. Fractional Power Damping Model of Joint Motion". In Progress in Motor Control (M. Latash, Ed.), in press.

R. Moll, T. Perkins, and A. Barto. "Machine Learning for Subproblem Selection". In Proceedings of the Seventeenth International Conference on Machine Learning (ICML-2000), P. Langley (Ed.), Morgan Kaufmann, San Francisco, CA, 2000, pp. 615-622.

R. Moll, T. Perkins, and A. Barto. (1999) "Learned Subproblem Selection Techniques for Combinatorial Optimization". Computer Science Technical Report 99-67, University of Massachusetts.

J. Randlov, A. Barto, and M. Rosenstein. "Combining Reinforcement Learning with a Local Control Algorithm". In Proceedings of the Seventeenth International Conference on Machine Learning (ICML-2000), P. Langley (Ed.), Morgan Kaufmann, San Francisco, CA, 2000, pp. 775-782.

R. Moll, A. Barto, T. Perkins, and R. Sutton. (1999) "Learning Instance-Independent Value Functions to Enhance Local Search". In Advances in Neural Information Processing Systems 11 (NIPS11), M. S. Kearns, S. A. Solla, and D. A. Cohn (Eds.), Cambridge, MA: MIT Press, pp. 1017-1023. (93518 bytes)

A. McGovern, E. Moss, and A. Barto. "Building a Basic Building Block Scheduler Using Reinforcement Learning and Rollouts". Machine Learning, Special Issue on Reinforcement Learning, to appear.

Barto, A.G., Fagg, A.H., Sitkoff, N., and Houk, J.C. (1999) A cerebellar model of timing and prediction in the control of reaching, Neural Computation, 11: 565-594.

R. Crites and A. Barto. (1998) "Elevator Group Control Using Multiple Reinforcement Learning Agents". Machine Learning 33, pp. 235-262. (81045 bytes)

Sutton, R.S. and Barto, A.G. (1998) Reinforcement Learning: An Introduction. MIT Press, Cambridge MA.

A. Fagg, L. Zelevinsky, A. Barto, and J. Houk. (1998) "A Pulse-Step Model of Control for Arm Reaching Movements". In Proceedings of the Spring Meeting on the Neural Control of Movement.

A. Fagg, A. Barto, and J. Houk. (1998) " Learning to Reach Via Corrective Movements". In Proceedings of the Tenth Yale Workshop on Adaptive and Learning Systems, New Haven, CT, pp. 179-185.

R. E. Kettner, S., Mahamud, H. -C. Leung, N. Sitkoff, J. C. Houk, B. W. Peterson, and A. G. Barto. (1997) "Prediction of Complex Two-Dimensional Trajectories by the Eye and by a Cerebellar Model of Smooth Eye Movements". Journal of Neurophysiology, 77, pp. 2115-2130.

A. Fagg, L. Zelevinsky, A. Barto, and J. Houk. (1997) "Using Crude Corrective Movements to Learn Accurate Motor Programs for Reaching". Presented at the NIPS workshop on Can Artificial Cerebellar Models Compete to Control Robots, Breckenridge, CO.

A. Fagg, N. Sitkoff, A. Barto, J. Houk.(1997) "Cerebellar Learning for Control of a Two-Link Arm in Muscle Space". In Proceedings of the IEEE Conference on Robotics and Automation, pp. 2638-2644.

A. Fagg, N. Sitkoff, A. Barto, J. Houk. (1997) "A Model of Cerebellar Learning for Control of Arm Movements Using Muscle Synergies". In Proceedings of the IEEE International Symposium on Computational Intelligence in Robotics and Automation, pp. 6-12.

R. Crites and A. Barto. (1996) "Improving Elevator Performance Using Reinforcement Learning". In Advances in Neural Information Processing Systems 8 (NIPS8), D. S. Touretzky, M. C. Mozer, and M. E. Hasslemo (Eds.), Cambridge, MA: MIT Press, pp. 1017-1023. (nips8.ps.Z: 58525 bytes)

A. G. Barto, S. J. Bradtke and S. P. Singh. (1996) "Learning to act using real-time dynamic programming". Artificial Intelligence, Special Volume: Computational Research on Interaction and Agency, 72, 1995, pp. 81-138. (Reprinted in Computational Theories of Interaction and Agency, P. E. Agre & S. J. Rosenschein (Eds.), Cambridge, MA: MIT Press (584607 bytes)

Barto, A.G., Buckingham, J.T., and Houk, J.C. (1996) A predictive switching model of cerebellar movement control. In D.S. Touretzky, M.C. Mozer and M.E. Hasselmo (Eds.) Advances in Neural Information Processing Systems 8, Cambridge, MA: MIT Press, pp. 138-144.

Barto, A. (1996) Reinforcement learning. In Neural Systems for Control, O. Omidvar and D. Elliott (Eds.) San Diego: Academic Press, pp. 7-30.

Houk, J.C., Buckingham, J.T. and Barto, A.G. (1996) Models of the cerebellum and motor learning. Behavioral and Brain Sciences, 19:368-383.

Mahamud, S., Barto, A.G., Kettner, R.E. and Houk, J.C. (1996) A model of prediction in smooth eye movements. In Computational Neuroscience, J. Bower (Ed.), New York: Academic, pp. 379-384.

Barto, A.G. (1995) Adaptive critics and the basal ganglia. In Models of Information Processing in the Basal Ganglia, J.C. Houk, J. Davis, and D. Beiser (eds.) MIT Press, Cambridge MA, pp. 215–232.

Barto, A.G. (1995) Reinforcement learning in motor control. In The Handbook of Brain Theory and Neural Networks, M.A. Arbib (Ed.) Cambridge, MA: The MIT Press, pp. 809-813.

Barto, A.G., Bradtke, S.J., and Singh, S.P. (1995) Learning to act using real-time dynamic programming, Artificial Intelligence 72: 81–138.

 

Back to NSB Faculty Index