SMILe

From PRLT
Jump to: navigation, search
SMILe.jpg


This project aims at designing algorithms that mimic the development process of human being and animal puppies.


Introduction

The main hypothesis of Reinforcement Learning is that a reinforcement singal is provided to the agent by a critic in the environment. On the other hand, human beings often learn without a direct and explicit reinforcement from the surrounding environment but they act guided by an intrinsic motivation at reaching one goal. This mechanism has been proven to be at the basis of many development processes in human beings and animal puppies. Recently, the Intrinsically Motivated Reinforcement Learning (IMRL) has been proposed as a framework to explain some of the development mechanisms that lead humans to develop skills and abilities through a direct interection with the envirionment and guided by an intrinsic motivation.

This research project has many facets. The main goal is to deepen the biological and psychological plausibility of the IMRL framework to explain human development. At the same time, it could represent a fundamental step towards the definition of techniques to provide robots with the capability to improve their abilities and to develop new skills that can be used in many different tasks. Finally, this research project is strictly related to the project of transfer learning that aims at enabling artificial agents to reuse their knowledge in analogous situations.

Inspired by studies from psychology, neuroscience, and machine learning, SMILe (Self-Motivated Incremental Learning) is a learning framework that allows artificial agents to autonomously identify and learn a set of abilities useful to face several different tasks, through an iterated three phase process: by means of a random exploration of the environment (babbling phase), the agent identifies interesting situations and generates an intrinsic motivation (motivating phase) aimed at learning how to get into these situations (skill acquisition phase). This process incrementally increases the skills of the agent, so that new interesting configurations can be experienced.

One of the main contributions of SMILe is the formalization of a very general self-development process that allows artificial agents to autonomously increase their skills without any prior information about the environment they operate in.

Ongoing work

SMILe is related to many different fields, such as Developmental Robotics, Subgoal Discovery in Reinforcement Learning, Exploration-Exploitation Issues and Intrinsically Motivated Learning. Current research is focused on deepening all the different aspects covered by the three-phase development process of SMILe. In particular, we are focusing on:

  • Factored SMILe: the perceptual capabilities of the agent increases as the development process progresses and as more and more complex skills are acquired. Therefore, in an environment characterized by a large set of variables, at the beginning the agent is not able to identify the correlations between all the variables but they are perceived as uncorrelated. As a result, the agent identifies as interesting only particular values of the variables and this leads to the creation of general skills. As the agent discovered general subgoals, its perceptual capabilities improve and it can learn skills for more specific and complex tasks. This line of research is focused on translating this process of incremental perceptual complexity into the SMILe development process.
  • Exploration-Exploitation Dilemma: the identification of interesting goals is strictly related to the concept of an efficient exploration of the environment. In fact, the skills generated during the development process are more effective when they allow the agent to reach difficult states in few steps. This line of research is focused on the study of the intrinsically motivated learning process under the perspective of the exploration-exploitation dilemma.
  • Transfer Learning: the ability to develop general and re-usable skills is related to the issue of transferring the knowledge among different domains and for solving different tasks. This line of research is focused on the study of SMILe as a possible approach to transfer learning.

Resources

An introduction to SMILe

ICDL Talk

SAB Poster

People involved

  • Andrea Bonarini, Full Professor
  • Alessandro Lazaric, PhD Student
  • Marcello Restelli, PhD
  • Matteo Lazzarotto, Master Student
  • Emanuele Venneri, Master Student

Publications

Andrea Bonarini, Alessandro Lazaric, Marcello Restelli - Incremental Skill Acquisition for Self-Motivated Learning Animats
From Animals to Animats 9. 9th International Conference on Simulation of Adaptive Behavior (SAB'06), Rome, Italy, September 2006 4095:357--368, Berlin, D, 2006
Bibtex
Auteur : Andrea Bonarini, Alessandro Lazaric, Marcello Restelli
Titre : Incremental Skill Acquisition for Self-Motivated Learning Animats
Dans : From Animals to Animats 9. 9th International Conference on Simulation of Adaptive Behavior (SAB'06), Rome, Italy, September 2006 -
Adresse : Berlin, D
Date : 2006

Andrea Bonarini, Alessandro Lazaric, Marcello Restelli - Self-Development Framework for Reinforcement Learning Agents
Fifth International Conference on Development and Learning (ICDL) , 2006
Bibtex
Auteur : Andrea Bonarini, Alessandro Lazaric, Marcello Restelli
Titre : Self-Development Framework for Reinforcement Learning Agents
Dans : Fifth International Conference on Development and Learning (ICDL) -
Adresse :
Date : 2006

Andrea Bonarini, Alessandro Lazaric, Marcello Restelli - Learning Reusable Skills through Self-Motivation
Workshop on Structural Knowledge Transfer for Machine Learning at the International Conference of Machine Learning , 2006
Bibtex
Auteur : Andrea Bonarini, Alessandro Lazaric, Marcello Restelli
Titre : Learning Reusable Skills through Self-Motivation
Dans : Workshop on Structural Knowledge Transfer for Machine Learning at the International Conference of Machine Learning -
Adresse :
Date : 2006

Patrick Vitali - Self-development of Artificial Agents with Self-motivated Learning
Thèse de master, Politecnico di Milano , 2005
Bibtex
Auteur : Patrick Vitali
Titre : Self-development of Artificial Agents with Self-motivated Learning
Dans : Thèse de master, Politecnico di Milano -
Adresse :
Date : 2005

Matteo Lazzarotto - Representation of Interest in Intrinsically Motivated Reinforcement Learning for complex environments
Thèse de master, Politecnico di Milano , 2007
Bibtex
Auteur : Matteo Lazzarotto
Titre : Representation of Interest in Intrinsically Motivated Reinforcement Learning for complex environments
Dans : Thèse de master, Politecnico di Milano -
Adresse :
Date : 2007