Machine learning technique where agents learn from demonstrations
Imitation learning is a paradigm inreinforcement learning, where an agent learns to perform a task bysupervised learning from expert demonstrations. It is also calledlearning from demonstration andapprenticeship learning.[1][2][3]
It has been applied to underactuated robotics,[4] self-driving cars,[5][6][7] quadcopter navigation,[8] helicopter aerobatics,[9] and locomotion.[10][11]
Behavior Cloning (BC) is the most basic form of imitation learning. Essentially, it uses supervised learning to train a policy such that, given an observation, it would output an action distribution that is approximately the same as the action distribution of the experts.[12]
BC is susceptible todistribution shift. Specifically, if the trained policy differs from the expert policy, it might find itself straying from expert trajectory into observations that would have never occurred in expert trajectories.[12]
This was already noted byALVINN, where they trained a neural network to drive a van using human demonstrations. They noticed that because a human driver never strays far from the path, the network would never be trained on what action to take if it ever finds itself straying far from the path.[5]
DAgger (DatasetAggregation)[13] improves on behavior cloning by iteratively training on a dataset of expert demonstrations. In each iteration, the algorithm first collects data by rolling out the learned policy. Then, it queries the expert for the optimal action on each observation encountered during the rollout. Finally, it aggregates the new data into the datasetand trains a new policy on the aggregated dataset.[12]
The Decision Transformer approach models reinforcement learning as a sequence modelling problem.[14] Similar to Behavior Cloning, it trains a sequence model, such as aTransformer, that models rollout sequenceswhere is the sum of future reward in the rollout. During training time, the sequence model is trained to predict each action, given the previous rollout as context:During inference time, to use the sequence model as an effective controller, it is simply given a very high reward prediction, and it would generalize by predicting an action that would result in the high reward. This was shown toscale predictably to a Transformer with 1 billion parameters that is superhuman on 41Atari games.[15]
Inverse Reinforcement Learning (IRL) learns a reward function that explains the expert's behavior and then uses reinforcement learning to find a policy that maximizes this reward.[18] Recent works have also explored multi-agent extensions of IRL in networked systems.[19]
Generative Adversarial Imitation Learning (GAIL) usesgenerative adversarial networks (GANs) to match the distribution of agent behavior to the distribution of expert demonstrations.[20] It extends a previous approach using game theory.[21][16]
^Russell, Stuart J.; Norvig, Peter (2021). "22.6 Apprenticeship and Inverse Reinforcement Learning".Artificial intelligence: a modern approach. Pearson series in artificial intelligence (Fourth ed.). Hoboken: Pearson.ISBN978-0-13-461099-3.
^Sutton, Richard S.; Barto, Andrew G. (2018).Reinforcement learning: an introduction. Adaptive computation and machine learning series (Second ed.). Cambridge, Massachusetts: The MIT Press. p. 470.ISBN978-0-262-03924-6.
^Bojarski, Mariusz; Del Testa, Davide; Dworakowski, Daniel; Firner, Bernhard; Flepp, Beat; Goyal, Prasoon; Jackel, Lawrence D.; Monfort, Mathew; Muller, Urs (2016-04-25). "End to End Learning for Self-Driving Cars".arXiv:1604.07316v1 [cs.CV].
^Kiran, B Ravi; Sobh, Ibrahim; Talpaert, Victor; Mannion, Patrick; Sallab, Ahmad A. Al; Yogamani, Senthil; Perez, Patrick (June 2022). "Deep Reinforcement Learning for Autonomous Driving: A Survey".IEEE Transactions on Intelligent Transportation Systems.23 (6):4909–4926.arXiv:2002.00444.Bibcode:2022ITITr..23.4909K.doi:10.1109/TITS.2021.3054625.ISSN1524-9050.
^V. S. Donge, B. Lian, F. L. Lewis and A. Davoudi, "Multiagent Graphical Games With Inverse Reinforcement Learning," in IEEE Transactions on Control of Network Systems, vol. 10, no. 2, pp. 841-852, June 2023, doi:10.1109/TCNS.2022.3210856.
