We present a simulation framework for biomimetic human perception and sensorimotor control. It features a biomechanically simulated, musculoskeletal human model actuated by numerous skeletal muscles, with two human-like eyes whose retinas have spatially nonuniform distributions of photoreceptors. Our prototype sensorimotor system for this model incorporates a set of 20 automatically-trained, deep neural networks (DNNs), half of which are neuromuscular DNN controllers comprising its motor subsystem, while the other half are devoted to visual perception. Within the sensory subsystem, which continuously operates on the retinal photoreceptor outputs, 2 DNNs drive eye and head movements, while 8 DNNs extract the sensory information needed to control the arms and legs. Exclusively by means of its egocentric, active visual perception, our biomechanical virtual human learns efficient, online visuomotor control of its eyes, head, and four limbs to perform tasks involving the foveation and visual pursuit of target objects coupled with visually-guided reaching actions to intercept the moving targets.

Authors and Affiliations

1. University of California, Los Angeles, USA

   Masaki Nakada, Honglin Chen & Demetri Terzopoulos

Corresponding author

Correspondence toMasaki Nakada.

Editors and Affiliations

1. University of Nevada, Reno, USA

   George Bebis

2. NASA Ames Research Center, Moffett Field, USA

   Richard Boyle

3. University of Nevada, Reno, USA

   Bahram Parvin

4. Desert Research Institute, Reno, USA

   Darko Koracin

5. DARPA, Arlington, USA

   Matt Turek

6. University of Utah, Salt Lake City, USA

   Srikumar Ramalingam

7. National University of Defense Technology, Changsha, China

   Kai Xu

8. Microsoft Research Asia, Beijing, China

   Stephen Lin

9. Bosch Research, Farmington Hills, MI, USA

   Bilal Alsallakh

10. University of North Carolina at Charlotte, Charlotte, USA

    Jing Yang

11. Microsoft Research, Redmond, USA

    Eduardo Cuervo

12. University of Colorado at Colorado Springs, Colorado Springs, USA

    Jonathan Ventura

Reprints and permissions

© 2018 Springer Nature Switzerland AG

Policies and ethics