Li, Zhibin (Alex)

Komura, Taku

Triantafyllidis, Eleftherios

Engineering and Physical Sciences Research Council (EPSRC)

Horizon 2020 project Harmony

FairSpace project

2024-02-16T12:41:19Z

2024-02-16T12:41:19Z

2024-02-16

From a biological perspective, humans possess incredible sensory, dexterous and cognitive abilities. By virtue of these abilities, humans and more broadly their sensory systems are able to adapt to environmental demands seamlessly. Switching from a biological perspective to robotic systems as embodied intelligence, achieving such adaptation is currently far from trivial. Understanding the biological mechanisms that govern and render humans proficient in interacting with their surroundings, could ultimately illuminate new pathways to replicate such human-like cognition and dexterity in machines. Inspired by the aforementioned narrative, this thesis delves into and addresses four main research areas. The first contribution of this thesis provides novel insights into the intricate interplay of multimodal interfaces, their impact on the human sensory-motor system and their correlation to the generation of meaningful motor actions. Different sensory modalities are examined, entailing a full factorial comparison of auditory, visual and somatosensory states and their influence on motor performance. Through a series of varying complexity motor tasks with human subjects, a correlation is established between sensory states and their influence on motor actions. Results provide novel evidence of which sensory combinations contribute to enhanced task performance and how these can be harnessed. The second contribution of this work is the derivation of a novel metric capable of quantifying motor actions stemming from the intricate human sensory-motor system. Measuring human motor performance is a complex phenomenon and the absence of a standardised metric renders inter-study comparability challenging. To this end, four motor tasks, increasing in spatial complexity, were devised to establish a correlation of which spatial variables influence motor performance. Results revealed which spatial variables had the most notable effect, highlighting that existing metrics are inadequate for modelling higher dimensions. To account for this, a novel metric is derived, capable of modelling human motor performance in full 3D space, underlining its value for quantifying commonly seen motor movements and enhancing inter-study comparability. The third and penultimate contribution builds on the foundation laid by the preceding segments, striving to align human capabilities closer to embodied intelligence. To realise this aim, inspired by a biological standpoint, the RObotic MAnipulation Network (ROMAN) is introduced. ROMAN is a novel Hybrid Hierarchical Learning (HHL) architecture designed to address the challenges of notably complex long-horizon sequential tasks. ROMAN utilises the exploratory nature of Reinforcement Learning (RL) while simultaneously exploiting the higher-level skills of humans in the form of imitation. Consisting of a plethora of specialising skills, ROMAN's hierarchical architecture demonstrates versatility in intricate, long-horizon sequential tasks; while exhibiting robustness against various levels of sensory uncertainties. By virtue of the HHL employed, ROMAN also exhibits adaptability beyond demonstrated behaviour; featuring failure recovery capabilities and adaptation in avoiding local minima. These results underline the significance of ROMAN for autonomous manipulation tasks necessitating intelligent and adaptive behaviour. The fourth and concluding contribution of this thesis investigates the potential of language-guided exploration in augmenting embodied intelligence. In pursuit of this goal, the Intrinsically Guided Exploration from Large Language Models (IGE-LLMs) framework is presented, capable of complementing the existing bio-inspired hierarchy of ROMAN. By harnessing LLMs as an assistive intrinsic reward source alongside the conventional RL paradigm, IGE-LLMs enhances the exploratory process to address intricate settings challenged by sparse rewards and long-horizons. Validated on environments challenged by exploration and long-horizons, IGE-LLMs exhibits notably higher performance over existing methods and is capable of complementing the shortcomings of using LLMs in isolation. Moreover, the modularity and robustness of IGE-LLMs is underscored, due to its ability to complement existing intrinsic reward methods and its insensitivity to most intrinsic scaling parameters. Finally, the framework's resilience is highlighted over existing methods when faced with increased uncertainties and horizons. Capable of fostering exploration and the ability to automate the orchestration of ROMAN's intricate macro-actions, IGE-LLMs value as a language-guided framework is underlined. This thesis provides novel findings on harnessing human sensory-motor abilities for generating meaningful motor actions, which can be adequately measured and quantified. Ultimately, these represent the inspiration for shaping the development of a novel bio-inspired learning method to align human capabilities closer to embodied intelligence that can further be complemented and automated by eliciting language-guided exploration; tailored to address notably intricate, long-horizon tasks with sparse rewards. Nevertheless, to further narrow the gap between humans and machines, a deeper understanding of designing artificial intelligence inspired by biological insights is necessitated.

https://hdl.handle.net/1842/41453

http://dx.doi.org/10.7488/era/4185

en

The University of Edinburgh

Intrinsic Language-Guided Exploration for Complex Long-Horizon Robotic Manipulation Tasks Triantafyllidis, E., Christianos, F. & Li, A., 29 Jan 2024, (Accepted/In press) International Conference on Robotics and Automation (ICRA, 2024). Yokohama, Japan: IEEE

Metrics for 3D Object Pointing and Manipulation in Virtual Reality Triantafyllidis, E., Hu, W., McGreavy, C. & Li, A., 1 Mar 2022, In: IEEE Robotics and Automation Magazine. 29, 1, p. 76-91 16 p.

Hybrid hierarchical learning for solving complex sequential tasks using the robotic manipulation network ROMAN Triantafyllidis, E., Acero, F., Liu, Z. & Li, A., 7 Sept 2023, In: Nature Machine Intelligence. 5, 9, p. 991-1005 15 p.

The Challenges in Modeling Human Performance in 3D Space with Fitts’ Law Triantafyllidis, E. & Li, Z., 8 May 2021, Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. CHI EA '21. ACM Association for Computing Machinery, p. 1-9 9 p. 56

Study of Multimodal Interfaces and the Improvements on Teleoperation Triantafyllidis, E., McGreavy, C., Gu, J. & Li, Z., 23 Apr 2020, In: IEEE Access. 8, p. 78213 - 78227 15 p.

E. Triantafyllidis, C. Yang, C. McGreavy, W. Hu and Z. Li. (2020). “Robot intelligence for real-world applications”. In: AI for Emerging Verticals: Human-robot computing, sensing and networking, IET Computing and Networks, Book Chapter, Chapter 4.0

E. Triantafyllidis, W. Hu, C. McGreavy and Z. Li. (2021). “Metrics for 3D Object Pointing and Manipulation in Virtual Reality: The Introduction and Validation of a Novel Approach in Measuring Human Performance”. In: IEEE Robotics & Automation Magazine, DOI: 10.1109/MRA.2021.3090070

E. Triantafyllidis and Z. Li. (2021). “The Challenges in Modeling Human Performance in 3D Space with Fitts’ Law”. In: CHI Conference on Human Factors in xxvii Computing Systems Extended Abstracts (CHI ’21 Extended Abstracts). Association for Computing Machinery, May 8–13, 2021, Yokohama, Japan. ACM, New York, NY, USA. DOI: 10.1145/3411763.3443442.

E. Triantafyllidis and Z. Li. (2021). “Considerations and Challenges of Measuring Operator Performance in Telepresence and Teleoperation Entailing Mixed Reality Technologies”. In: CHI Conference on Human Factors in Computing Systems Workshop CHI ’21 (Evaluating User Experiences in Mixed Reality). Association for Computing Machinery, May 7, 2021, Yokohama, Japan. ACM, New York, NY, USA.

W. Yu, C. Yang, C. McGreavy, E. Triantafyllidis, G. Bellegarda, M. Shafiee, A. J. Ijspeert and Z. Li. (2023). “Identifying Important Sensory Feedback for Learning Locomotion Skills”. In: Nature Machine Intelligence, Volume: 5, Issue: 8, Pages: 919–932, DOI: 10.1038/s42256-023-00701-w.

W. Hu, F. Acero, E. Triantafyllidis, Z. Liu and Zhibin Li. (2023). “Modular Neural Network Policies for Learning In-flight Object Catching with a Robot Hand- Arm System”. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

Robotics

Machine Learning

Human Factors

Human-Computer Interaction

Human-Robot Interfaces

Manipulation

Long-Horizon Tasks

Reinforcement Learning

Imitation Learning

Learning from Demonstrations

Embodied Intelligence

Hybrid Learning

Hierarchical Learning

Robot Learning

Biologically-Inspired Intelligence

Advancements in sensory-motor perception and biologically-inspired hierarchical learning for embodied intelligence

Thesis or Dissertation

Doctoral

PhD Doctor of Philosophy