Manipulation and Vision
Real-world manipulation remains one of the biggest challenges in robotics. Compared to humans, robots are still very poor at ‘understanding’ the world they see around them, and using hand-eye-coordination to manipulate objects. The Centre is working hard to address both of these shortcomings through the Manipulation and Vision Research Project.
Project Leaders
Jürgen “Juxi” Leitner
Queensland University of Technology
Jürgen “Juxi” Leitner is co-founder of LYRO Robotics, a startup creating robotic systems that can pick and pack a large range of objects, thanks to a tight integration of computer vision and robotic manipulation. He is also a part-time post-doctoral research fellow at QUT leading the Manipulation and Vision project within the Centre. He works with students at the interaction of vision, robotics, and manipulation, most notable he was leading Team ACRV into victory at the 2017 Amazon Robotics Challenge team.
Juxi received a PhD for his work on making the iCub humanoid robot see and interact with the world at the Robotics Lab of the Dalle Molle Institute for AI (IDSIA). He previously worked on AI and robotics at the European Space Agency’s Advanced Concepts Team. His background includes a Joint European Master in Space Science and Technology (SpaceMaster) and a BSc degree in Computer Science from the Vienna University of Technology.
Akansel Cosgun
Monash University, Australia
Akansel Cosgun is a Research Fellow with the Centre at Monash University. He holds a PhD in Robotics from Georgia Institute of Technology.
Akansel is interested in developing robotic systems that can do useful tasks in the real world. His research interests include robot manipulation, machine learning, and human-robot interaction.
Team Members
Robert Mahony
Australian National University (ANU), Australia
Rob Mahony is a Professor in the Research School of Engineering at the Australian National University and has been a Chief Investigator with the Centre since its inauguration in 2014. His research interests are in non-linear systems theory with applications in robotics and computer vision.
He wrote the seminal paper providing a clear exposition of non-linear complementary filters on the special orthogonal group for attitude estimation; an enabling technology in the early development of quadrotor aerial robotic vehicles. He was the first to provide a principled analysis for using optical flow of control of aerial robotic vehicles and was a coauthor on the first experimental paper that demonstrated landing of a quadrotor vehicle on a textured but featureless moving surface. In 2016, Rob was named a Fellow of the IEEE, recognising his contribution to the control aspects of aerial robotics.
Peter Corke
Queensland University of Technology (QUT), Australia
Peter Corke is a distinguished Professor at the Queensland University of Technology and director of the ARC Centre of Excellence for Robotic Vision in Australia. Previously he was a Senior Principal Research Scientist at the CSIRO ICT Centre where he founded and led the Autonomous Systems laboratory. He is a Fellow of the IEEE, the Australian Academy of Technology and Engineering (ATSE) and Senior Fellow of the Higher Education Academy (HEA).
He was the Editor-in-Chief of the IEEE Robotics and Automation magazine; founding editor of the Journal of Field Robotics; member of the editorial board of the International Journal of Robotics Research, and the Springer STAR series. He has over 500 publications in the field, a h-index of 63 and over 20,000 citations. Peter has held visiting positions at the University of Pennsylvania, University of Illinois at Urbana-Champaign, Carnegie-Mellon University Robotics Institute, and Oxford University.
Douglas Morrison
Queensland University of Technology (QUT), Australia
Doug is a PhD researcher QUT, supervised by Dr Juxi Leitner and Professor Peter Corke. His research is developing new strategies for robotic grasping in the unstructured and dynamic environments of the real world, that is, strategies which are general, reactive and knowledgeable about their environments. The goal: create robots that can grasp objects anywhere, all the time. Doug was also the lead developer of Cartman, the ACRV’s winning entry into the 2017 Amazon Robotics Challenge!
Vibhavari Dasagi
Queensland University of Technology (QUT), Australia
Vibha completed her Bachelor degree in Mechatronics Engineering from Monash University (Malaysian campus) before completing her Masters in Robotics at the University of Pennsylvania. While at UPenn, she was part of the team who competed in RoboCup 2013 and won the Humanoid League. She joined the Centre in 2019 and is completing her PhD in Artificial Curiosity supervised by Research Fellow Juxi Leitner and Associate Investigator Thierry Peynot.
Vibha is highly interested in understanding how the human brain works and emulating it in artificial agents, and believes curious agents are a step towards achieving it.
Jordan Erskine
Queensland University of Technology (QUT), Australia
Jordan graduated from QUT in 2017 with first class honours in a Bachelor of Mechatronic Engineering. While there, he worked with QUT’s team for the Amazon Picking Challenge, as well as working on a CSIRO project involving developing autonomous surveying with UAVs. He started his PhD in 2018 and is supervised by Centre Research Affiliate and QUT Research Fellow Chris Lehnert, Research Fellow Juxi Leitner and Centre Director Peter Corke. His field of study involves developing and improving generalisable robotic manipulation skills.
Jesse Haviland
Queensland University of Technology (QUT), Australia
Jesse graduated from QUT in 2018 with First Class Honours in a Bachelor of Electrical Engineering. He was welcomed to the world of research by his supervisor, Peter Corke when he completed his honours project, and a Vacation Research Experience Scheme (VRES) on the project “Interactive Voice Interface for Robotic Manipulation Demonstrator” in 2018/2019. Jesse currently pursuing his PhD with the Centre supervised by Peter Corke. His research is concerned with mobile manipulation and how vision based mobile platforms can achieve useful outcomes in unstructured environments.
Zheyu Zhuang
Australian National University
Zheyu Zhuang joined the Centre in 2017. His thesis topic is “Learning Robust Hand-eye Coordination for Grasping in Novel Environments” and his research is aligned with the Vision & Action Research Program and VA1 Project, supervised by Rob Mahony, Juxi Leitner, Nick Barnes and Richard Hartley.
Zheyu graduated from the Australian National University with first class honours, majoring in Electronics Engineering and Mechatronics. His research interests are visual servoing, control, machine learning and deep learning. He is working on the 2017 Amazon Robotics Challenge with researchers from QUT and Adelaide.
Robert Lee
Queensland University of Technology (QUT), Australia
Robert graduated from QUT in 2017 with first class honours in a Bachelor of Mechatronics Engineering. During his degree he worked with the Centre on the LunaRoo hopping lunar payload robot, as well as a CSIRO collaboration project involving evolving spiking neural networks for quadrotor control. Robert started his PhD in 2018 and is supervised by Research Fellow Juxi Leitner, Research Fellow Valerio Ortenzi and Centre Director Peter Corke. His interests lie in deep reinforcement learning, control and vision, and he is currently working on applying these tools to improve robotic grasping.
Thomas Rowntree
University of Adelaide
Tom is a Research Programmer based at the University of Adelaide. He joined the Centre in February 2017 and was a key member of Team ACRV who competed and won the 2017 Amazon Robotics Challenge in Japan. Tom completed his Bachelor of Engineering in Mechatronics, Robotics and Automation Engineering in 2012.
Chris Lehnert
Queensland University of Technology (QUT), Australia
Dr Chris Lehnert is a Robotics Lecturer within the Robotics and Autonomous Systems (RAS) discipline at QUT. His research interests lie in the development of novel methods for robotic manipulation in real world and challenging environments. A particular focus of his research has been on enabling robots to perform autonomous harvesting operations in horticulture. He led a small team of PhD students, post-doctoral fellows and engineers in developing new robotic technologies for horticulture through the Strategic Investment in Farm Robotics (SIFR) program at QUT.
Project Aim
Real-world manipulation remains one of the biggest challenges in robotics. Compared to humans, robots are still very poor at ‘understanding’ the world they see around them, and using hand-eye-coordination to manipulate objects. The Centre is working hard to address both of these shortcomings through the Manipulation and Vision Research Project (and Manipulation Demonstrator project). Almost all robot manipulators in operation today are used in the manufacturing industry. They are generally separated from humans in tightly-controlled environments. The aim of this project is to develop methods that enable robot manipulators to interact with people and everyday objects. The project team is focussed on three key challenges: vision-guided reaching, vision-guided grasping and task-focused manipulation.
Key Results
PhD Researcher Zheyu Zhang and Chief Investigator Robert Mahony developed an algorithm that enables a robot arm to reach towards an object using a single camera. Instead of detecting object instances in the image, and guiding the robot towards a given object, they trained an algorithm that moves the arm towards the closest object using just data. Technically, the approach merges ideas from Control Theory which is a century-old field and provides theoretical guarantees, and Deep Learning which allows learning from large datasets.
Visiting PhD student Shray Bansal (Georgia Institute of Technology) spent three months at the Centre’s Monash University node working on collaborative manipulation. This is when a robot manipulator and human work in close proximity. Shray’s work tested how robots should behave when they are working alongside people – should they focus on their own task or assist human workers as much as possible? In a user study with non-robotics people, it was found that people preferred the assistive robot, but the use of assistive robots increased the time to complete a manipulation. That’s because the robot sometimes took actions that slowed the task at the expense of assisting a human worker.
The project team developed approaches for robot learning using the technique “Reinforcement Learning”. During the Reinforcement Learning process, a robot takes actions and learns from reward. PhD Researcher Vibhavari Dasagi worked on eliminating ‘catastrophic forgetting’, which can happen during the learning process. She showed that this can be avoided by restoring a previous version of the trained neural network when necessary
Vibhavari also collaborated with PhD Researcher Robert Lee on ways to train a robot to sort different types of objects into relevant groups (pick and place) via simulation. The task was successfully transferred from simulation to real world on the Franka-Emika Panda robot. The advantage of training the robot via simulation is that it fast-tracks data collection and reduces any risk of damage to a robot or its environment.
PhD Researcher Jesse Haviland developed a visual servoing method that switches between using a depth camera, which provides 3D distances for every pixel, and a standard colour camera for robot manipulation. For robots with ‘eye-in-hand’ configuration (a camera attached to the end of a robotic arm), one problem is that a depth camera operates ‘blindly’ at distances less than approximately 20cm. Jesse proposed a more robust visual servoing method for object grasping by switching from a depth camera to a standard camera once a robot reaches the blind zone.
Visiting academic Professor Andrea Cherubini spent a year at the Centre’s QUT node, in 2018, working on deformable object manipulation. He collaborated with Centre Director Peter Corke, Research Fellow Akansel Cosgun and PhD Researcher Robert Lee to investigate how robots can learn to shape kinetic sand using a tool to push the sand. The results are included in a paper Model-free vision-based shaping of deformable plastic materials submitted for publication in 2019.
Activity Plan for 2020
- Propose a dataset of 2,000+ 3D object models that will be methodically generated to be diverse in shape complexity and grasping difficulty which would make it a universal benchmarking tool for robotic grasping research.
- Develop a mobile manipulator or prototype ‘robotic butler’ that can perform a range of real-world tasks (primarily domestic chores).
- Develop a system that will allow a robot to interact with humans, handing over a range of everyday objects.
- Develop methods for robots to successfully (and correctly) place objects on a flat surface such as a table or shelf. For example, placing a bottle of water, upright, and not on its side. This forms part of the project’s grasping with intent work.
