Robotic Vision Evaluation and Benchmarking
This project combines the variety and complexity of real-world data with the flexibility of synthetic graphics and physics engines. It puts a focus on evaluating our own algorithms on real robot hardware, under realistic conditions.
Project Leaders
Niko Sünderhauf
Queensland University of Technology (QUT), Australia
Dr Niko Suenderhauf is a Chief Investigator of the Australian Centre for Robotic Vision, and a Senior Lecturer at Queensland University of Technology (QUT) in Brisbane, Australia (a Senior Lecturer is roughly equivalent to a junior Associate Professor in the US system).
Niko conducts research in robotic vision, at the intersection of robotics, computer vision, and machine learning. His research interests focus on scene understanding and how robots can learn to perform complex tasks that require navigation and interaction with objects, the environment, and with humans.
Feras Dayoub
Queensland University of Technology (QUT), Australia
Feras is interested in the reliable deployment of machine learning and computer vision on mobile robots in challenging environments. Autonomous Underwater Vehicles (AUV) as the computer vision lead on the COTSBot project. Unmanned Aerial Vehicles (UAV) as part of a project on assisted autonomy during the inspection of power infrastructure. Mobile service robots as a research fellow on an Australian research council discovery project on lifelong robotic navigation using visual perception.
Team Members
David Hall
Queensland University of Technology (QUT), Australia
David became a Research Fellow with the Centre in April 2108 after submitting his PhD in January 2018. He performed his PhD research at the Queensland University of Technology (QUT) with a focus on autonomous weed recognition systems for agricultural robotics working under Dr Feras Dayoub, Dr Chris McCool and Prof. Tristan Perez.
David’s main research focuses are on making systems which are adaptable to real-world variations and conditions. He is pleased to be working on the Centre project Robotic Vision Evaluation & Benchmarking which will address issues such as adaptable vision and continuous learning.
Haoyang Zhang
Queensland University of Technology (QUT), Australia
Haoyang joined the Centre as a Research Fellow in December 2018 after completing his PhD at Australian National University (ANU) and Data61, CSIRO in July 2018. During his PhD, Haoyang mainly worked with A/Prof. Xuming He on visual object detection and segmentation. His research interests include computer vision and its application to robots.
Suman Bista
Queensland University of Technology (QUT), Australia
Suman Bista joined the Centre in 2017 as a Research Fellow based at QUT. He worked on visual navigation and recognition for the Pepper humanoid robot and was supervised by Centre Director, Professor Peter Corke. His research interests include Visual Navigation, Visual learning, Robotics Vision and Optimisation. In August 2019, Suman joined the Centre’s Robotic Vision Evaluation and Benchmarking Project, continuing as a Research Fellow in this role.
Suman completed his PhD titled “Indoor Navigation of Mobile Robots based on Visual Memory and Image-Based Visual Servoing” in 2016 with the Lagadic Group, INRIA Rennes Bretagne Atlantique, Rennes, France under the supervision of Dr Francois Chaumette and Dr Paolo Robuffo Giordano.
Rohan Smith
Queensland University of Technology (QUT), Australia
Rohan received his Bachelor of Mechatronics from QUT in 2016. He worked on QUT’s Robotronica event in 2015 and was part of the team of postdoctoral research fellows, PhD researchers and undergraduate students working on the winning entry to the Amazon Robotics Challenge in 2017.
Rohan has been working as a research engineer at QUT on the Centre’s Robotic Vision Evaluation and Benchmarking project since mid 2018. He is establishing and maintaining multiple mobile robot platforms for use by Centre researchers. Rohan hopes to make it easier for researchers to develop better ways for robots to interact with the real world.
Ben Talbot
Queensland University of Technology
Ben is a Research Associate based at the Queensland University of Technology (QUT). He joined the Centre in September 2018 after being awarded his PhD in the same month. Ben’s PhD research explored how mobile robots can understand the symbols humans use to navigate (like labels, arrows, signs, gestures, spoken directions, maps, floor plans, etc.), and use this understanding to navigate unseen built environments. His research interests include artificial intelligence, cognitive robotics, and real robot outcomes.
In his role within the Robotic Vision and Evaluation Benchmarking project he is working on establishing multiple mobile robot platforms for use by Centre researchers. By increasing the flexibility, reliability, and ease of access to mobile robots, Ben hopes to help researchers evaluate and demonstrate their latest exciting research outcomes on real robots.
Project Aim
This project will develop new standardised benchmark tasks, evaluation metrics, and a new, annual robotic vision challenge competition as part of the Centre’s legacy. We aim to recreate for robotic vision the positive effects that competitions such as the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) or Common Objects in Context (COCO) competition had for the advances of deep learning and computer vision. The project combines the variety and complexity of real-world data with the flexibility of synthetic graphics and physics engines. It puts a focus on evaluating algorithms developed in other Centre projects on real robot hardware, under realistic conditions, and providing valuable feedback on the robustness of the evaluated algorithms and approaches.
Key Results
The project team released a new robotic vision challenge and competition for probabilistic object detection. This novel task requires participant’s software to detect objects in images, but also quantify the spatial and semantic uncertainty of their detections. To compare the quality of different approaches for this challenging new task, the team developed a new evaluation metric, the probabilistic detection quality measure (PDQ). The task of probabilistic object detection is highly relevant for robots that operate in the real world where they often encounter objects that they were not trained to recognise.
In collaboration with Centre researchers from the University of Adelaide, and long-standing partners from Google AI, the project team introduced a Robotic Vision competition called the Probabilistic Object Detection (PrOD) Challenge to the global research community. A total of 18 international teams participated in the competition, and the four best teams presented their results at a workshop we organised at the Conference on Computer Vision and Pattern Recognition. Our work on the new competition and evaluation metric resulted in conference and journal publications as well as a workshop paper: Probabilistic Object Detection: Definition and Evaluation; A Probabilistic Challenge for Object Detection; Benchmarking Sampling-based Probabilistic Object Detectors.
The project team organised a second workshop on the topic of ‘The Importance of Uncertainty in Deep Learning for Robotics’ at the International Conference on Intelligent Robots and Systems. In tutorial-style talks, we covered the state-of-the-art of uncertainty quantification in deep learning – specifically Bayesian and non-Bayesian approaches, spanning perception, world-modelling, decision making, and actions. Invited expert speakers discussed the importance of uncertainty in deep learning for robotic perception, and action.
The project team worked towards completion of BenchBot, a new software framework that makes it easy to evaluate robotic vision algorithms in simulation and on real robots. BenchBot provides a software interface and API that allows researchers to use exactly the same code to control simulated and real robots. BenchBot interfaces the commonly used Robot Operating System, the new Nvidia Isaac SDK, as well as the Isaac Simulator.
Activity Plan for 2020
Robotic Vision Challenge
- Establish a new competition and research challenge that focuses on robotic Scene Understanding and Semantic Object-based Mapping and SLAM.
- Task competition participants to develop algorithms that enable a robot to explore an indoor environment, such as an apartment or office, and create a map that contains all the objects in the environment. The competition will enable researchers from around the world to compete and compare their algorithms.
- Introduce the new competition to the research community at the IEEE International Conference on Robotics and Automation (ICRA).
BenchBot
- Publicly release BenchBot, our new software framework that makes it easy to evaluate robotic vision algorithms in simulation and on real robots.
- BenchBot is a core enabling part of the new Robotic Vision competition on robotic Scene Understanding and Semantic Object-based Mapping and SLAM to be presented at ICRA.
