Construction sites are dangerous places, with injuries and some fatalities each year. Trip hazards are an ever-present danger, and represent an interesting practical and scientific challenge.

It’s not feasible for a robot observer to directly manipulate or test every trip hazard in the environment, nor is it safe to watch thousands of humans trip over objects as a training method. The most practical method is to learn from RGB or RGB-D data to detect trip hazards.

The problem is also interesting and challenging because it’s not the object type itself that primarily determines how much of a trip hazard something is, but rather its current affordance. A ladder standing up against a wall is far less of a trip hazard than when lying on the floor – so any learning system must be capable of distinguishing affordance from object class. This research is now being expanded into other domains including mining.

We conducted a comprehensive investigation into the performance characteristics of 11 different deep learning colour and depth fusion approaches, including four fusion and one non fusion approach; using both colour and two types of depth images. The resulting best performing systems were able to achieve a high level of competency in automatically detecting trip hazards in environments they had never seen before.

PhD researcher Sean McMahon’s paper “Multimodal Trip Hazard Affordance Detection on Construction Sites” co-authored with CI Niko Sünderhauf, CI Michael Milford and Research Affiliate Ben Upcroft was accepted to the IEEE Robotics and Automation Letters journal and also presented at the International Conference on Intelligent Robots and Systems (IROS) 2017.