Motion-triggered 3D imaging with a Time of Flight CMOS SPAD image sensor

Abstract

Time of flight (ToF) imaging has become a prime 3D sensing technology in consumer and industrial applications, favoured for its low mechanical and computational complexity amongst ranging techniques. Despite its advantages, ToF sensors exhibit high data rate and power due to their temporal resolution and fast frame rates. In Internet of Things (IoT) networks, energy and output data efficiency are crucial for sensors to prevent bandwidth saturation of the processing nodes, while scaling the number of connected devices and extending the life of systems on limited energy supplies. 3D imaging applications, such as surveillance, object tracking and classification in industrial robotic automation, feature conditions where the event interval and frequency of events can be far lower than the time spent by the sensor capturing static scenes. ToF cameras lack the embedded intelligence to recognise frames of interest. Temporal contrast vision sensors, on the other hand, efficiently limit the expense of energy and data bandwidth to detect motion by outputting low data rate binary frames. However, while these sensors integrate well with standard intensity imaging, they do not provide 3D image sensing functionality. The goal of this research is to investigate the feasibility of combining vision algorithms with ToF imaging topologies to design a motion-triggered 3D image sensor. A frame differencing vision architecture is leveraged to limit ToF acquisition to short event-driven duty-cycles, reducing the high consumption and data traffic to external processing and shortening the active time of the ToF emitter. The design strategy is founded on hardware resource sharing in the implementation of both motion detection and ranging functions to minimise pixel area scaling. The shot noise-limited light detection provided by single photon avalanche diodes (SPAD), further reinforced by their recent integration in advanced digital CMOS technologies, makes this detector the enabling factor for the design of the first fully digital motion-triggered 3D image sensor, superior in motion contrast sensitivity and noise performance to analogue-based vision sensors. A 128×128 SPAD motion-triggered 3D image sensor was designed in STMicroelectronics 40 nm CMOS process. 40 µm×20 µm pixels integrate two 16-bit time-gated counters to acquire a 6-bin ToF histogram. The counters are repurposed for acquisition of consecutive frames for motion detection by a frame comparison algorithm. An embedded column-parallel processor performs relative frame differencing with an adaptive per pixel threshold. The sensor achieves a 78% power saving and a data rate reduction of 10 times when operating the sensor in motion-triggered 3D imaging at 30 fps, compared to continuous ToF acquisition. High motion sensitivity below 4% contrast is achieved with a 1.39% pixel-to-pixel contrast FPN by shot noise-limited integration and spatial noise filtering over a dynamic range exceeding 60 dB. ToF 3D imaging is accomplished by a 100 fps histogram acquisition triggered by motion in the scene with 1.5 cm depth accuracy over a measured 3 m range.