Radar and Camera Fusion for Object Forecasting in Driving Scenarios

Authors: Albert Budi Christian; Yu-Hsuan Wu; Chih-Yu Lin; Lan-Da Van; Yu-Chee Tseng

Publication Date: January 16, 2023 (Conference held: 19–22 December 2022)

Conference: 2022 IEEE 15th International Symposium on Embedded Multicore/Many-core Systems-on-Chip (MCSoC) Penang, Malaysia

Abstract: This paper proposes a multi-stage sensor fusion architecture that combines camera and radar data for object trajectory prediction in real-world driving scenarios. Following a detect–track–predict paradigm, the system detects road users using camera images and radar point clouds, tracks them through an online tracking method, and predicts future trajectories using a recurrent neural network. A radar association method is designed to extract radar velocity information for each object. Experiments conducted on the nuScenes autonomous driving dataset demonstrate that incorporating radar velocity significantly improves trajectory prediction accuracy, particularly by enhancing the estimation of object bounding box centers.