Paper: (https://arxiv.org/pdf/2310.07602.pdf)
# News [2023.10.29] We have released the dataset download [link](#downloadlink). [2023.10.27] Our Code currently supports some baselines including Voxel RCNN, Second, Pointpillars, RDIOU. Other baselines will be updated soon. [2023.10.15] Our code and data are still being maintained and will be released soon. # 1. IntroductionDual-Radar is a new dataset based on 4D radar that can be used for studies on 3D object detection and tracking in the field of autonomous driving. The perception system of ego vehicle includes a high-resolution camera, a 80-line LiDAR and two up-to-date and different models of 4D radars operating in different modes(Arbe and ARS548). The dataset comprises of raw data collected from ego vehicle, including scenarios such as urban and tunnels, with weather conditions of rainy, cloudy, sunny and so on. Our dataset also includes data from different time periods, including dusk, nighttime, and daytime. Our collected raw data amounts to a total of 12.5 hours, encompassing a total distance of over 600 kilometers. Our dataset covers a route distance of approximately 50 kilometers. It consists of 151 continuous time sequences, with the majority being 20-second sequences, resulting in a total of 10,007 carefully time-synchronized frames.
a) Ego vehicle's work scenario |
b) Data projection visualization |
Figure 1. Data collection vehicle and data projection visualization
Figure 2. Sensor Configuration and Coordinate Systems
* The specification of the autonomous vehicle system platform. Our proposed dataset is collected from a high-resolution camera, an 80-line mechanical LiDAR, and two types of 4D radars, the Arbe Phoenix and the ARS548 RDI radar. Our dataset provides GPS information for timing implementation. The sensor configurations are shown in Table 1.Table 1. The specification of the autonomous vehicle system platform
| Sensors | Type | Resolution | Fov | FPS | ||||
| Range | Azimuth | Elevation | Range | Azimuth | Elevation | |||
| camera | acA1920-40uc | - | 1920X | 1200X | - | - | - | 10 |
| LiDAR | RS-Ruby Lite | 0.05m | 0.2° | 0.2° | 230m | 360° | 40° | 10 |
| 4D radar | ARS 548RDI | 0.22m | 1.2°@0...±15° 1.68°@±45° |
2.3° | 300m | ±60° | ±4°@300m ±14°@<140m |
20 |
| Arbe Phoenix | 0.3m | 1.25° | 2° | 153.6m | 100° | 30° | 20 | |
Table 2. The statistics of number of points cloud per frame
| Transducers | Minimum Values | Average Values | Maximum Values |
| LiDAR | 74386 | 116096 | 133538 |
| Arbe Phnoeix | 898 | 11172 | 93721 |
| ARS548 RDI | 243 | 523 | 800 |
a) Sunny,Daytime |
b) Sunny,Nightime |
c) Rainy,Daytime |
d) Cloudy,Daytime |
Figure 3. Visualization of raw data sequences under different weather conditions. On the left is a color RGB image, while on the right side, the cyan represents the Arbe point cloud, the white represents the LiDAR point cloud, and the yellow represents the ARS548 point cloud.
Figure 4. Visualization of 3D bounding box projection on data. The first column represents the 3D frame markers on the image. The Column 2, 3, and 4 represent the point cloud from Lidar, Arbe Phoenix radar point cloud, and ARS548 RDI radar point cloud, respectively. Each row represents a scenario type.(a): downtown daytime normal light. (b): downtown daytime backlight. (c): downtown dusk normal light. (d): downtown dusk backlight. (e): downtown clear night. (f): downtown daytime cloudy. (g): downtown rainy day. (h): downtown cloudy dusk. (i): downtown cloudy night. (j): downtown rainy night. (k): daytime tunnel. (l): nighttime tunnel.
# 3. Download Link * Our dataset is freely available to researchers. Please download and sign our [agreement](https://docs.google.com/document/d/1ZZ_Y_BujAx24jG64u6OyAKtupmZ1QTWN/edit?usp=sharing&ouid=113489683910089362811&rtpof=true&sd=true) and send it to the provided email address (lwang_hit@hotmail.com). You will receive the download link within one week. * When unzipping the data, please file and organize it by following the format below: ``` └─Dual Radar ├─ImageSets.zip ├─testing │ ├─testing_arbe.zip │ ├─testing_ars548.zip │ ├─testing_calib.zip │ ├─testing_image.zip │ ├─testing_label.zip │ ├─testing_robosense.zip ├─training │ ├─training_arbe.zip │ ├─training_ars548.zip │ ├─training_calib.zip │ ├─training_image.zip │ ├─training_label.zip │ ├─training_robosense.zip └─README_dual_radar.txt ``` * This folder contains 10007 frames of labeled pointclouds and image data. The structure of the folder is shown as blow: ``` └─Dual Radar ├─ImageSets │ test.txt │ train.txt │ trainval.txt │ val.txt ├─testing │ ├─arbe │ │ 000000.bin # Raw pointclouds (removed None) of the Arbe. │ │ ............... │ ├─ars548 │ │ 000000.bin # Raw pointclouds (removed None) of the ARS548. │ │ ............... │ ├─calib │ │ 000000.txt │ │ ............... │ ├─image │ │ 000000.png # Undistort images of the camera. │ │ ............... │ ├─label │ │ 000000.txt # Label in txt format, explain later. │ │ ............... │ ├─robosense │ │ 000000.bin # Raw pointclouds (removed None) of the LiDAR. │ │ ............... ├─training │ ├─arbe │ │ 000000.bin │ │ ............... │ ├─ars548 │ │ 000000.bin │ │ ............... │ ├─calib │ │ 000000.txt │ │ ............... │ ├─image │ │ 000000.png │ │ ............... │ ├─label │ │ 000000.txt │ │ ............... │ ├─robosense │ │ 000000.bin │ │ ............... └─README.txt ``` # 4. The Description of Calib Format * The calib.txt contains tree parts. The dataset consists of two parts: the data part and the alignment calibration file. The data part is image data in png format and point cloud data in bin format. The alignment calibration file includes calibration parameters for the four sensors. The camera-LiDAR, camera-4D radar joint calibration are shown here as examples for illustration. ``` Dual Radar_cam.Intrinsics.RadialDistortion: Barrel distortion of Dual Radar_cam [ k1, k2, k3 ] Dual Radar_cam.Intrinsics.TangentialDistortion: radial distortion of Dual Radar_cam [ p1, p2 ] Dual Radar_cam.IntrinsicMatrix: Dual Radar_cam's intrinsic matrix [ af, 0, 0; 0, bf, 0; u, v, 1] Dual Radar_LiDAR-->Dual Radar_cam: Dual Radar_lidar to Dual Radar cam's single response matrix P(4×4) Dual Radar_radar--> Dual Radar_cam: Dual Radar_radar to Dual Radar_cam rotation matrix + translation matrix P(3×4) ``` # 5. Label Files Discription * All values (numerical or strings) are separated via spaces, each row corresponds to one object. The 19 columns represent: ``` Value Name Description ------------------------------------------------------------------------------------------------------- 1 type Describes the type of object: 'Car', 'Van', 'Truck', 'Pedestrian', 'Person_sitting', 'Cyclist', 'Tram', 'Misc' or 'DonCare' 1 truncated Float from 0 (non-truncated) to 1 (truncated), where truncated refers to the object leaving image boundaries 1 occluded Integer (0,1,2,3) indicating occlusion state: 0 = fully visible, 1 = partly ccluded, 2 = largely occluded, 3 = unknown 1 alpha Observation angle of object, ranging [-pi..pi] 4 bbox 2D bounding box of object in the image (0-based index): contains left, top, right, bottom pixel coordinates. 3 dimensions 3D object dimensions: height, width, length (in meters). 3 location 3D object location x,y,z in camera coordinates (in meters). 1 rotation_y Rotation ry around Y-axis in camera coordinates [-pi..pi]. 1 score Only for results: Float,indicating confidence in detection, needed for p/r curves , higher is better. 1 track_id Path tracking of the same object ``` * Since the labeling work is done in label coordinate, the bounding box out of the image FOV(1920×1080) needs to be cut. * Location mean the xyz in label coordinate. The same coordinate origen and the relation of axis is shown below.
Figure 5. Illustration of sensor coordinate systems
* The difference between rotation_y and alpha are, that rotation_y is directly given in camera coordinates, while alpha also considers the vector from the camera center to the object center, to compute the relative orientation of the object with respect to the camera. For example, a car which is facing along the X-axis of the camera coordinate system corresponds to rotation_y=0, no matter where it is located in the X/Z plane (bird's eye view), while alpha is zero only, when this object is located along the Z-axis of the camera. When moving the car away from the Z-axis, the observation angle will change. # 6. Data Statistics
Figure 6. Distribution of weather conditions.
* We separately count the number of instances for each category in the Dual-Radar dataset and the distribution of different types of weather. About two-thirds of our data are collected under normal weather conditions, and about one-third are collected under rainy and cloudy conditions. We collect 577 frames in rainy weather, which is about 5.5% of the total dataset. The rainy weather data we collect can be used to test the performance of different 4D radars in adverse weather conditions.
Figure 7. Distribution of instance conditions.
* We also conduct a statistical analysis of the number of objects with each label at different distance ranges from our vehicle, as shown in Figure 7. Most objects are within 60 meters of our ego vehicle. # 7. Getting Started ### Environment This is the documentation for how to use our detection frameworks with Dual-Radar dataset. We test the Dual-Radar detection frameworks on the following environment: * Python 3.8.16 (3.10+ does not support open3d.) * Ubuntu 18.04/20.04 * Torch 1.10.1+cu113 * CUDA 11.3 * opencv 4.2.0.32 ### Preparing The Dataset * After all files are downloaded, please arrange the workspace directory with the following structure: * Organize your code structure as follows ``` Frameworks ├── checkpoints ├── data ├── docs ├── pcdet ├── output ``` * Organize the dataset according to the following file structure ``` dual_radar ├── lidar ├── ImageSets ├── train.txt ├── trainval.txt ├── val.txt ├── test.txt ├── training ├── calib ├── image ├── label ├── velodyne ├── testing ├── calib ├── image ├── velodyne ├── radar_arbe ├── ImageSets ├── train.txt ├── trainval.txt ├── val.txt ├── test.txt ├── training ├── calib ├── image ├── label ├── arbe ├── testing ├── calib ├── image ├── arbe ├── radar_ars548 ├── ImageSets ├── train.txt ├── trainval.txt ├── val.txt ├── test.txt ├── training ├── calib ├── image ├── label ├── ars548 ├── testing ├── calib ├── image ├── ars548 ``` ### Requirements * Clone the repository ``` git clone https://github.com/adept-thu/Dual-Radar.git cd Dual-Radar ``` * Create a conda environment ``` conda create -n Dual-Radardet python=3.8.16 conda activate Dual-Radardet ``` * Install PyTorch (We recommend pytorch 1.10.1.) * Install the dependencies ``` pip install -r requirements.txt ``` * Install Spconv（our cuda version is 113） ``` pip install spconv-cu113 ``` * Build packages for Dual-Radardet ``` python setup.py develop ``` ### Train & Evaluation * Generate the data infos by running the following command: ``` using lidar data python -m pcdet.datasets.dual_radar.dual_radar_dataset create_dual_radar_infos tools/cfgs/dataset_configs/dual_radar_dataset.yaml using arbe data python -m pcdet.datasets.dual_radar.dual_radar_dataset_arbe create_dual_radar_infos tools/cfgs/dataset_configs/dual_radar_dataset_arbe.yaml using ars548 data python -m pcdet.datasets.dual_radar.dual_radar_dataset_ars548 create_dual_radar_infos tools/cfgs/dataset_configs/dual_radar_dataset_ars548.yaml ``` * To train the model on single GPU, prepare the total dataset and run ``` python train.py --cfg_file ${CONFIG_FILE} ``` * To train the model on multi-GPUs, prepare the total dataset and run ``` sh scripts/dist_train.sh ${NUM_GPUS} --cfg_file ${CONFIG_FILE} ``` * To evaluate the model on single GPU, modify the path and run ``` python test.py --cfg_file ${CONFIG_FILE} --batch_size ${BATCH_SIZE} --ckpt ${CKPT} ``` * To evaluate the model on multi-GPUs, modify the path and run ``` sh scripts/dist_test.sh ${NUM_GPUS} \ --cfg_file ${CONFIG_FILE} --batch_size ${BATCH_SIZE} ``` ### Quick Demo Here we provide a quick demo to test a pretrained model on the custom point cloud data and visualize the predicted results * Download the pretrained model as shown in Table 4~8. * Make sure you have installed the Open3d and mayavi visualization tools. If not, you could install it as follow: ``` pip install open3d pip install mayavi ``` * prepare your point cloud data ``` points[:, 3] = 0 np.save(`my_data.npy`, points) ``` * Run the demo with a pretrained model and point cloud data as follows ``` python demo.py --cfg_file ${CONFIG_FILE} \ --ckpt ${CKPT} \ --data_path ${POINT_CLOUD_DATA} ``` # 8. Experimental ResultsTable 3. Multi-modal experimental results(3D@0.5 0.25 0.25)
| Baseline | Data | Car | Pedestrain | Cyclist | model pth | ||||||
| 3D@0.5 | 3D@0.25 | 3D@0.25 | |||||||||
| Easy | Mod. | Hard | Easy | Mod. | Hard | Easy | Mod. | Hard | |||
| VFF | camera+LiDAR | 94.60 | 84.14 | 78.77 | 39.79 | 35.99 | 36.54 | 55.87 | 51.55 | 51.00 | model |
| camera+Arbe | 31.83 | 14.43 | 11.30 | 0.01 | 0.01 | 0.01 | 0.20 | 0.07 | 0.08 | model | |
| camera+ARS548 | 12.60 | 6.53 | 4.51 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | model | |
| M2-Fusion | LiDAR+Arbe | 89.71 | 79.70 | 64.32 | 27.79 | 20.41 | 19.58 | 41.85 | 36.20 | 35.14 | model |
| LiDAR+ARS548 | 89.91 | 78.17 | 62.37 | 34.28 | 29.89 | 29.17 | 42.42 | 40.92 | 39.98 | model | |
Table 4. Multi-modal experimental results(BEV@0.5 0.25 0.25)
| Baseline | Data | Car | Pedestrain | Cyclist | model pth | ||||||
| BEV@0.5 | BEV@0.25 | BEV@0.25 | |||||||||
| Easy | Mod. | Hard | Easy | Mod. | Hard | Easy | Mod. | Hard | |||
| VFF | camera+Lidar | 94.60 | 84.28 | 80.55 | 40.32 | 36.59 | 37.28 | 55.87 | 51.55 | 51.00 | model |
| camera+Arbe | 36.09 | 17.20 | 13.23 | 0.01 | 0.01 | 0.01 | 0.20 | 0.08 | 0.08 | model | |
| camera+ARS548 | 16.34 | 9.58 | 6.61 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | model | |
| M2-Fusion | LiDAR+Arbe | 90.91 | 85.73 | 70.16 | 28.05 | 20.68 | 20.47 | 53.06 | 47.83 | 46.32 | model |
| LiDAR+ARS548 | 91.14 | 82.57 | 66.65 | 34.98 | 30.28 | 29.92 | 43.12 | 41.57 | 40.29 | model | |
Table 5. Single-modal experimental results(3D@0.5 0.25 0.25)
| Baseline | Data | Car | Pedestrain | Cyclist | model pth | ||||||
| 3D@0.5 | 3D@0.25 | 3D@0.25 | |||||||||
| Easy | Mod. | Hard | Easy | Mod. | Hard | Easy | Mod. | Hard | |||
| Pointpillars | LiDAR | 81.78 | 55.40 | 44.53 | 43.22 | 38.87 | 38.45 | 25.60 | 24.35 | 23.97 | model |
| Arbe | 49.06 | 27.64 | 18.63 | 0.00 | 0.00 | 0.00 | 0.19 | 0.12 | 0.12 | model | |
| ARS548 | 11.94 | 6.12 | 3.76 | 0.00 | 0.00 | 0.00 | 0.99 | 0.63 | 0.58 | model | |
| RDIou | LiDAR | 63.43 | 40.80 | 32.92 | 33.71 | 29.35 | 28.96 | 38.26 | 35.62 | 35.02 | model |
| Arbe | 51.49 | 26.74 | 17.83 | 0.00 | 0.00 | 0.00 | 0.51 | 0.37 | 0.35 | model | |
| ARS548 | 5.96 | 3.77 | 2.29 | 0.00 | 0.00 | 0.00 | 0.21 | 0.15 | 0.15 | model | |
| VoxelRCNN | LiDAR | 86.41 | 56.91 | 42.38 | 52.65 | 46.33 | 45.80 | 38.89 | 35.13 | 34.52 | model |
| Arbe | 55.47 | 30.17 | 19.82 | 0.03 | 0.02 | 0.02 | 0.15 | 0.06 | 0.06 | model | |
| ARS548 | 18.37 | 8.24 | 4.97 | 0.00 | 0.00 | 0.00 | 0.24 | 0.21 | 0.21 | model | |
| Cas-V | LiDAR | 80.60 | 58.98 | 49.83 | 55.43 | 49.11 | 48.47 | 42.84 | 40.32 | 39.09 | model |
| Arbe | 27.96 | 10.27 | 6.21 | 0.02 | 0.01 | 0.01 | 0.05 | 0.04 | 0.04 | model | |
| ARS548 | 7.71 | 3.05 | 1.86 | 0.00 | 0.00 | 0.00 | 0.08 | 0.06 | 0.06 | model | |
| Cas-T | LiDAR | 73.41 | 45.74 | 35.09 | 58.84 | 52.08 | 51.45 | 35.42 | 33.78 | 33.36 | model |
| Arbe | 14.15 | 6.38 | 4.27 | 0.00 | 0.00 | 0.00 | 0.09 | 0.06 | 0.05 | model | |
| ARS548 | 3.16 | 1.60 | 1.00 | 0.00 | 0.00 | 0.00 | 0.36 | 0.20 | 0.20 | model | |
Table 6. Single-modal experimental results(BEV@0.5 0.25 0.25)
| Baseline | Data | Car | Pedestrain | Cyclist | model pth | ||||||
| BEV@0.5 | BEV@0.25 | BEV@0.25 | |||||||||
| Easy | Mod. | Hard | Easy | Mod. | Hard | Easy | Mod. | Hard | |||
| Pointpillars | LiDAR | 81.81 | 55.49 | 45.69 | 43.60 | 39.59 | 38.92 | 38.78 | 38.74 | 38.42 | model |
| Arbe | 54.63 | 35.09 | 25.19 | 0.00 | 0.00 | 0.00 | 0.41 | 0.24 | 0.23 | model | |
| ARS548 | 14.40 | 8.14 | 5.26 | 0.00 | 0.00 | 0.00 | 2.27 | 1.64 | 1.53 | model | |
| RDIou | LiDAR | 63.44 | 41.25 | 33.74 | 33.97 | 29.62 | 29.22 | 49.33 | 47.48 | 46.85 | model |
| Arbe | 55.27 | 31.48 | 21.80 | 0.01 | 0.01 | 0.01 | 0.84 | 0.66 | 0.65 | model | |
| ARS548 | 7.13 | 5.00 | 3.21 | 0.00 | 0.00 | 0.00 | 0.61 | 0.46 | 0.44 | model | |
| VoxelRCNN | LiDAR | 86.41 | 56.95 | 42.43 | 41.21 | 53.50 | 45.93 | 47.47 | 45.43 | 43.85 | model |
| Arbe | 59.32 | 34.86 | 23.77 | 0.02 | 0.02 | 0.02 | 0.21 | 0.15 | 0.15 | model | |
| ARS548 | 21.34 | 9.81 | 6.11 | 0.00 | 0.00 | 0.00 | 0.33 | 0.30 | 0.30 | model | |
| Cas-V | LiDAR | 80.60 | 59.12 | 51.17 | 55.66 | 49.35 | 48.72 | 51.51 | 50.03 | 49.35 | model |
| Arbe | 30.52 | 12.28 | 7.82 | 0.02 | 0.02 | 0.02 | 0.13 | 0.05 | 0.05 | model | |
| ARS548 | 8.81 | 3.74 | 2.38 | 0.00 | 0.00 | 0.00 | 0.25 | 0.21 | 0.19 | model | |
| Cas-T | LiDAR | 73.42 | 45.79 | 35.31 | 59.06 | 52.36 | 51.74 | 44.35 | 44.41 | 42.88 | model |
| Arbe | 22.85 | 13.06 | 9.18 | 0.00 | 0.00 | 0.00 | 0.17 | 0.08 | 0.08 | model | |
| ARS548 | 4.21 | 2.21 | 1.49 | 0.00 | 0.00 | 0.00 | 0.68 | 0.43 | 0.42 | model | |
Table 7. Single-modal experimental results in the rainy scenario(3D@0.5 0.25 0.25)
| Baseline | Data | Car | Pedestrain | Cyclist | model pth | ||||||
| 3D@0.5 | 3D@0.25 | 3D@0.25 | |||||||||
| Easy | Mod. | Hard | Easy | Mod. | Hard | Easy | Mod. | Hard | |||
| Pointpillars | LiDAR | 60.57 | 44.31 | 41.91 | 32.74 | 28.82 | 28.67 | 29.12 | 25.75 | 24.24 | model |
| Arbe | 68.24 | 48.98 | 42.80 | 0.00 | 0.00 | 0.00 | 0.19 | 0.10 | 0.09 | model | |
| ARS548 | 11.87 | 8.41 | 7.32 | 0.11 | 0.09 | 0.08 | 0.93 | 0.36 | 0.30 | model | |
| RDIou | LiDAR | 44.93 | 39.32 | 39.09 | 24.28 | 21.63 | 21.43 | 52.64 | 43.92 | 42.04 | model |
| Arbe | 67.81 | 49.59 | 43.24 | 0.00 | 0.00 | 0.00 | 0.38 | 0.30 | 0.28 | model | |
| ARS548 | 5.87 | 5.48 | 4.68 | 0.00 | 0.00 | 0.00 | 0.09 | 0.01 | 0.01 | model | |
Table 8. Single-modal experimental results(BEV@0.5 0.25 0.25) in the rainy scenario
| Baseline | Data | Car | Pedestrain | Cyclist | model pth | ||||||
| BEV@0.5 | BEV@0.25 | BEV@0.25 | |||||||||
| Easy | Mod. | Hard | Easy | Mod. | Hard | Easy | Mod. | Hard | |||
| Pointpillars | LiDAR | 60.57 | 44.56 | 42.49 | 32.74 | 28.82 | 28.67 | 44.39 | 40.36 | 38.64 | model |
| Arbe | 74.50 | 59.68 | 54.34 | 0.00 | 0.00 | 0.00 | 0.32 | 0.16 | 0.15 | model | |
| ARS548 | 14.16 | 11.32 | 9.82 | 0.11 | 0.09 | 0.08 | 2.26 | 1.43 | 1.20 | model | |
| RDIou | LiDAR | 44.93 | 39.39 | 39.86 | 24.28 | 21.63 | 21.43 | 10.80 | 52.44 | 50.28 | model |
| Arbe | 70.09 | 54.17 | 47.64 | 0.00 | 0.00 | 0.00 | 0.63 | 0.45 | 0.45 | model | |
| ARS548 | 6.36 | 6.51 | 5.46 | 0.00 | 0.00 | 0.00 | 0.13 | 0.08 | 0.08 | model | |