# II-World **Repository Path**: gangchen1043/II-World ## Basic Information - **Project Name**: II-World - **Description**: No description available - **Primary Language**: Unknown - **License**: Not specified - **Default Branch**: main - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 1 - **Forks**: 0 - **Created**: 2025-08-27 - **Last Updated**: 2025-08-27 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README

# 🌍 $I^2$-World: Intra-Inter Tokenization for Efficient Dynamic 4D Scene Forecasting

arXiv

https://github.com/user-attachments/assets/693ba6d3-46ba-4529-ae3f-dc8a3456e4cc ## 🚀 News * **[2025-06]** **$I^2$-World** is accepted to ICCV 2025. ## 🛠️Environment Install Pytorch 1.13 + CUDA 11.6 ```setup conda create --name ii-world python=3.8 conda activate ii-world pip install torch==1.13.0+cu116 torchvision==0.14.0+cu116 torchaudio==0.13.0 --extra-index-url https://download.pytorch.org/whl/cu116 ``` Install mmdet3d (v1.0.0rc4) related packages and build this project ```setup pip install mmcv-full==1.7.0 -f https://download.openmmlab.com/mmcv/dist/cu117/torch1.13/index.html pip install mmdet==2.28.2 pip install mmsegmentation==0.30.0 pip install mmengine pip install -v -e . ``` Install other dependencies ```setup pip install -r requirements.txt ``` ## 🤗 Model Zoo > We utilize 8 RTX4090 GPUs to train our model. | Method | Dataset | Task | Rec.mIoU (%) | Rec.IoU (%) | Weights | |:------------:|:-----------:|:----:|:------------:|:-----------:|:-----------------------------------------------------------------------------------------------------:| | II-Tokenizer | Occ3D-nus | Rec | 81.1 | 68.1 | [Google-drive](https://drive.google.com/file/d/1JqFEtU4xbhfiOLWktS6n3PM1Q_CZXLO9/view?usp=drive_link) | | | STCOcc-Res | Rec | 24.8 | 32.2 | - | | | Occ3D-Waymo | Rec | 76.3 | 74.6 | - | | II-World | Occ3D-nus | Fore | 38.4 | 49.2 | [Google-drive](https://drive.google.com/file/d/11qFkO-lpdFdfd443eospIRPii9QGSzFD/view?usp=sharing) | | | STCOcc-Res | Fore | 18.9 | 28.8 | - | | | Occ3D-Waymo | Fore | 43.7 | 60.9 | - | ## 📦 Prepare Dataset 1. Download nuScenes from [nuScenes](https://www.nuscenes.org/nuscenes) 2. Download Occ3D-nus from [Occ3D-nus](https://drive.google.com/file/d/1kiXVNSEi3UrNERPMz_CfiJXKkgts_5dY/view?usp=drive_link) 3. (Optional) Download Occ3D-Waymo from [Occ3D-Waymo](https://drive.google.com/drive/folders/13WxRl9Zb_AshEwvD96Uwz8cHjRNrtfQk) and unzip it to the `data/waymo` folder. We only use the validation of Occ3D-Waymo in our experiments. 4. (Optional) Download STCOcc-Res from [STCOcc-Res](https://drive.google.com/file/d/1dXB9mtROLWChycBZlhYIf_JBLshXogBs/view?usp=drive_link) and unzip it to the `data/nuscenes` folder. 5. Download the generated info file from [Google Drive](https://drive.google.com/drive/folders/1nXMyP9FueCuJ-1kFLvFbzvdnaUkwL1L6?usp=drive_link) and unzip it to the `data/nuscenes` folder. These `*pkl` files can be generated by running the `tools/create_data.py` 6. (Optional) Download the visualization car model [Google Drive](https://drive.google.com/file/d/1Uds-14smeKPYJkLC_DhH9ajap_zawfdi/view?usp=drive_link) 7. Organize your folder structure as below: ``` ├── project ├── visualizer/ │ ├── 3d_model.obj/ (optional) ├── ckpts/ │ ├── ii_scene_tokenizer_4f.pth │ ├── ii_generate_world.pth ├── data/ │ ├── nuscenes/ │ │ ├── samples/ │ │ ├── v1.0-trainval/ │ │ ├── gts/ (Occ3D-nus) │ │ ├── stc-results/ (prediction from STCOcc) (optional) │ │ ├── world-nuscenes_infos_train.pkl │ │ ├── world-nuscenes_infos_val.pkl │ ├── waymo(optional)/ │ │ ├── validation/ │ │ ├── cam_infos_vali.pkl/ │ │ ├── waymo_infos_val.pkl/ ``` ## 🎇 Training and Evaluation Train II-Tokenizer with 8GPUs: ```bash bash tools/dist_train.sh configs/scene_tokenizer/ii_scene_tokenizer_4f.py 8 ``` Evaluate II-Tokenizer with 6GPUs: ```bash bash tools/dist_test.sh configs/scene_tokenizer/ii_scene_tokenizer_4f.py TO/CKPTS ``` > [!IMPORTANT] > Before training or evaluating II-World, you should first evaluate the II-Tokenizer to generate the prediction tokens. By default, the II-Tokenizer will save the prediction tokens to `data/nuscenes/save_dir/token_4f` folder. > > You can change the `test_data_config` in the tokenizer config for different datasets. > > When generate the training set prediction tokens, you can set the `ann_file` in `test_data_config` to `world-nuscenes_infos_train.pkl` Train II-World with 8GPUs: ```bash bash tools/dist_train.sh configs/world_model/ii_generate_world.py 8 ``` Evaluate II-World with 6GPUs: ```bash bash tools/dist_test.sh configs/world_model/ii_generate_world.py TO/CKPTS ``` ## 🎥 Visualization We provide a simple visualization to visualize the high-level control (utilize different cmd) of the world generation. ```bash python tools/generate.py configs/world_model/ii_generate_world.py ckpts/ii_generate_world.pth \ --generate_path generate_output --generate_scene_name scene-0564 --generate_frame 12 --task_mode high-level-control ``` Also, you can visualize the generated world with the fine-grained control (utilize different transformation matrix) ```bash python tools/generate.py configs/world_model/ii_generate_world.py ckpts/ii_generate_world.pth \ --generate_path generate_output --generate_scene_name scene-0270 --generate_frame 12 --task_mode generate ``` If you want to visualize the 3D occupancy map, you can set `--save_npz` in the script above, and the generated 3D occupancy npz will be saved in the `generate_output` folder. Utilize the following command to visualize the generated 3D occupancy map: ```bash python tools/vis_occ_3d.py --vis-single-data \PATH/TO/GENERATED/3D_OCCUPANCY.npz --vis-path demo_output ``` More visualization options can be found in the `tools/vis_occ_3d.py` file. ## Acknowledgement Thanks to the following excellent projects: - [OccWorld](https://github.com/wzzheng/OccWorld) - [OccSora](https://github.com/wzzheng/OccSora) - [mmdetection3d](https://github.com/open-mmlab/mmdetection3d)