π Project Page β’ π Demo β’ βΆοΈ YouTube β’ πΊ Bilibili π Data β’ π Benchmark β’ π¦ LAMM Models
### Official Repository of [LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark](https://arxiv.org/abs/2306.06687) [](https://www.youtube.com/watch?v=M7XlIe8hhPk) # Updates π [**2023-09-02**] 1. LAMM is accepted by NeurIPS2023 Datasets & Benchmark Track! See you in December! 2. Finetuning data for ScanQA in multiple choice format is available on [huggingface](https://huggingface.co/datasets/openlamm/LAMM_Dataset/blob/main/3D_Instruct/meta_file/VQA_ScanQA_multiplechoice_finetune.json). Please check it out. π [**2023-09-02**] 1. Deepspeed ZeRO stage3, [xformers](https://github.com/facebookresearch/xformers) & [flashattention v2 (V100 not available)](https://github.com/Dao-AILab/flash-attention) are available for efficient training. LightLLM enabled for inference. Training your models on **V100** or **RTX3090**! 2. Checkpoints finetuning LLaMA2 online, please check it out. 3. Our demo is moved to [](https://openxlab.org.cn/apps/detail/LAMM/LAMM), and previous session on Huggingface will be discarded. π [**2023-07-28**] 1. Checkpoints of LAMM on [huggingface](https://huggingface.co/openlamm) updated on new code base. [LAMM Performance](#leaderboard) updated, Please check it out. π [**2023-07-06**] 1. Evaluation code for both 2D and 3D tasks are ready. 2. [3D Benchmark meta files](https://huggingface.co/datasets/openlamm/LAMM_Dataset/tree/main/3D_Benchmark/meta_file) & [2D Instruction images files](https://opendatalab.com/LAMM/download) updated! Missing files and missing keys fixed. Please update accordingly. 3. Update scripts for [demo in command line](./src/cli_demo.py). π [**2023-06-30**] 1. Watch demo video for LAMM at [Youtube](https://www.youtube.com/watch?v=M7XlIe8hhPk) and [Bilibili](https://www.bilibili.com/video/BV1kN411D7kt/?share_source=copy_web&vd_source=ab4c734425ed0114898300f2c037ac0b)! π [**2023-06-20**] 1. [Full paper with Appendix](https://arxiv.org/abs/2306.06687) is online. π [**2023-06-16**] 1. [LAMM dataset](#lamm-dataset) is available for Research community! π [**2023-06-12**] 1. GPT Evaluation part available. 2. Our Paper will release tomorrow. Please stay tuned! π [**2023-06-11**] 1. LAMM code is available for Research community! 2. Try out the [Interactive Demo](https://huggingface.co/spaces/openlamm/LAMM) on Huggingface! (Time to build app depends on the server load) # Demos ## Online Demo For cases of 2D images, we provide an [online demo](https://huggingface.co/spaces/openlamm/LAMM) deployed on huggingface spaces. ``` Due to limitation of hardware capacity, online version only supports LLM of 7B parameters and load pretrained model takes few minutes. ``` [](https://huggingface.co/spaces/openlamm/LAMM) ## CLI Demo We also provide a CLI demo for local test. Point cloud data are required to be in format of `npy`, we suggest to use data from LAMM-Benchmark-3D. ```bash cd ./src python cli_demo.py \ --model lamm_peft \ --vision_type pcl or image \ --encoder_pretrain epcl or clip \ --encoder_ckpt_path $EPCL_CKPT_PATH or '' \ --llm_ckpt_path $LLM_CKPT_PATH \ --delta_ckpt_path $LAMM_CKPT_PATH ``` # Overview Large language models have become a potential pathway toward achieving artificial general intelligence. Recent works on multi-modal large language models have demonstrated their effectiveness in handling visual modalities. In this work, we extend the research of MLLMs to point clouds and present the LAMM-Dataset and LAMM-Benchmark for 2D image and 3D point cloud understanding. We also establish an extensible framework to facilitate the extension of MLLMs to additional modalities. Our main contribution is three-fold: 1) We present the LAMM-Dataset and LAMM-Benchmark, which cover almost all high-level vision tasks for 2D and 3D vision. Extensive experiments validate the effectiveness of our dataset and benchmark. 2) We demonstrate the detailed methods of constructing instruction-tuning datasets and benchmarks for MLLMs, which will enable future research on MLLMs to scale up and extend to other domains, tasks, and modalities faster. 3) We provide a primary but potential MLLM training framework optimized for modalities' extension. We also provide baseline models, comprehensive experimental observations, and analysis to accelerate future research. # LAMM-Dataset LAMM-Dataset is a comprehensive multi-modal instruction tuning dataset, which contains 186K language-image instruction-response pairs, and 10K lanuage-3D instruction-response pairs.In LAMM-Dataset, the instruction-response pairs are gathered from 8 image datasets and 4 point cloud datasets. Here we design four type of multi-modal instruction-response pairs, - C1: n-round daily dialogue focuses on multi-modal daily conversations. - C2: n-round factual knowledge dialogue aims at factual knowledge reasoning. - C3: 1-round detailed description aims to elaborate images and 3D scenes in texts. - C4: 1-round visual task dialogue transfers various vision tasks into instruction-response pairs, aiming at enhancing generalizability towards domain tasks in other modalities. # Download Download LAMM-Dataset from [here](https://opendatalab.com/LAMM/download). If you would like to download the entire LAMM Dataset and LAMM Benchmark, you can do so from the opendatalab website using the provided [LAMM link](https://opendatalab.com/LAMM/download). Here is the table illustrating the correspondence between each Meta file and image collection in the LAMM dataset:| Task | Dataset | LAMM(Zero-Shot) | LAMM(Finetune) | | -------------------------- | -------- | --------------- | -------------- | | Classification **(Acc)** | CIFAR10 | 37.90 | 91.2 | | Object Detection **(Acc)** | VOC2012 | 7.20 | 13.48 | | VQA **(mAP@0.5)** | SQAimage | 49.88 | 74.27 |
| Task | Dataset | SOTA | LAMM (Zero-Shot) | LAMM (Finetune) | | -------------------------------------------- | --------- | ----- | ---------------- | --------------- | | 3D Object Detection **(mAP@0.5)** | ScanNet | 63.2 | 8.2 | 11.89 | | Visual Grounding **(mAP@0.5)** | ScanRefer | 54.59 | Failed | 3.38 | | 3D VQA **(Acc of multiple choice prolblem)** | ScanQA | N/A | 24.90 | 99.89 |
| | [LLaVA](https://github.com/haotian-liu/LLaVA) | [MiniGPT4](https://github.com/Vision-CAIR/MiniGPT-4) | [mPLUG-owl](https://github.com/X-PLUG/mPLUG-Owl) | LAMM | | ----------------- | ----- | -------- | --------- | --------------- | | Binary-Loc Metric | 14.73 | 13.12 | 4.42 | **36.53** | | GPT Metric | 11 | - | - | **89** |
Bold fonts for the best results.
| Task | Dataset | Metric | SOTA | [LLaVA](https://github.com/haotian-liu/LLaVA) | [MiniGPT4](https://github.com/Vision-CAIR/MiniGPT-4) | [mPLUG-owl](https://github.com/X-PLUG/mPLUG-Owl) | LAMM |
| --------------------- | ------------------------------- | ---------- | -------------- | ---------------------------- | --------------------------- | ----------------- | ------------------------------------- |
| Classification | CIFAR10 | Acc β | 99.5 | **60.83** | 46.22 | 42.5 | 37.9 |
| Detection | VOC2012 | mAP β | 97.2 | 1.42 | 0.92 | 0.158 | **7.20** |
| VQA | SQAimage
AI2D | Acc β | 92.53
N/A | 40.5
18.13 | 43.43
Failed | 36.39
19.31 | **49.88**
**20.92** |
| Image Caption | flickr30k | BLEU4 β | 30.1 | **6.65** | 5.1 | 2.74 | 2.56 |
| F-g clasification | UCMerced | Acc β | 100 | **47** | 33.6 | 32.5 | 18.23 |
| Counting | FSC147 | MAE β | 10.79 | 56.2 | Failed | 60.67 | **46.88** |
| OCR | SVT | Word Acc β | 97.9 | **37.78** | 16.97 | 30.39 | 29.14 |
| Facial Classification | CelebA(Smile)
CelebA(Hair) | Acc β | N/A
N/A | Failed
46.42 | **66.36**
43.47 | Failed
40.93 | 57.60
**56.96** |
| Keypoints Detection | LSP | PCK β | 99.5 | Failed | Failed | Failed | Failed |