1. π« Embodied Task
2. π« Deep Reasoning Model
3. π« Multimodal Scene
4. π« Long-horizon Decision
5. π« Multi-turn Interaction
Embodied-Reasoner
β¨This is the official implementation of paper
Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Tasks
|
1. π« Embodied Task 2. π« Deep Reasoning Model 3. π« Multimodal Scene 4. π« Long-horizon Decision 5. π« Multi-turn Interaction |
|
π€ Hugging Face
| 
Arxiv
| π WebPage
| πΊ Bilibili
> **Long CoT with Diverse Thinking Pattern:** analysis, spatial reasoning, reflection, planning, and verification. These coherent, image-text interleaved trajectories boost its spatial, temporal reasoning capabilities.
> **Iterative Training Pipeline:** A three-stage iterative training pipeline that combines **imitation learning**, **self-exploration tunning**, and **self-correction tunning**.
> **Interactive Evaluation Framework:** 809 test cases across 12 novel scenarios: `
## Performance πΏπΏ
We compare the performance of Embodied-Reasoner against advanced VLMs and visual reasoning models.
- Success Rate (%) measures whether a task is successfully completed.
- Search Efficiency (%) evaluates task efficiencyβmore steps indicate lower efficiency.
- Task Completeness (%) computes the proportion of predicted actions that belong to the set of key actions.
## Examples π π
### Simulator Experiments
Embodied-Reasoner exhibits spontaneous thinking behaviors, e.g., analyzing environmental states (#1,3), reflecting on missed details (#4), reasoning based on the latest observations (#5), and recalling cues for efficient planning (#9). These thoughts remain coherent and logically consistent despite spanning multiple rounds. In contrast, general VLMs lacking thinking abilities struggle with long-horizon interactive tasks and produce unreasonable actions, e.g., forget tasks or repetitive searching.
### Real-World Experiments
To evaluate the generalization of our reasoning model, we design a real-world experiment. Our model rules out the countertop and dining table after two explorations (steps 1,2), ultimately locating the coffee (#7) in the cabinet and placing it in the microwave for heating (#11). However, we observe that OpenAI o3-mini fails to formulate a reasonable plan, heading to the microwave first instead of searching for the coffee.
## QuickStart π―π―
### Training
#### Step 1. Install Requirements
```shell
conda create -n llama-factory python=3.11
conda activate llama-factory
git clone -b embodied-reasoner https://github.com/iGangao/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e ".[torch,metrics]"
pip install wandb accelerate deepspeed importlib-metadata
```
#### Step 2. Data prepare
Please refer to `data/README.md` for checking the details about the format of dataset files.
#### Step 3. Run training scripts
Run the training scripts:
```shell
bash scripts/train.sh
```
### Evaluation
#### Step 1. Install Requirements
```shell
conda create -n embodied-reasoner python=3.9
conda activate embodied-reasoner
pip install -r requirements.txt
```
#### Step 2. Run evaluation scripts
Run the evaluation scripts:
```shell
bash scripts/eval.sh
```
## Task and Trajectory Engine β²β²
You can navigate to the data_engine folder to synthesize tasks and trajectories. Below are the key files within the data_engine:
```plaintext
data_engine/
βββ taskgenerate/ # Item information and room metadata for task generation
β βββ bathrooms/
β βββ bedrooms/
β βββ kitchens/
β βββ living_rooms/
β βββ pick_up_and_put.json
βββ TaskGenerate.py # Task synthesis script
βββ o1StyleGenerate.py # Trajectory synthesis script
βββ o1StyleGenerate_ordered.py # Complex task trajectory synthesis script
βββ vlmCall.py # Script to call the VLM
βββ vlmCallapi_keys.py # Please Set your API keys here
```
#### Step 1. Generate Task
`TaskGenerate.py` can synthesize task templates and corresponding key actions. The generated task-related data will be stored in the `
Below is an example of the JSON file contents:
```json
{
"scene": "FloorPlan1",
"tasktype": "...",
"taskname": "Locate the Apple in the room.",
"trajectory": [
"<...>...",
"<...>...",
"..."
],
"images": [
".../init_observe.png",
"..."
],
"flag": "",
"time": "...",
"task_metadata": {
"..."
}
}
```
- **scene:** the scene where the task is performed.
- **tasktype:** the type of the task.
- **taskname:** the name of the task.
- **trajectory:** reasoning and decision-making content of the trajectory
- **images:** paths to corresponding images (the first image represents the initial state; each subsequent image corresponds to the state after performing each action listed in trajectory).
- **time and flag:** records the generation timestamp and exceptions encountered during trajectory generation.
- **task_metadata:** task information generated during Step 1.
To view our complete trajectory dataset, please visit our Hugging Face Page.
Please refer to `data_endine/README.md` for checking the details about the data engine.
## Citation
If you find our work helpful, feel free to give us a cite.
```
@article{embodied-reasoner,
title = {Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Tasks},
author = {Wenqi Zhang and Mengna Wang and Gangao Liu and Huixin Xu and Yiwei Jiang and Yongliang Shen and Guiyang Hou and Zhe Zheng and Hang Zhang and Xin Li and Weiming Lu and Peng Li and Yueting Zhuang},
journal = {arXiv preprint arXiv:2503.21696},
year = {2025}
}
```
## License
[](LICENSE)
The codebase is licensed under ζ¨ε
°.
## Contact Us
If you have any questions, please contact us by email:
zhangwenqi@zju.edu.cn, lipeng@iscas.ac.cn
## Acknowledgements
Our training code uses [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) and uses the Simulator with [Ai2-THOR](https://github.com/allenai/ai2thor). Thanks for their wonderful works.
