# Loong
**Repository Path**: traveller2001/Loong
## Basic Information
- **Project Name**: Loong
- **Description**: No description available
- **Primary Language**: Unknown
- **License**: Apache-2.0
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No
## Statistics
- **Stars**: 0
- **Forks**: 0
- **Created**: 2024-07-11
- **Last Updated**: 2024-07-11
## Categories & Tags
**Categories**: Uncategorized
**Tags**: None
## README
Loong: Benchmarking Long-Context LLMs with Extended Multi-Doc QA
📃 
• 🔔 
## 👀Overview
This repository contains code for our paper [Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA](https://arxiv.org/abs/2406.17419). We propose a novel long-context benchmark, 🐉 **Loong**, aligning with realistic scenarios through extended multi-document question answering (QA). Loong typically consists of 11 documents per test instance on average, spanning three real-world scenarios in English and Chinese: (1) *Financial Reports*, (2) *Legal Cases*, and (3) *Academic Papers*. Meanwhile, Loong introduces new evaluation tasks from the perspectives of *Spotlight Locating*, *Comparison*, *Clustering*, and *Chain of Reasoning*, to facilitate a more realistic and comprehensive evaluation of long-context understanding. Furthermore, Loong features inputs of varying lengths (e.g., *10K-50K*, *50K-100K*, *100K-200K*, *beyond 200K*) and evaluation tasks of diverse difficulty, enabling fine-grained assessment of LLMs across different context lengths and task complexities.
> *Please find more details of this work in our paper.*
> Showcase of four evaluation tasks in Loong (\...\ marks the content of the i-th document). (a) *Spotlight Locating*: Locate the evidence. (b) *Comparison*: Locate and compare the evidence. (c) *Clustering*: Locate and cluster the evidence into groups. (d) *Chain of Reasoning*: Locate and reasoning along a logical chain.
## 📰News
`[2024-07-03]` 🔥The code and benchmark are releasing. If you encounter any issues, please feel free to contact us.
`[2024-06-25]` 👨💻The code is currently being refined, and we plan to release the evaluation code and benchmark within the next one or two weeks. If you encounter any issues, please feel free to contact me at wangminzheng2023@ia.ac.cn.
## 🏆Leaderboard
| Models |
Claimed Length |
Spotlight Locating |
Comparison |
Clustering |
Chain of Reason |
Overall |
| Gemini-1.5-pro |
1000K |
75.02 | 0.56 |
49.94 | 0.27 |
44.10 | 0.09 |
64.97 | 0.37 |
55.37 | 0.27 |
| GPT-4o |
128K |
73.95 | 0.62 |
50.50 | 0.28 |
44.29 | 0.09 |
57.95 | 0.28 |
53.47 | 0.26 |
| Claude3.5-Sonnet |
200K |
58.45 | 0.49 |
54.21 | 0.35 |
45.77 | 0.07 |
43.92 | 0.25 |
48.85 | 0.23 |
| Claude3-Haiku |
200K |
68.68 | 0.59 |
42.10 | 0.21 |
35.04 | 0.02 |
47.59 | 0.17 |
44.88 | 0.19 |
| Qwen2-72B-Instruct (72B) |
128K |
54.17 | 0.36 |
42.38 | 0.20 |
36.71 | 0.04 |
47.76 | 0.18 |
43.29 | 0.15 |
| GLM4-Chat (9B) |
1000K |
57.35 | 0.47 |
40.38 | 0.20 |
28.52 | 0.02 |
39.94 | 0.16 |
38.31 | 0.16 |
| Kimi-Chat |
200K |
60.98 | 0.50 |
34.74 | 0.13 |
28.76 | 0.04 |
38.52 | 0.15 |
37.49 | 0.16 |
> Overall results on four evaluation tasks. For each task, the indicator on the left represents the **_Avg Scores`(0～100)`_**, while the right one represents the **_Perfect Rate`(0~1)`_**.
| Model |
Claimed Length |
Spotlight Locating |
Comparison |
Clustering |
Chain of Reasoning |
Overall |
| Set1 (10K-50K) |
| GPT-4o |
128K |
85.67 | 0.81 |
64.27 | 0.33 |
57.01 | 0.24 |
81.58 | 0.55 |
70.40 | 0.44 |
| Claude3.5-Sonnet |
200K |
60.85 | 0.55 |
69.07 | 0.47 |
58.63 | 0.13 |
68.57 | 0.50 |
63.69 | 0.37 |
| Gemini-1.5-pro |
1000K |
75.00 | 0.60 |
54.88 | 0.28 |
56.15 | 0.23 |
70.64 | 0.37 |
63.36 | 0.34 |
| Qwen2-72B-Instruct |
200K |
68.49 | 0.55 |
60.60 | 0.37 |
47.08 | 0.08 |
70.39 | 0.36 |
60.11 | 0.29 |
| Claude3-Haiku |
200K |
60.94 | 0.55 |
59.97 | 0.40 |
45.53 | 0.04 |
66.85 | 0.34 |
57.14 | 0.28 |
| Kimi-Chat |
200K |
81.11 | 0.74 |
46.70 | 0.20 |
47.84 | 0.07 |
53.77 | 0.17 |
55.02 | 0.24 |
| GLM4-9B-Chat |
1000K |
63.11 | 0.53 |
54.10 | 0.27 |
39.50 | 0.08 |
56.32 | 0.28 |
51.43 | 0.25 |
| Set2 (50K-100K) |
| GPT-4o |
128K |
86.76 | 0.72 |
59.81 | 0.40 |
47.83 | 0.11 |
62.09 | 0.34 |
58.38 | 0.29 |
| Gemini-1.5-pro |
1000K |
76.50 | 0.57 |
54.51 | 0.34 |
44.58 | 0.09 |
64.87 | 0.34 |
55.56 | 0.26 |
| Claude3.5-Sonnet |
200K |
63.83 | 0.53 |
58.90 | 0.39 |
50.96 | 0.10 |
46.09 | 0.26 |
52.73 | 0.24 |
| Qwen2-72B-Instruct |
128K |
64.53 | 0.43 |
42.60 | 0.21 |
38.52 | 0.05 |
51.18 | 0.20 |
45.71 | 0.17 |
| Claude3-Haiku |
200K |
73.71 | 0.66 |
41.90 | 0.22 |
36.18 | 0.02 |
50.20 | 0.15 |
45.45 | 0.17 |
| Kimi-Chat |
200K |
72.82 | 0.52 |
46.77 | 0.21 |
33.46 | 0.06 |
40.51 | 0.15 |
42.40 | 0.16 |
| GLM4-9B-Chat |
1000K |
65.04 | 0.54 |
41.80 | 0.23 |
30.72 | 0.02 |
42.34 | 0.17 |
40.19 | 0.17 |
| Set3 (100K-200K) |
| Gemini-1.5-pro |
1000K |
81.25 | 0.56 |
44.66 | 0.20 |
39.90 | 0.05 |
58.38 | 0.36 |
52.05 | 0.24 |
| GPT-4o |
128K |
74.84 | 0.65 |
42.40 | 0.21 |
38.70 | 0.04 |
45.06 | 0.09 |
46.95 | 0.19 |
| Claude3.5-Sonnet |
200K |
65.36 | 0.56 |
50.32 | 0.34 |
37.79 | 0.03 |
25.95 | 0.11 |
42.06 | 0.19 |
| Claude3-Haiku |
200K |
77.81 | 0.67 |
37.07 | 0.17 |
30.94 | 0.01 |
36.87 | 0.12 |
41.41 | 0.18 |
| Qwen2-72B-Instruct |
128K |
46.99 | 0.27 |
37.06 | 0.13 |
31.50 | 0.02 |
35.01 | 0.07 |
35.94 | 0.09 |
| GLM4-9B-Chat |
1000K |
69.19 | 0.56 |
37.99 | 0.18 |
26.63 | 0.01 |
32.30 | 0.09 |
37.36 | 0.16 |
| Kimi-Chat |
200K |
62.13 | 0.54 |
24.20 | 0.05 |
21.98 | 0.01 |
31.02 | 0.14 |
31.37 | 0.14 |
| Set4 (200K-250K) |
| Gemini-1.5-pro |
1000K |
62.23 | 0.49 |
43.08 | 0.20 |
36.48 | 0.00 |
68.51 | 0.49 |
50.70 | 0.25 |
| Claude3-Haiku |
200K |
53.26 | 0.40 |
27.00 | 0.03 |
25.36 | 0.00 |
28.11 | 0.05 |
32.15 | 0.10 |
| GPT-4o |
128K |
36.79 | 0.19 |
23.97 | 0.08 |
30.40 | 0.00 |
32.89 | 0.07 |
31.11 | 0.07 |
| Claude3.5-Sonnet |
200K |
36.91 | 0.24 |
28.82 | 0.05 |
28.68 | 0.00 |
28.77 | 0.08 |
30.51 | 0.08 |
| Qwen2-72B-Instruct |
128K |
33.18 | 0.16 |
26.59 | 0.08 |
29.84 | 0.01 |
25.81 | 0.04 |
28.92 | 0.06 |
| GLM4-9B-Chat |
1000K |
15.67 | 0.12 |
21.33 | 0.05 |
12.35 | 0.00 |
21.04 | 0.05 |
16.84 | 0.05 |
| Kimi-Chat |
200K |
20.17 | 0.12 |
9.17 | 0.00 |
5.65 | 0.00 |
22.61 | 0.11 |
13.50 | 0.05 |
> The performance of LLMs on four evaluation tasks with different length sets. For each task, the indicator on the left represents the **_Avg Scores`(0～100)`_**, while the right one represents the **_Perfect Rate`(0~1)`_**.
- Following previous work, we prompt GPT-4 as a judge to evaluate the model's output based on the golden answer and the question's requirements from three aspects: *Accuracy*, *Hallucinations*, and *Completeness*, scoring from 0 to 100. For a detailed prompt, please refer to our paper.
- We design two indicators: (1) **_Avg Scores_**: the average value of scores given by GPT-4 for all questions; (2) **_Perfect Rate_**: the proportion of cases scoring 100 out of the total cases. The latter is a more stringent evaluation metric compared to the former.
- We set `temperature = 0` to eliminate randomness and keep other hyper-parameters default. For API-Based LLMs, we directly utilize the official API for testing. Since the Kimi-Chat-200k currently does not provide an interface, we manually input content on the web. As for open-source models, we conduct experiments on a server with 8 $\times$ A100 80GB.
## 🔧Evaluate long-context LLMs
**Step1** Download Loong benchmark
```shell
git clone https://github.com/MozerWang/Loong.git
cd Loong
```
**Step2** Create a conda environment and Install other dependencies.
```shell
conda create --name loong python=3.9 -y
conda activate loong
pip install -r requirements.txt
```
**Step3** Preparing the Model
1. (**Must**) Set up your OPENAI key in config/models/gpt4.yaml
```shell
api_key: "Your OPENAI key"
```
2. If you are using API-based LLM
```shell
# Firstly, Set up your key in config/models/*.yaml
api_key: "Your API key"
```
3. If you are using Open-sourced LLM
```shell
# We recommend using vLLM. And we use HTTP server that implements OpenAI’s Completions and Chat API.
# We have provided using example for Qwen2 and GLM4. See details in Loong/src/vllm_eample.sh
cd src
sh vllm_example.sh
```
**Step4** Evaluate
```shell
cd src
sh run.sh
```
**Things To Know**
- We provide a complete evaluation process:
`step1_load_data.py` # Data loading
`step2_model_generate.py` # Model generation
`step3_model_evaluate.py` # GPT-4 evaluation
`step4_cal_metric.py` # Result statistics
- For `step2_model_generate.py`, you can design the model generation part yourself, modifying it to use your own model's inference method. Just make sure the input and output interfaces in `src/utils/generate.py` remain consistent:
```shell
# Input
generate(prompts, config, output_path, process_num, tag)
# Output
result = prompt.copy() # for prompt in prompts
result[tag] = response_content # Your LLM's response
with open(output_path, 'a', encoding='utf-8') as fw:
fw.write(json.dumps(result, ensure_ascii=False) + '\n')
```
- In `data/loong.jsonl`, the `level` key means task:
`Spotlight Locating` # level1
`Comparison` # level2
`Clustering` # level3
`Chain of Reasoning` # level4
## Citation
```
@article{wang2024loong,
title={Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA},
author={Minzheng Wang, Longze Chen, Cheng Fu, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang, Fei Huang, Yongbin Li},
year={2024}
journal={arXiv preprint arXiv:2406.17419},
}
```