# ComplexFuncBench

**Repository Path**: hf-datasets/ComplexFuncBench

## Basic Information

- **Project Name**: ComplexFuncBench
- **Description**: Mirror of https://huggingface.co/datasets/zai-org/ComplexFuncBench
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2025-12-12
- **Last Updated**: 2025-12-12

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

---
license: apache-2.0
task_categories:
- question-answering
language:
- en
size_categories:
- 1K<n<10K
---

# Introduction
Complex Function Calling Benchmark (`ComplexFuncBench`) is specillly designed for complex function calling evaluation. The ComplexFuncBench dataset encompass 1,000 complex function calling samples from five aspects: (1) Function calling with **multi-step** in single turn; (2) Function calling with user-provided **constraints**; (3) Function calling that requires **parameter value reasoning** from implicit information; (4) Function calling with **long parameter values** that exceed 500 tokens; and (5) Function calling with **128k long-context** length.

If you wish to use this dataset for automated evaluation, please refer to our [github](https://github.com/THUDM/ComplexFuncBench).

Paper: https://huggingface.co/papers/2501.10132

# Leaderboard
| Model                        | Overall Success Rate | Overall Call Acc. | Completeness | Correctness |
| :--------------------------- | :------------------: | :---------------: | :----------: | :---------: |
| Claude-3.5-Sonnet (20241022) |      **61.00**       |       79.27       |   **1.84**   |  **1.85**   |
| GPT-4o (2024-08-06)          |        60.50         |     **80.55**     |     1.66     |    1.75     |
| GLM-4-Long                   |        57.10         |       76.35       |     1.72     |    1.74     |
| GPT-4-Turbo (2024-04-09)     |        49.50         |       71.38       |     1.72     |    1.81     |
| Claude-3.5-Haiku (20241022)  |        45.80         |       69.50       |     1.79     |    1.71     |
| Qwen2.5-72B                  |        40.10         |       58.32       |     1.80     |    1.75     |
| Mistral Large 2              |        20.10         |       48.78       |     0.94     |     1.0     |
| GLM-4-9B                     |         9.40         |       27.97       |     1.15     |    1.03     |
| Qwen2.5-7B                   |         5.0          |       18.19       |     1.5      |    1.47     |
| Llama-3.1-405B               |         4.00         |       11.87       |     0.43     |    0.30     |
| Llama-3.1-70B                |         2.70         |       8.17        |     0.67     |    0.36     |
| Llama-3.1-8B                 |         0.10         |       1.34        |     0.18     |    0.09     |

# Dataset Statistics
|             | Hotels | Flights | Car Rental | Attraction | Cross | Total |
| :---------- | :----: | :-----: | :--------: | :--------: | :---: | :---: |
| Num Samples |  150   |   150   |    150     |    150     |  400  |  600  |
| Avg. Steps  |  3.33  |   3.33  |    2.87    |   2.86     |  3.5  |  3.26 |
| Avg. Calls  |  4.29  |   5.33  |    4.57    |    3.6     |  6.0  |  5.07 |

# Citation
If you find our work helpful for your research, please consider citing our work.
```
@misc{zhong2025complexfuncbench,
      title={ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenario}, 
      author={Lucen Zhong and Zhengxiao Du and Xiaohan Zhang and Haiyi Hu and Jie Tang},
      year={2025},
      eprint={2501.10132},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2501.10132}, 
}
```