[V1] Support eagle3 for qwen2 models. · Pull Request !1020 · MindSpore/vllm-mindspore

[v1][Eagle3] Support qwen2 eagle3 for vllm-ms 0.9.1.
1. 测试场景：
[Device] Atlas 3000, 310P3 npu device.
[Dadaset] mt-bench 前80轮数据.
[Target Model] Qwen2.5-7B.
[Draft Model] Eagle3-Qwen2.5-7B.

2. 无投机测试:
测试代码：
```
vllm-mindspore serve "/mnt/models/Qwen2.5-7B-new" --trust_remote_code --tensor_parallel_size=1 --host 0.0.0.0 --port 8001 --max-num-seqs=256 --block-size=128 --gpu-memory-utilization=0.6  --max_model_len=32768 --max-num-batched-tokens=16384
```

```
测试结果
============ Serving Benchmark Result ============
Successful requests:                     80        
Benchmark duration (s):                  470.47    
Total input tokens:                      1577      
Total generated tokens:                  4340      
Request throughput (req/s):              0.17      
Output token throughput (tok/s):         9.22      
Total Token throughput (tok/s):          12.58     
---------------Time to First Token----------------
Mean TTFT (ms):                          119.03    
Median TTFT (ms):                        117.61    
P99 TTFT (ms):                           135.41    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          108.18    
Median TPOT (ms):                        108.10    
P99 TPOT (ms):                           109.13    
---------------Inter-token Latency----------------
Mean ITL (ms):                           108.18    
Median ITL (ms):                         108.10    
P99 ITL (ms):                            109.55    
==================================================
```

3. eagle3投机测试, spec=3:
测试代码：
```
vllm-mindspore serve "/mnt/models/Qwen2.5-7B-new" --trust_remote_code --tensor_parallel_size=1 --host 0.0.0.0 --port 8001 --max-num-seqs=256 --block-size=128 --gpu-memory-utilization=0.6  --max_model_len=32768 --max-num-batched-tokens=16384 \
     --speculative_config '{ "method": "eagle3", "model": "'$EAGEL3DRAFT'", "draft_tensor_parallel_size": 1, "num_speculative_tokens": 3, "max_model_len": 32768 }'
```

```
测试结果
============ Serving Benchmark Result ============
Successful requests:                     80        
Benchmark duration (s):                  303.74    
Total input tokens:                      3897      
Total generated tokens:                  4345      
Request throughput (req/s):              0.26      
Output token throughput (tok/s):         14.30     
Total Token throughput (tok/s):          27.13     
---------------Time to First Token----------------
Mean TTFT (ms):                          162.64    
Median TTFT (ms):                        161.07    
P99 TTFT (ms):                           179.98    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          65.76     
Median TPOT (ms):                        65.12     
P99 TPOT (ms):                           92.65     
---------------Inter-token Latency----------------
Mean ITL (ms):                           151.62    
Median ITL (ms):                         151.41    
P99 ITL (ms):                            155.19    
==================================================
```

---
<!--  Thanks for sending a pull request!  Here are some tips for you:

1) If this is your first time, please read our contributor guidelines: https://gitee.com/mindspore/mindspore/blob/master/CONTRIBUTING.md

2) If you want to contribute your code but don't know who will review and merge, please add label `mindspore-assistant` to the pull request, we will find and do it as soon as possible.
-->

**What type of PR is this?**

/kind feature

**What does this PR do / why do we need it**:

**Which issue(s) this PR fixes**:

Fixes #

**Code review checklist [【代码检视checklist说明】](https://gitee.com/mindspore/community/blob/master/security/code_review_checklist_mechanism.md)**:

+ - [ ] 是否进行返回值校验 (禁止使用void屏蔽安全函数、自研函数返回值，C++标准库函数确认无问题可以屏蔽)
+ - [ ] 是否遵守 ***SOLID原则 / 迪米特法则***
+ - [ ] 是否具备UT测试用例看护 && 测试用例为有效用例 (若新特性无测试用例看护请说明原因)
+ - [ ] 是否为对外接口变更
+ - [ ] 是否涉及官网文档修改

MindSpore/vllm-mindspore .gitee-modal { width: 500px !important; }

搜索帮助

MindSpore/vllm-mindspore