代码拉取完成，页面将自动刷新
We provide several profiling tools to benchmark our models.
Download the dataset below or create your own dataset.
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
Profiling your model with profile_throughput.py
python profile_throughput.py \
ShareGPT_V3_unfiltered_cleaned_split.json \
/path/to/your/model \
--concurrency 64
profile_generation.py perform benchmark with dummy data.
pip install nvidia-ml-py
python profile_generation.py \
/path/to/your/model \
--concurrency 1 8 --prompt-tokens 1 512 --completion-tokens 2048 512
Tools above profile models with Python API. profile_serving.py is used to do benchmark on serving.
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
python profile_serving.py \
${TritonServerAddress} \
/path/to/tokenizer \ # ends with .model for most models. Otherwise, please pass model_path/triton_models/tokenizer.
ShareGPT_V3_unfiltered_cleaned_split.json \
--concurrency 64
profile_restful_api.py is used to do benchmark on api server.
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
python profile_restful_api.py \
${ServerAddress} \
/path/to/tokenizer \ # ends with .model for most models. Otherwise, please pass model_path/triton_models/tokenizer.
ShareGPT_V3_unfiltered_cleaned_split.json \
--concurrency 64
此处可能存在不合适展示的内容，页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容，可点击提交进行申诉，我们将尽快为您处理。
马建仓 AI 助手