代码拉取完成，页面将自动刷新
This document explains how to benchmark the models supported by TensorRT-LLM on a single GPU, a single node with multiple GPUs or multiple nodes with multiple GPUs.
The benchmark implementation and entrypoint can be found in benchmarks/python/benchmark.py. There are some other scripts in the directory:
benchmarks/python/allowed_configs.py to define configuration for each supported model.benchmarks/python/build.py to build supported models for benchmarking.benchmarks/python/base_benchmark.py to implement the base class for benchmark.benchmarks/python/gpt_benchmark.py to implement benchmark scripts for GPT and GPT-like(LLaMA/OPT/GPT-J/SmoothQuant-GPT) models.benchmarks/python/bert_benchmark.py to implement benchmark scripts for BERT models.benchmarks/python/enc_dec_benchmark.py to implement benchmark scripts for Encoder-Decoder models.Please use help option for detailed usages.
python benchmark.py -h
Take GPT-350M as an example:
python benchmark.py \
-m gpt_350m \
--mode plugin \
--batch_size "1;8;64" \
--input_output_len "60,20;128,20"
Expected outputs:
[BENCHMARK] model_name gpt_350m world_size 1 num_heads 16 num_kv_heads 16 num_layers 24 hidden_size 1024 vocab_size 51200 precision float16 batch_size 1 input_length 60 output_length 20 gpu_peak_mem(gb) 4.2 build_time(s) 25.67 tokens_per_sec 483.54 percentile95(ms) 41.537 percentile99(ms) 42.102 latency(ms) 41.362 compute_cap sm80
[BENCHMARK] model_name gpt_350m world_size 1 num_heads 16 num_kv_heads 16 num_layers 24 hidden_size 1024 vocab_size 51200 precision float16 batch_size 8 input_length 60 output_length 20 gpu_peak_mem(gb) 4.28 build_time(s) 25.67 tokens_per_sec 3477.28 percentile95(ms) 46.129 percentile99(ms) 46.276 latency(ms) 46.013 compute_cap sm80
[BENCHMARK] model_name gpt_350m world_size 1 num_heads 16 num_kv_heads 16 num_layers 24 hidden_size 1024 vocab_size 51200 precision float16 batch_size 64 input_length 60 output_length 20 gpu_peak_mem(gb) 4.8 build_time(s) 25.67 tokens_per_sec 19698.07 percentile95(ms) 65.739 percentile99(ms) 65.906 latency(ms) 64.981 compute_cap sm80
...
Please note that the expected outputs is only for reference, specific performance numbers depend on the GPU you're using.
Take GPT-175B as an example:
mpirun -n 8 python benchmark.py \
-m gpt_175b \
--mode plugin \
--batch_size "1;8;64" \
--input_output_len "60,20;128,20"
此处可能存在不合适展示的内容，页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容，可点击提交进行申诉，我们将尽快为您处理。
马建仓 AI 助手