llama2 7b 分布式多卡推理报错：EZ9999 Op Mul does not has any binary.

一、问题现象（附报错日志上下文）：
使用四张910b卡加载llama2 7B模型报错；
![输入图片说明](https://foruda.gitee.com/images/1706838422530806520/7a27c031_11496407.png "屏幕截图")

二、软件版本:
-- CANN 版本 (e.g., CANN 3.0.x，5.x.x):  7.0.0
--Tensorflow/Pytorch/MindSpore 版本:pytoch 2.1.0
--Python 版本 (e.g., Python 3.7.5): 3.8.5
-- MindStudio版本 (e.g., MindStudio 2.0.0 (beta3)):
--操作系统版本 (e.g., Ubuntu 18.04): centos 7

三、测试步骤：
环境配置：
Ascend-cann-kernels-910b_7.0.0_linux.run； 
Ascend-cann-toolkit_7.0.0_linux-aarch64.run； 
cmake=3.12.0， 
gcc=7.3.1 
使用ascendspeed提供demo测试7b模型，模型初始化报错：
[E OpParamMaker.cpp:286] call aclnnInplaceNormal failed, detail:EZ9999: Inner Error!
EZ9999  Op Mul does not has any binary.
        TraceBack (most recent call last):
        Kernel Run failed. opType: 174, Mul
        launch failed for Mul, errno:561000.

Exception raised from operator() at third_party/op-plugin/op_plugin/ops/base_ops/opapi/NormalKernelNpuOpApi.cpp:36 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x68 (0xffff7fdcd898 in /usr/local/python3.8.5/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x6c (0xffff7fd862a8 in /usr/local/python3.8.5/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #2: <unknown function> + 0x8ffc0c (0xffff76d3fc0c in /usr/local/python3.8.5/lib/python3.8/site-packages/torch_npu/lib/libtorch_npu.so)
frame #3: <unknown function> + 0xb61028 (0xffff76fa1028 in /usr/local/python3.8.5/lib/python3.8/site-packages/torch_npu/lib/libtorch_npu.so)
frame #4: <unknown function> + 0x45cf80 (0xffff7689cf80 in /usr/local/python3.8.5/lib/python3.8/site-packages/torch_npu/lib/libtorch_npu.so)
frame #5: <unknown function> + 0x45d2e8 (0xffff7689d2e8 in /usr/local/python3.8.5/lib/python3.8/site-packages/torch_npu/lib/libtorch_npu.so)
frame #6: <unknown function> + 0x45cd24 (0xffff7689cd24 in /usr/local/python3.8.5/lib/python3.8/site-packages/torch_npu/lib/libtorch_npu.so)
frame #7: <unknown function> + 0x946ec (0xffff7fdf46ec in /usr/local/python3.8.5/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #8: <unknown function> + 0x7d38 (0xffff89867d38 in /lib64/libpthread.so.0)
frame #9: <unknown function> + 0xdf690 (0xffff8968f690 in /lib64/libc.so.6)

四、日志信息:
make: Leaving directory `/tool/Megatron-LM-23.05/megatron/data'
>>> done with dataset index builder. Compilation time: 1.251 seconds
[WARN]operator(),NPUStream.cpp:150:Feature is not supportted and the possible cause is that driver and firmware packages do not match.
[WARN]operator(),NPUStream.cpp:158:Feature is not supportted and the possible cause is that driver and firmware packages do not match.
[E OpParamMaker.cpp:286] call aclnnInplaceNormal failed, detail:EZ9999: Inner Error!
EZ9999  Op Mul does not has any binary.
        TraceBack (most recent call last):
        Kernel Run failed. opType: 174, Mul
        launch failed for Mul, errno:561000.

Exception raised from operator() at third_party/op-plugin/op_plugin/ops/base_ops/opapi/NormalKernelNpuOpApi.cpp:36 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x68 (0xffff7fdcd898 in /usr/local/python3.8.5/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x6c (0xffff7fd862a8 in /usr/local/python3.8.5/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #2: <unknown function> + 0x8ffc0c (0xffff76d3fc0c in /usr/local/python3.8.5/lib/python3.8/site-packages/torch_npu/lib/libtorch_npu.so)
frame #3: <unknown function> + 0xb61028 (0xffff76fa1028 in /usr/local/python3.8.5/lib/python3.8/site-packages/torch_npu/lib/libtorch_npu.so)
frame #4: <unknown function> + 0x45cf80 (0xffff7689cf80 in /usr/local/python3.8.5/lib/python3.8/site-packages/torch_npu/lib/libtorch_npu.so)
frame #5: <unknown function> + 0x45d2e8 (0xffff7689d2e8 in /usr/local/python3.8.5/lib/python3.8/site-packages/torch_npu/lib/libtorch_npu.so)
frame #6: <unknown function> + 0x45cd24 (0xffff7689cd24 in /usr/local/python3.8.5/lib/python3.8/site-packages/torch_npu/lib/libtorch_npu.so)
frame #7: <unknown function> + 0x946ec (0xffff7fdf46ec in /usr/local/python3.8.5/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #8: <unknown function> + 0x7d38 (0xffff89867d38 in /lib64/libpthread.so.0)
frame #9: <unknown function> + 0xdf690 (0xffff8968f690 in /lib64/libc.so.6)

Traceback (most recent call last):
  File "./tasks/inference/inference_llama.py", line 41, in <module>
    model = GPTModel.from_pretrained(
  File "test/ascendspeed/model/module.py", line 457, in from_pretrained
    args.model = get_model(model_provider)
  File "test/ascendspeed/training.py", line 402, in get_model
    model = model_provider_func(
  File "./tasks/inference/inference_llama.py", line 22, in model_provider
    init_model = GPTModel(
  File "test/ascendspeed/model/gpt_model.py", line 94, in __init__
    self.language_model, self._language_model_key = get_language_model(
  File "test/ascendspeed/model/language_model.py", line 91, in get_language_model
    language_model = TransformerLanguageModel(
  File "test/ascendspeed/model/language_model.py", line 421, in __init__
    self.embedding = Embedding(self.hidden_size,
  File "test/ascendspeed/model/language_model.py", line 181, in __init__
    self.word_embeddings = tensor_parallel.VocabParallelEmbedding(
  File "test/ascendspeed/core/tensor_parallel/layers.py", line 220, in __init__
    _initialize_affine_weight_gpu(self.weight, init_method, partition_dim=0, stride=1)
  File "test/ascendspeed/core/tensor_parallel/layers.py", line 112, in _initialize_affine_weight_gpu
    init_method(weight)
  File "test/ascendspeed/core/utils.py", line 177, in init_
    return torch.nn.init.normal_(tensor, mean=0.0, std=sigma)
  File "/usr/local/python3.8.5/lib/python3.8/site-packages/torch/nn/init.py", line 155, in normal_
    return _no_grad_normal_(tensor, mean, std)
  File "/usr/local/python3.8.5/lib/python3.8/site-packages/torch/nn/init.py", line 19, in _no_grad_normal_
    return tensor.normal_(mean, std)
RuntimeError: The Inner error as above.
 ASCEND kernel errors might be asynchronously reported at some other API call, so the stacktrace may not correct.
For getting the stacktrace of OP in PyTorch, consider passing ASCEND_LAUNCH_BLOCKING=1.
[WARN]operator(),NPUStream.cpp:150:Feature is not supportted and the possible cause is that driver and firmware packages do not match.
[WARN]operator(),NPUStream.cpp:158:Feature is not supportted and the possible cause is that driver and firmware packages do not match.
[E OpParamMaker.cpp:286] call aclnnInplaceNormal failed, detail:EZ9999: Inner Error!
EZ9999  Op Mul does not has any binary.
        TraceBack (most recent call last):
        Kernel Run failed. opType: 174, Mul
        launch failed for Mul, errno:561000.

Exception raised from operator() at third_party/op-plugin/op_plugin/ops/base_ops/opapi/NormalKernelNpuOpApi.cpp:36 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x68 (0xffff8af8d898 in /usr/local/python3.8.5/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x6c (0xffff8af462a8 in /usr/local/python3.8.5/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #2: <unknown function> + 0x8ffc0c (0xffff81effc0c in /usr/local/python3.8.5/lib/python3.8/site-packages/torch_npu/lib/libtorch_npu.so)
frame #3: <unknown function> + 0xb61028 (0xffff82161028 in /usr/local/python3.8.5/lib/python3.8/site-packages/torch_npu/lib/libtorch_npu.so)
frame #4: <unknown function> + 0x45cf80 (0xffff81a5cf80 in /usr/local/python3.8.5/lib/python3.8/site-packages/torch_npu/lib/libtorch_npu.so)
frame #5: <unknown function> + 0x45d2e8 (0xffff81a5d2e8 in /usr/local/python3.8.5/lib/python3.8/site-packages/torch_npu/lib/libtorch_npu.so)
frame #6: <unknown function> + 0x45cd24 (0xffff81a5cd24 in /usr/local/python3.8.5/lib/python3.8/site-packages/torch_npu/lib/libtorch_npu.so)
frame #7: <unknown function> + 0x946ec (0xffff8afb46ec in /usr/local/python3.8.5/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #8: <unknown function> + 0x7d38 (0xffff94a27d38 in /lib64/libpthread.so.0)
frame #9: <unknown function> + 0xdf690 (0xffff9484f690 in /lib64/libc.so.6)

Traceback (most recent call last):
  File "./tasks/inference/inference_llama.py", line 41, in <module>
    model = GPTModel.from_pretrained(
  File "test/ascendspeed/model/module.py", line 457, in from_pretrained
    args.model = get_model(model_provider)
  File "test/ascendspeed/training.py", line 402, in get_model
    model = model_provider_func(
  File "./tasks/inference/inference_llama.py", line 22, in model_provider
    init_model = GPTModel(
  File "test/ascendspeed/model/gpt_model.py", line 94, in __init__
    self.language_model, self._language_model_key = get_language_model(
  File "test/ascendspeed/model/language_model.py", line 91, in get_language_model
    language_model = TransformerLanguageModel(
  File "test/ascendspeed/model/language_model.py", line 421, in __init__
    self.embedding = Embedding(self.hidden_size,
  File "test/ascendspeed/model/language_model.py", line 181, in __init__
    self.word_embeddings = tensor_parallel.VocabParallelEmbedding(
  File "test/ascendspeed/core/tensor_parallel/layers.py", line 220, in __init__
    _initialize_affine_weight_gpu(self.weight, init_method, partition_dim=0, stride=1)
  File "test/ascendspeed/core/tensor_parallel/layers.py", line 112, in _initialize_affine_weight_gpu
    init_method(weight)
  File "test/ascendspeed/core/utils.py", line 177, in init_
    return torch.nn.init.normal_(tensor, mean=0.0, std=sigma)
  File "/usr/local/python3.8.5/lib/python3.8/site-packages/torch/nn/init.py", line 155, in normal_
    return _no_grad_normal_(tensor, mean, std)
  File "/usr/local/python3.8.5/lib/python3.8/site-packages/torch/nn/init.py", line 19, in _no_grad_normal_
    return tensor.normal_(mean, std)
RuntimeError: The Inner error as above.
 ASCEND kernel errors might be asynchronously reported at some other API call, so the stacktrace may not correct.
For getting the stacktrace of OP in PyTorch, consider passing ASCEND_LAUNCH_BLOCKING=1.
[WARN]operator(),NPUStream.cpp:150:Feature is not supportted and the possible cause is that driver and firmware packages do not match.
[WARN]operator(),NPUStream.cpp:158:Feature is not supportted and the possible cause is that driver and firmware packages do not match.
[WARN]operator(),NPUStream.cpp:150:Feature is not supportted and the possible cause is that driver and firmware packages do not match.
[WARN]operator(),NPUStream.cpp:158:Feature is not supportted and the possible cause is that driver and firmware packages do not match.
[E OpParamMaker.cpp:286] call aclnnInplaceNormal failed, detail:EZ9999: Inner Error!
EZ9999  Op Mul does not has any binary.
        TraceBack (most recent call last):
        Kernel Run failed. opType: 174, Mul
        launch failed for Mul, errno:561000.

Exception raised from operator() at third_party/op-plugin/op_plugin/ops/base_ops/opapi/NormalKernelNpuOpApi.cpp:36 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x68 (0xffff7b4dd898 in /usr/local/python3.8.5/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x6c (0xffff7b4962a8 in /usr/local/python3.8.5/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #2: <unknown function> + 0x8ffc0c (0xffff7244fc0c in /usr/local/python3.8.5/lib/python3.8/site-packages/torch_npu/lib/libtorch_npu.so)
frame #3: <unknown function> + 0xb61028 (0xffff726b1028 in /usr/local/python3.8.5/lib/python3.8/site-packages/torch_npu/lib/libtorch_npu.so)
frame #4: <unknown function> + 0x45cf80 (0xffff71facf80 in /usr/local/python3.8.5/lib/python3.8/site-packages/torch_npu/lib/libtorch_npu.so)
frame #5: <unknown function> + 0x45d2e8 (0xffff71fad2e8 in /usr/local/python3.8.5/lib/python3.8/site-packages/torch_npu/lib/libtorch_npu.so)
frame #6: <unknown function> + 0x45cd24 (0xffff71facd24 in /usr/local/python3.8.5/lib/python3.8/site-packages/torch_npu/lib/libtorch_npu.so)
frame #7: <unknown function> + 0x946ec (0xffff7b5046ec in /usr/local/python3.8.5/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #8: <unknown function> + 0x7d38 (0xffff84f77d38 in /lib64/libpthread.so.0)
frame #9: <unknown function> + 0xdf690 (0xffff84d9f690 in /lib64/libc.so.6)

Traceback (most recent call last):
  File "./tasks/inference/inference_llama.py", line 41, in <module>
    model = GPTModel.from_pretrained(
  File "test/ascendspeed/model/module.py", line 457, in from_pretrained
    args.model = get_model(model_provider)
  File "test/ascendspeed/training.py", line 402, in get_model
    model = model_provider_func(
  File "./tasks/inference/inference_llama.py", line 22, in model_provider
    init_model = GPTModel(
  File "test/ascendspeed/model/gpt_model.py", line 94, in __init__
    self.language_model, self._language_model_key = get_language_model(
  File "test/ascendspeed/model/language_model.py", line 91, in get_language_model
    language_model = TransformerLanguageModel(
  File "test/ascendspeed/model/language_model.py", line 421, in __init__
    self.embedding = Embedding(self.hidden_size,
  File "test/ascendspeed/model/language_model.py", line 181, in __init__
    self.word_embeddings = tensor_parallel.VocabParallelEmbedding(
  File "test/ascendspeed/core/tensor_parallel/layers.py", line 220, in __init__
    _initialize_affine_weight_gpu(self.weight, init_method, partition_dim=0, stride=1)
  File "test/ascendspeed/core/tensor_parallel/layers.py", line 112, in _initialize_affine_weight_gpu
    init_method(weight)
  File "test/ascendspeed/core/utils.py", line 177, in init_
    return torch.nn.init.normal_(tensor, mean=0.0, std=sigma)
  File "/usr/local/python3.8.5/lib/python3.8/site-packages/torch/nn/init.py", line 155, in normal_
    return _no_grad_normal_(tensor, mean, std)
  File "/usr/local/python3.8.5/lib/python3.8/site-packages/torch/nn/init.py", line 19, in _no_grad_normal_
    return tensor.normal_(mean, std)
RuntimeError: The Inner error as above.
 ASCEND kernel errors might be asynchronously reported at some other API call, so the stacktrace may not correct.
For getting the stacktrace of OP in PyTorch, consider passing ASCEND_LAUNCH_BLOCKING=1.
[E OpParamMaker.cpp:286] call aclnnInplaceNormal failed, detail:EZ9999: Inner Error!
EZ9999  Op Mul does not has any binary.
        TraceBack (most recent call last):
        Kernel Run failed. opType: 174, Mul
        launch failed for Mul, errno:561000.

Exception raised from operator() at third_party/op-plugin/op_plugin/ops/base_ops/opapi/NormalKernelNpuOpApi.cpp:36 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x68 (0xffffa863d898 in /usr/local/python3.8.5/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x6c (0xffffa85f62a8 in /usr/local/python3.8.5/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #2: <unknown function> + 0x8ffc0c (0xffff9f5afc0c in /usr/local/python3.8.5/lib/python3.8/site-packages/torch_npu/lib/libtorch_npu.so)
frame #3: <unknown function> + 0xb61028 (0xffff9f811028 in /usr/local/python3.8.5/lib/python3.8/site-packages/torch_npu/lib/libtorch_npu.so)
frame #4: <unknown function> + 0x45cf80 (0xffff9f10cf80 in /usr/local/python3.8.5/lib/python3.8/site-packages/torch_npu/lib/libtorch_npu.so)
frame #5: <unknown function> + 0x45d2e8 (0xffff9f10d2e8 in /usr/local/python3.8.5/lib/python3.8/site-packages/torch_npu/lib/libtorch_npu.so)
frame #6: <unknown function> + 0x45cd24 (0xffff9f10cd24 in /usr/local/python3.8.5/lib/python3.8/site-packages/torch_npu/lib/libtorch_npu.so)
frame #7: <unknown function> + 0x946ec (0xffffa86646ec in /usr/local/python3.8.5/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #8: <unknown function> + 0x7d38 (0xffffb20d7d38 in /lib64/libpthread.so.0)
frame #9: <unknown function> + 0xdf690 (0xffffb1eff690 in /lib64/libc.so.6)

Traceback (most recent call last):
  File "./tasks/inference/inference_llama.py", line 41, in <module>
    model = GPTModel.from_pretrained(
  File "test/ascendspeed/model/module.py", line 457, in from_pretrained
    args.model = get_model(model_provider)
  File "test/ascendspeed/training.py", line 402, in get_model
    model = model_provider_func(
  File "./tasks/inference/inference_llama.py", line 22, in model_provider
    init_model = GPTModel(
  File "test/ascendspeed/model/gpt_model.py", line 94, in __init__
    self.language_model, self._language_model_key = get_language_model(
  File "test/ascendspeed/model/language_model.py", line 91, in get_language_model
    language_model = TransformerLanguageModel(
  File "test/ascendspeed/model/language_model.py", line 421, in __init__
    self.embedding = Embedding(self.hidden_size,
  File "test/ascendspeed/model/language_model.py", line 181, in __init__
    self.word_embeddings = tensor_parallel.VocabParallelEmbedding(
  File "test/ascendspeed/core/tensor_parallel/layers.py", line 220, in __init__
    _initialize_affine_weight_gpu(self.weight, init_method, partition_dim=0, stride=1)
  File "test/ascendspeed/core/tensor_parallel/layers.py", line 112, in _initialize_affine_weight_gpu
    init_method(weight)
  File "test/ascendspeed/core/utils.py", line 177, in init_
    return torch.nn.init.normal_(tensor, mean=0.0, std=sigma)
  File "/usr/local/python3.8.5/lib/python3.8/site-packages/torch/nn/init.py", line 155, in normal_
    return _no_grad_normal_(tensor, mean, std)
  File "/usr/local/python3.8.5/lib/python3.8/site-packages/torch/nn/init.py", line 19, in _no_grad_normal_
    return tensor.normal_(mean, std)
RuntimeError: The Inner error as above.
 ASCEND kernel errors might be asynchronously reported at some other API call, so the stacktrace may not correct.
For getting the stacktrace of OP in PyTorch, consider passing ASCEND_LAUNCH_BLOCKING=1.
[2024-02-02 01:36:38,787] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 2727) of binary: /usr/local/python3.8.5/bin/python3
Traceback (most recent call last):
  File "/usr/local/python3.8.5/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/python3.8.5/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/python3.8.5/lib/python3.8/site-packages/torch/distributed/launch.py", line 196, in <module>
    main()
  File "/usr/local/python3.8.5/lib/python3.8/site-packages/torch/distributed/launch.py", line 192, in main
    launch(args)
  File "/usr/local/python3.8.5/lib/python3.8/site-packages/torch/distributed/launch.py", line 177, in launch
    run(args)
  File "/usr/local/python3.8.5/lib/python3.8/site-packages/torch/distributed/run.py", line 797, in run
    elastic_launch(
  File "/usr/local/python3.8.5/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/usr/local/python3.8.5/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
请根据自己的运行环境参考以下方式搜集日志信息，如果涉及到算子开发相关的问题，建议也提供UT/ST测试和单算子集成测试相关的日志。

日志提供方式:
将日志打包后作为附件上传。若日志大小超出附件限制，则可上传至外部网盘后提供链接。

获取方法请参考wiki：
https://gitee.com/ascend/modelzoo/wikis/%E5%A6%82%E4%BD%95%E8%8E%B7%E5%8F%96%E6%97%A5%E5%BF%97%E5%92%8C%E8%AE%A1%E7%AE%97%E5%9B%BE?sort_id=4097825

Ascend/MindSpeed-LLM
暂停

内容风险标识

评论 (6)

Ascend/MindSpeed-LLM暂停 .gitee-modal { width: 500px !important; }

内容风险标识