torch_npu多卡微调大模型报错ERR01005 OPS internal error Exception raised from operator()

一、问题现象（附报错日志上下文）：
通过网盘分享的文件：详细日志
链接: https://pan.baidu.com/s/1DmKzqgdTlclS0ejdxqM_lw?pwd=8q69 提取码: 8q69

ubuntu20.04  Atlas 300V Pro Video  python版本3.10.13 torch-npu2.1.0   torch2.1.0  llama-factory，0.9.1  glm-4-9b-chat-hf

三、测试步骤：
llama界面训练

四、日志信息:
[INFO|trainer.py:2322] 2024-12-25 02:45:52,413 >>   Number of trainable parameters = 23,797,760
  0%|                                                                                                                                                                                                                                                   | 0/9 [00:00<?, ?it/s]..[E OpParamMaker.cpp:273] call aclnnNLLLossBackward failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 997623] 2024-12-25-02:46:02.738.329 Op NLLLossGrad does not has any binary.
        TraceBack (most recent call last):
       Kernel Run failed. opType: 39, NLLLossGrad
       launch failed for NLLLossGrad, errno:561000.

[ERROR] 2024-12-25-02:46:02 (PID:997623, Device:1, RankID:1) ERR01005 OPS internal error
Exception raised from operator() at third_party/op-plugin/op_plugin/ops/base_ops/opapi/NLLLossBackwardKernelNpuOpApi.cpp:36 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x68 (0xffff81396898 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x6c (0xffff8134f2a8 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: <unknown function> + 0xad5004 (0xffff7638a004 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #3: <unknown function> + 0xe27840 (0xffff766dc840 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #4: <unknown function> + 0x56aa70 (0xffff75e1fa70 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #5: <unknown function> + 0x56ae98 (0xffff75e1fe98 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #6: <unknown function> + 0x568d70 (0xffff75e1dd70 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #7: <unknown function> + 0x946ec (0xffff813bd6ec in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #8: <unknown function> + 0x7624 (0xffff8adf0624 in /lib/aarch64-linux-gnu/libpthread.so.0)
frame #9: <unknown function> + 0xd162c (0xffff8ac7462c in /lib/aarch64-linux-gnu/libc.so.6)

[E OpParamMaker.cpp:273] call aclnnNLLLossBackward failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 997622] 2024-12-25-02:46:02.720.474 Op NLLLossGrad does not has any binary.
        TraceBack (most recent call last):
       Kernel Run failed. opType: 39, NLLLossGrad
       launch failed for NLLLossGrad, errno:561000.

[ERROR] 2024-12-25-02:46:02 (PID:997622, Device:0, RankID:0) ERR01005 OPS internal error
Exception raised from operator() at third_party/op-plugin/op_plugin/ops/base_ops/opapi/NLLLossBackwardKernelNpuOpApi.cpp:36 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x68 (0xffff7af94898 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x6c (0xffff7af4d2a8 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: <unknown function> + 0xad5004 (0xffff6ff8b004 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #3: <unknown function> + 0xe27840 (0xffff702dd840 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #4: <unknown function> + 0x56aa70 (0xffff6fa20a70 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #5: <unknown function> + 0x56ae98 (0xffff6fa20e98 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #6: <unknown function> + 0x568d70 (0xffff6fa1ed70 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #7: <unknown function> + 0x946ec (0xffff7afbb6ec in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #8: <unknown function> + 0x7624 (0xffff849ee624 in /lib/aarch64-linux-gnu/libpthread.so.0)
frame #9: <unknown function> + 0xd162c (0xffff8487262c in /lib/aarch64-linux-gnu/libc.so.6)

Traceback (most recent call last):
Traceback (most recent call last):
  File "/app/LLaMA-Factory-0.9.1/src/llamafactory/launcher.py", line 23, in <module>
  File "/app/LLaMA-Factory-0.9.1/src/llamafactory/launcher.py", line 23, in <module>
        launch()launch()

File "/app/LLaMA-Factory-0.9.1/src/llamafactory/launcher.py", line 19, in launch
  File "/app/LLaMA-Factory-0.9.1/src/llamafactory/launcher.py", line 19, in launch
        run_exp()run_exp()

File "/app/LLaMA-Factory-0.9.1/src/llamafactory/train/tuner.py", line 50, in run_exp
  File "/app/LLaMA-Factory-0.9.1/src/llamafactory/train/tuner.py", line 50, in run_exp
        run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)

File "/app/LLaMA-Factory-0.9.1/src/llamafactory/train/sft/workflow.py", line 101, in run_sft
  File "/app/LLaMA-Factory-0.9.1/src/llamafactory/train/sft/workflow.py", line 101, in run_sft
        train_result = trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint)train_result = trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint)

File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 2122, in train
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 2122, in train
    return inner_training_loop(
      File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 2474, in _inner_training_loop
return inner_training_loop(
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 2474, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 3606, in training_step
    tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 3606, in training_step
    self.accelerator.backward(loss, **kwargs)
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/accelerate/accelerator.py", line 2242, in backward
    self.accelerator.backward(loss, **kwargs)
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/accelerate/accelerator.py", line 2242, in backward
    self.scaler.scale(loss).backward(**kwargs)
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/_tensor.py", line 492, in backward
    self.scaler.scale(loss).backward(**kwargs)
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/_tensor.py", line 492, in backward
        torch.autograd.backward(torch.autograd.backward(

File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/autograd/__init__.py", line 251, in backward
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/autograd/__init__.py", line 251, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/autograd/function.py", line 288, in apply
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/autograd/function.py", line 288, in apply
    return user_fn(self, *args)
      File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 288, in backward
return user_fn(self, *args)
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 288, in backward
    torch.autograd.backward(outputs_with_grad, args_with_grad)
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/autograd/__init__.py", line 251, in backward
    torch.autograd.backward(outputs_with_grad, args_with_grad)
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/autograd/__init__.py", line 251, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: The Inner error is reported as above.
 Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, pleace set the environment variable ASCEND_LAUNCH_BLOCKING=1.
[ERROR] 2024-12-25-02:46:02 (PID:997623, Device:1, RankID:1) ERR00005 PTA internal error
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: The Inner error is reported as above.
 Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, pleace set the environment variable ASCEND_LAUNCH_BLOCKING=1.
[ERROR] 2024-12-25-02:46:02 (PID:997622, Device:0, RankID:0) ERR00005 PTA internal error
.[E OpParamMaker.cpp:273] call aclnnNLLLossBackward failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 997624] 2024-12-25-02:46:03.208.266 Op NLLLossGrad does not has any binary.
        TraceBack (most recent call last):
       Kernel Run failed. opType: 39, NLLLossGrad
       launch failed for NLLLossGrad, errno:561000.

[ERROR] 2024-12-25-02:46:03 (PID:997624, Device:2, RankID:2) ERR01005 OPS internal error
Exception raised from operator() at third_party/op-plugin/op_plugin/ops/base_ops/opapi/NLLLossBackwardKernelNpuOpApi.cpp:36 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x68 (0xffff7a072898 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x6c (0xffff7a02b2a8 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: <unknown function> + 0xad5004 (0xffff6f066004 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #3: <unknown function> + 0xe27840 (0xffff6f3b8840 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #4: <unknown function> + 0x56aa70 (0xffff6eafba70 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #5: <unknown function> + 0x56ae98 (0xffff6eafbe98 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #6: <unknown function> + 0x568d70 (0xffff6eaf9d70 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #7: <unknown function> + 0x946ec (0xffff7a0996ec in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #8: <unknown function> + 0x7624 (0xffff83acc624 in /lib/aarch64-linux-gnu/libpthread.so.0)
frame #9: <unknown function> + 0xd162c (0xffff8395062c in /lib/aarch64-linux-gnu/libc.so.6)

Traceback (most recent call last):
  File "/app/LLaMA-Factory-0.9.1/src/llamafactory/launcher.py", line 23, in <module>
    launch()
  File "/app/LLaMA-Factory-0.9.1/src/llamafactory/launcher.py", line 19, in launch
    run_exp()
  File "/app/LLaMA-Factory-0.9.1/src/llamafactory/train/tuner.py", line 50, in run_exp
    run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
  File "/app/LLaMA-Factory-0.9.1/src/llamafactory/train/sft/workflow.py", line 101, in run_sft
    train_result = trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint)
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 2122, in train
    return inner_training_loop(
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 2474, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 3606, in training_step
    self.accelerator.backward(loss, **kwargs)
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/accelerate/accelerator.py", line 2242, in backward
    self.scaler.scale(loss).backward(**kwargs)
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/_tensor.py", line 492, in backward
    torch.autograd.backward(
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/autograd/__init__.py", line 251, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/autograd/function.py", line 288, in apply
    return user_fn(self, *args)
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 288, in backward
    torch.autograd.backward(outputs_with_grad, args_with_grad)
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/autograd/__init__.py", line 251, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: The Inner error is reported as above.
 Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, pleace set the environment variable ASCEND_LAUNCH_BLOCKING=1.
[ERROR] 2024-12-25-02:46:03 (PID:997624, Device:2, RankID:2) ERR00005 PTA internal error
.[E OpParamMaker.cpp:273] call aclnnNLLLossBackward failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 997625] 2024-12-25-02:46:03.540.116 Op NLLLossGrad does not has any binary.
        TraceBack (most recent call last):
       Kernel Run failed. opType: 39, NLLLossGrad
       launch failed for NLLLossGrad, errno:561000.

[ERROR] 2024-12-25-02:46:03 (PID:997625, Device:3, RankID:3) ERR01005 OPS internal error
Exception raised from operator() at third_party/op-plugin/op_plugin/ops/base_ops/opapi/NLLLossBackwardKernelNpuOpApi.cpp:36 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x68 (0xffffa7eed898 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x6c (0xffffa7ea62a8 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: <unknown function> + 0xad5004 (0xffff9cee4004 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #3: <unknown function> + 0xe27840 (0xffff9d236840 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #4: <unknown function> + 0x56aa70 (0xffff9c979a70 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #5: <unknown function> + 0x56ae98 (0xffff9c979e98 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #6: <unknown function> + 0x568d70 (0xffff9c977d70 in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #7: <unknown function> + 0x946ec (0xffffa7f146ec in /app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #8: <unknown function> + 0x7624 (0xffffb1947624 in /lib/aarch64-linux-gnu/libpthread.so.0)
frame #9: <unknown function> + 0xd162c (0xffffb17cb62c in /lib/aarch64-linux-gnu/libc.so.6)

Traceback (most recent call last):
  File "/app/LLaMA-Factory-0.9.1/src/llamafactory/launcher.py", line 23, in <module>
    launch()
  File "/app/LLaMA-Factory-0.9.1/src/llamafactory/launcher.py", line 19, in launch
    run_exp()
  File "/app/LLaMA-Factory-0.9.1/src/llamafactory/train/tuner.py", line 50, in run_exp
    run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
  File "/app/LLaMA-Factory-0.9.1/src/llamafactory/train/sft/workflow.py", line 101, in run_sft
    train_result = trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint)
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 2122, in train
    return inner_training_loop(
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 2474, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/transformers/trainer.py", line 3606, in training_step
    self.accelerator.backward(loss, **kwargs)
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/accelerate/accelerator.py", line 2242, in backward
    self.scaler.scale(loss).backward(**kwargs)
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/_tensor.py", line 492, in backward
    torch.autograd.backward(
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/autograd/__init__.py", line 251, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/autograd/function.py", line 288, in apply
    return user_fn(self, *args)
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 288, in backward
    torch.autograd.backward(outputs_with_grad, args_with_grad)
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/autograd/__init__.py", line 251, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: The Inner error is reported as above.
 Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, pleace set the environment variable ASCEND_LAUNCH_BLOCKING=1.
[ERROR] 2024-12-25-02:46:03 (PID:997625, Device:3, RankID:3) ERR00005 PTA internal error
  0%|                                                                                                                                                                                                                                                   | 0/9 [00:12<?, ?it/s]
[2024-12-25 02:46:14,939] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 997622) of binary: /app/anaconda3/envs/python31013llama/bin/python
Traceback (most recent call last):
  File "/app/anaconda3/envs/python31013llama/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/distributed/run.py", line 806, in main
    run(args)
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/distributed/run.py", line 797, in run
    elastic_launch(
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/app/LLaMA-Factory-0.9.1/src/llamafactory/launcher.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2024-12-25_02:46:14
  host      : hbjj
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 997623)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
  time      : 2024-12-25_02:46:14
  host      : hbjj
  rank      : 2 (local_rank: 2)
  exitcode  : 1 (pid: 997624)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
  time      : 2024-12-25_02:46:14
  host      : hbjj
  rank      : 3 (local_rank: 3)
  exitcode  : 1 (pid: 997625)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-12-25_02:46:14
  host      : hbjj
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 997622)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
Traceback (most recent call last):
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/gradio/queueing.py", line 575, in process_events
    response = await route_utils.call_process_api(
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/gradio/route_utils.py", line 322, in call_process_api
    output = await app.get_blocks().process_api(
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/gradio/blocks.py", line 1935, in process_api
    result = await self.call_function(
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/gradio/blocks.py", line 1532, in call_function
    prediction = await utils.async_iteration(iterator)
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/gradio/utils.py", line 671, in async_iteration
    return await iterator.__anext__()
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/gradio/utils.py", line 664, in __anext__
    return await anyio.to_thread.run_sync(
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2505, in run_sync_in_worker_thread
    return await future
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 1005, in run
    result = context.run(func, *args)
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/gradio/utils.py", line 647, in run_sync_iterator_async
    return next(iterator)
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/gradio/utils.py", line 809, in gen_wrapper
    response = next(iterator)
  File "/app/LLaMA-Factory-0.9.1/src/llamafactory/webui/runner.py", line 343, in run_train
    yield from self._launch(data, do_train=True)
  File "/app/LLaMA-Factory-0.9.1/src/llamafactory/webui/runner.py", line 324, in _launch
    yield from self.monitor()
  File "/app/LLaMA-Factory-0.9.1/src/llamafactory/webui/runner.py", line 397, in monitor
    output_box: self._finalize(lang, finish_info) + "\n\n" + running_log,
  File "/app/LLaMA-Factory-0.9.1/src/llamafactory/webui/runner.py", line 106, in _finalize
    torch_gc()
  File "/app/LLaMA-Factory-0.9.1/src/llamafactory/extras/misc.py", line 228, in torch_gc
    torch.npu.empty_cache()
  File "/app/anaconda3/envs/python31013llama/lib/python3.10/site-packages/torch_npu/npu/memory.py", line 143, in empty_cache
    torch_npu._C._npu_emptyCache()
RuntimeError: npuSynchronizeDevice:torch_npu/csrc/core/npu/NPUStream.cpp:363 NPU error, error code is 507018
[ERROR] 2024-12-25-02:46:21 (PID:996630, Device:0, RankID:-1) ERR00100 PTA call acl api failed
[Error]: The aicpu execution is abnormal. 
        Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
        rtDeviceSynchronize execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 996630] 2024-12-25-02:46:21.663.380 wait for compute device to finish failed, runtime result = 507018.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        TraceBack (most recent call last):

EZ9999: [PID: 997623] 2024-12-25-02:46:02.738.329 Op NLLLossGrad does not has any binary.
TraceBack (most recent call last):
Kernel Run failed. opType: 39, NLLLossGrad
launch failed for NLLLossGrad, errno:561000.
报错显示算子问题，请检测kernels包是否安装

安装了：Ascend-hdk-310p-npu-driver_24.1.rc2_linux-aarch64.run
Ascend-hdk-310p-npu-firmware_7.3.0.1.231.run
Ascend-cann-toolkit_8.0.RC3.alpha003_linux-aarch64.run
Ascend-cann-kernels-310p_8.0.RC3.alpha003_linux-aarch64.run

百度云盘放了详细日志，麻烦看下链接: https://pan.baidu.com/s/1DmKzqgdTlclS0ejdxqM_lw?pwd=8q69 提取码: 8q69

当前推理芯片暂不支持该算子，请使用训练芯片。推理芯片的支持需要等待后续版本。

Ascend/pytorch

内容风险标识

评论 (4)

Ascend/pytorch .gitee-modal { width: 500px !important; }

内容风险标识