torch.repeat_interleave

一、问题现象（附报错日志上下文）：
index1 = (
        torch.div(
            atom_count_sqr, num_atoms_per_image_expand, rounding_mode="floor"
        )
    ) + index_offset_expand

data_cell = torch.transpose(data.cell, 1, 2)
pbc_offsets = torch.bmm(data_cell, unit_cell_batch)
print(f"1 index1:{index1}")
pbc_offsets_per_atom = torch.repeat_interleave(
    pbc_offsets, num_atoms_per_image_sqr, dim=0
)
print(f"index1:{index1}")
上面2个print，第一个可以打印，第二个报错。
执行torch.repeat_interleave之前可以打印，执行之后不能打印了

二、软件版本:
-- CANN 版本 (e.g., CANN 3.0.x，5.x.x):  8.0.RC3.alpha001
--Tensorflow/Pytorch/MindSpore 版本:
torch                     2.1.0
torch_cluster             1.6.3
torch-geometric           2.6.1
torch-npu                 2.1.0.post3

--Python 版本 (e.g., Python 3.7.5):3.10.14

--操作系统版本 (e.g., Ubuntu 18.04):
(D910_PyTorch2.1) root@bms-d731:/equiformer_v2# cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
(D910_PyTorch2.1) root@bms-d731:/equiformer_v2# uname -r
4.19.36-vhulk1907.1.0.h1665.eulerosv2r8.aarch64

四、日志信息:
index1:tensor([  0,   0,   0,  ..., 182, 182, 182], device='npu:0')
1 index1:tensor([  0,   0,   0,  ..., 182, 182, 182], device='npu:0')
EZ9999: Inner Error!
EZ9999: 2025-06-18-09:45:14.236.730  Kernel task happen error, retCode=0x26, [aicore exception].[FUNC:PreCheckTaskErr][FILE:davinic_kernel_task.cc][LINE:1219]
        TraceBack (most recent call last):
        The error from device(0), serial number is 14, there is an aicore error, core id is 14, error code = 0x40000000, dump info: pc start: 0x10001240003f4000, current: 0x1240003f4a2c, vec error info: 0x1dfe5def, mte error info: 0x89, ifu error info: 0x31dfbeaeff600, ccu error info: 0, cube error info: 0xfb, biu error info: 0, aic error mask: 0x65000200d000288, para base: 0x12c0c055b000, errorStr: The burst length of the mte instruction is incorrect.[FUNC:PrintCoreErrorInfo][FILE:device_error_proc.cc][LINE:532]
        The extend info from device(0), serial number is 14, there is aicore error, core id is 14, aicore int: 0x1, aicore error2: 0, axi clamp ctrl: 0, axi clamp state: 0x1717, biu status0: 0x101e44800000000, biu status1: 0x940002092a0000, clk gate mask: 0x1000, dbg addr: 0, ecc en: 0, mte ccu ecc 1bit error: 0, vector cube ecc 1bit error: 0, run stall: 0x1, dbg data0: 0, dbg data1: 0, dbg data2: 0, dbg data3: 0, dfx data: 0[FUNC:PrintCoreErrorInfo][FILE:device_error_proc.cc][LINE:563]
        The device(0), core list[0-0], error code is:[FUNC:PrintCoreInfoErrMsg][FILE:device_error_proc.cc][LINE:586]
        coreId( 0):      0x40000000    [FUNC:PrintCoreInfoErrMsg][FILE:device_error_proc.cc][LINE:600]
        Aicore kernel execute failed, device_id=0, stream_id=10, report_stream_id=10, task_id=3436, flip_num=0, fault kernel_name=RepeatInterleave_03c0542f2023bc999d5085c3cf4d9b0b_high_performance__kernel0, fault kernel info ext=RepeatInterleave_03c0542f2023bc999d5085c3cf4d9b0b_high_performance__kernel0, program id=41, hash=11668339097575480947.[FUNC:GetError][FILE:stream.cc][LINE:1082]
        [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1082]
        rtStreamSynchronizeWithTimeout execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
        synchronize stream failed, runtime result = 507015[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]

DEVICE[0] PID[60440]:
EXCEPTION STREAM:
  Exception info:TGID=160532, model id=65535, stream id=10, stream phase=3
  Message info[0]:RTS_HWTS: aicore exception, slot_id=18, stream_id=10
    Other info[0]:time=2025-06-18-09:45:14.217.788, function=int_process_hwts_task_exception, line=290, error code=0x26
Traceback (most recent call last):
  File "/equiformer_v2/main_oc20.py", line 139, in <module>
    Runner()(config)
  File "/equiformer_v2/main_oc20.py", line 79, in __call__
    self.task.run()
  File "/fairchem/ocpmodels/tasks/task.py", line 56, in run
    raise e
  File "/fairchem/ocpmodels/tasks/task.py", line 49, in run
    self.trainer.train(
  File "/equiformer_v2/oc20/trainer/forces_trainer_v2.py", line 337, in train
    out = self._forward(batch)
  File "/equiformer_v2/oc20/trainer/forces_trainer_v2.py", line 451, in _forward
    out_energy, out_forces = self.model(batch_list)
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1519, in forward
    else self._run_ddp_forward(*inputs, **kwargs)
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1355, in _run_ddp_forward
    return self.module(*inputs, **kwargs)  # type: ignore[index]
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/fairchem/ocpmodels/common/data_parallel.py", line 64, in forward
    return self.module(
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/fairchem/ocpmodels/common/utils.py", line 132, in cls_method
    return f(self, *args, **kwargs)
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/equiformer_v2/nets/equiformer_v2/equiformer_v2_oc20.py", line 375, in forward
    ) = self.generate_graph(data)
  File "/fairchem/ocpmodels/models/base.py", line 61, in generate_graph
    edge_index, cell_offsets, neighbors = radius_graph_pbc(
  File "/fairchem/ocpmodels/common/utils.py", line 679, in radius_graph_pbc
    print(f"index01:{index01}")
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/_tensor.py", line 934, in __format__
    return object.__format__(self, format_spec)
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/_tensor.py", line 431, in __repr__
    return torch._tensor_str._str(self, tensor_contents=tensor_contents)
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/_tensor_str.py", line 664, in _str
    return _str_intern(self, tensor_contents=tensor_contents)
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/_tensor_str.py", line 595, in _str_intern
    tensor_str = _tensor_str(self, indent)
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/_tensor_str.py", line 347, in _tensor_str
    formatter = _Formatter(get_summarized_data(self) if summarize else self)
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/_tensor_str.py", line 133, in __init__
    value_str = f"{value}"
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/_tensor.py", line 933, in __format__
    return self.item().__format__(format_spec)
RuntimeError: ACL stream synchronize failed.
Traceback (most recent call last):
  File "/equiformer_v2/main_oc20.py", line 139, in <module>
    Runner()(config)
  File "/equiformer_v2/main_oc20.py", line 79, in __call__
    self.task.run()
  File "/fairchem/ocpmodels/tasks/task.py", line 56, in run
    raise e
  File "/fairchem/ocpmodels/tasks/task.py", line 49, in run
    self.trainer.train(
  File "/equiformer_v2/oc20/trainer/forces_trainer_v2.py", line 337, in train
    out = self._forward(batch)
  File "/equiformer_v2/oc20/trainer/forces_trainer_v2.py", line 451, in _forward
    out_energy, out_forces = self.model(batch_list)
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1519, in forward
    else self._run_ddp_forward(*inputs, **kwargs)
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1355, in _run_ddp_forward
    return self.module(*inputs, **kwargs)  # type: ignore[index]
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/fairchem/ocpmodels/common/data_parallel.py", line 64, in forward
    return self.module(
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/fairchem/ocpmodels/common/utils.py", line 132, in cls_method
    return f(self, *args, **kwargs)
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/equiformer_v2/nets/equiformer_v2/equiformer_v2_oc20.py", line 375, in forward
    ) = self.generate_graph(data)
  File "/fairchem/ocpmodels/models/base.py", line 61, in generate_graph
    edge_index, cell_offsets, neighbors = radius_graph_pbc(
  File "/fairchem/ocpmodels/common/utils.py", line 679, in radius_graph_pbc
    print(f"index01:{index01}")
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/_tensor.py", line 934, in __format__
    return object.__format__(self, format_spec)
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/_tensor.py", line 431, in __repr__
    return torch._tensor_str._str(self, tensor_contents=tensor_contents)
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/_tensor_str.py", line 664, in _str
    return _str_intern(self, tensor_contents=tensor_contents)
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/_tensor_str.py", line 595, in _str_intern
    tensor_str = _tensor_str(self, indent)
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/_tensor_str.py", line 347, in _tensor_str
    formatter = _Formatter(get_summarized_data(self) if summarize else self)
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/_tensor_str.py", line 133, in __init__
    value_str = f"{value}"
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/_tensor.py", line 933, in __format__
    return self.item().__format__(format_spec)
RuntimeError: ACL stream synchronize failed.
wandb:
wandb: You can sync this run to the cloud by running:
wandb: wandb sync models/oc20/s2ef/2M/equiformer_v2/N@12_L@6_M@2/bs@32_lr@2e-4_wd@1e-3_epochs@12_warmup-epochs@0.1_g@8/logs/wandb/2025-06-18-09-44-32/wandb/offline-run-20250618_094500-2025-06-18-09-44-32
[W NPUStream.cpp:382] Warning: NPU warning, error code is 507015[Error]:
[Error]: The aicore execution is abnormal.
        Rectify the fault based on the error information in the ascend log.
EZ9999: Inner Error!
EZ9999: 2025-06-18-09:45:14.628.239  Kernel task happen error, retCode=0x26, [aicore exception].[FUNC:PreCheckTaskErr][FILE:davinic_kernel_task.cc][LINE:1219]
        TraceBack (most recent call last):
        Aicore kernel execute failed, device_id=0, stream_id=10, report_stream_id=10, task_id=3436, flip_num=0, fault kernel_name=RepeatInterleave_03c0542f2023bc999d5085c3cf4d9b0b_high_performance__kernel0, fault kernel info ext=RepeatInterleave_03c0542f2023bc999d5085c3cf4d9b0b_high_performance__kernel0, program id=41, hash=11668339097575480947.[FUNC:GetError][FILE:stream.cc][LINE:1082]
        [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1082]
        rtDeviceSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
        wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
 (function npuSynchronizeUsedDevices)
[W NPUStream.cpp:365] Warning: NPU warning, error code is 507015[Error]:
[Error]: The aicore execution is abnormal.
        Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
        rtDeviceSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: 2025-06-18-09:45:14.637.154  wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        TraceBack (most recent call last):
 (function npuSynchronizeDevice)
[W NPUStream.cpp:365] Warning: NPU warning, error code is 507015[Error]:
[Error]: The aicore execution is abnormal.
        Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
        rtDeviceSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: 2025-06-18-09:45:14.638.324  wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        TraceBack (most recent call last):
 (function npuSynchronizeDevice)
[W NPUStream.cpp:365] Warning: NPU warning, error code is 507015[Error]:
[Error]: The aicore execution is abnormal.
        Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
        rtDeviceSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: 2025-06-18-09:45:14.639.121  wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        TraceBack (most recent call last):
 (function npuSynchronizeDevice)
[W NPUStream.cpp:365] Warning: NPU warning, error code is 507015[Error]:
[Error]: The aicore execution is abnormal.
        Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
        rtDeviceSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: 2025-06-18-09:45:14.639.932  wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        TraceBack (most recent call last):
 (function npuSynchronizeDevice)
[W NPUStream.cpp:365] Warning: NPU warning, error code is 507015[Error]:
[Error]: The aicore execution is abnormal.
        Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
        rtDeviceSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: 2025-06-18-09:45:14.640.746  wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        TraceBack (most recent call last):
 (function npuSynchronizeDevice)
[W NPUStream.cpp:365] Warning: NPU warning, error code is 507015[Error]:
[Error]: The aicore execution is abnormal.
        Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
        rtDeviceSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: 2025-06-18-09:45:14.641.520  wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        TraceBack (most recent call last):
 (function npuSynchronizeDevice)
[W NPUStream.cpp:365] Warning: NPU warning, error code is 507015[Error]:
[Error]: The aicore execution is abnormal.
        Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
        rtDeviceSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: 2025-06-18-09:45:14.642.299  wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        TraceBack (most recent call last):
 (function npuSynchronizeDevice)
[W NPUStream.cpp:365] Warning: NPU warning, error code is 507015[Error]:
[Error]: The aicore execution is abnormal.
        Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
        rtDeviceSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: 2025-06-18-09:45:14.643.065  wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        TraceBack (most recent call last):
 (function npuSynchronizeDevice)
[2025-06-18 09:45:27,777] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 60440) of binary: /root/miniconda3/envs/D910_PyTorch2.1/bin/python
Traceback (most recent call last):
  File "/root/miniconda3/envs/D910_PyTorch2.1/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/distributed/run.py", line 806, in main
    run(args)
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/distributed/run.py", line 797, in run
    elastic_launch(
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/root/miniconda3/envs/D910_PyTorch2.1/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
main_oc20.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2025-06-18_09:45:27
  host      : bms-d731
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 60440)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
![输入图片说明](https://foruda.gitee.com/images/1750212936102662595/39b668eb_9147206.png "屏幕截图")

Ascend/pytorch
暂停

内容风险标识

评论 (5)

Ascend/pytorch暂停 .gitee-modal { width: 500px !important; }

内容风险标识