NPU error code is:507018，E39999: Inner Error!

一、问题现象（附报错日志上下文）：
 warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", DeprecationWarning)
/home/ma-user/work/Grounded-Segment-Anything-main/GroundingDINO/groundingdino/models/GroundingDINO/ms_deform_attn.py:31: UserWarning: Failed to load custom C++ ops. Running on CPU mode Only!
  warnings.warn("Failed to load custom C++ ops. Running on CPU mode Only!")
/home/ma-user/anaconda3/envs/python-3.7.10/lib/python3.7/site-packages/torch_npu/contrib/transfer_to_npu.py:129: ImportWarning: 
    *************************************************************************************************************
    The torch.Tensor.cuda and torch.nn.Module.cuda are replaced with torch.Tensor.npu and torch.nn.Module.npu now..
    The torch.cuda.DoubleTensor is replaced with torch.npu.FloatTensor cause the double type is not supported now..
    The backend in torch.distributed.init_process_group set to hccl now..
    The torch.cuda.* and torch.cuda.amp.* are replaced with torch.npu.* and torch.npu.amp.* now..
    The device parameters have been replaced with npu in the function below:
    torch.logspace, torch.randint, torch.hann_window, torch.rand, torch.full_like, torch.ones_like, torch.rand_like, torch.randperm, torch.arange, torch.frombuffer, torch.normal, torch._empty_per_channel_affine_quantized, torch.empty_strided, torch.empty_like, torch.scalar_tensor, torch.tril_indices, torch.bartlett_window, torch.ones, torch.sparse_coo_tensor, torch.randn, torch.kaiser_window, torch.tensor, torch.triu_indices, torch.as_tensor, torch.zeros, torch.randint_like, torch.full, torch.eye, torch._sparse_csr_tensor_unsafe, torch.empty, torch._sparse_coo_tensor_unsafe, torch.blackman_window, torch.zeros_like, torch.range, torch.sparse_csr_tensor, torch.randn_like, torch.from_file, torch._cudnn_init_dropout_state, torch._empty_affine_quantized, torch.linspace, torch.hamming_window, torch.empty_quantized, torch._pin_memory, torch.device, torch.Tensor.new_empty, torch.Tensor.new_empty_strided, torch.Tensor.new_full, torch.Tensor.new_ones, torch.Tensor.new_tensor, torch.Tensor.new_zeros, torch.Tensor.to, torch.nn.Module.to, torch.nn.Module.to_empty
    *************************************************************************************************************
    
  warnings.warn(msg, ImportWarning)
True
npu
/home/ma-user/anaconda3/envs/python-3.7.10/lib/python3.7/site-packages/torch/functional.py:568: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at  /usr1/v1.11.0/pytorch/aten/src/ATen/native/TensorShape.cpp:2228.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
final text_encoder_type: bert-base-uncased
Some weights of the model checkpoint at /home/ma-user/work/Grounded-Segment-Anything-main/huggingface were not used when initializing BertModel: ['cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.seq_relationship.weight']
- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
_IncompatibleKeys(missing_keys=[], unexpected_keys=['label_enc.weight'])
tensor([[ True, False,  True,  True]], device='npu:0')
NonZero
Traceback (most recent call last):
  File "grounded_sam_demo.py", line 194, in <module>
    model, image, text_prompt, box_threshold, text_threshold, device=device
  File "grounded_sam_demo.py", line 64, in get_grounding_output
    outputs = model(image[None], captions=[caption])
  File "/home/ma-user/anaconda3/envs/python-3.7.10/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/ma-user/work/Grounded-Segment-Anything-main/GroundingDINO/groundingdino/models/GroundingDINO/groundingdino.py", line 242, in forward
    tokenized, self.specical_tokens, self.tokenizer
  File "/home/ma-user/work/Grounded-Segment-Anything-main/GroundingDINO/groundingdino/models/GroundingDINO/bertwarper.py", line 241, in generate_masks_with_special_tokens_and_transfer_map
    idxs = torch.nonzero(special_tokens_mask)
RuntimeError: Run:/usr1/workspace/FPTA_Daily_open_pytorchv1.11.0-5.0.rc1/CODE/torch_npu/csrc/framework/OpParamMaker.cpp:138 NPU error,NPU error code is:507018
E39999: Inner Error!
E39999  An exception occurred during AICPU execution, stream_id:411, task_id:18, errcode:21008, msg:inner error[FUNC:ProcessAicpuErrorInfo][FILE:device_error_proc.cc][LINE:667]
        Aicpu kernel execute failed, device_id=0, stream_id=411, task_id=18.[FUNC:PrintAicpuErrorInfo][FILE:task.cc][LINE:747]
        Aicpu kernel execute failed, device_id=0, stream_id=411, task_id=18, fault op_name=NonZero[FUNC:GetError][FILE:stream.cc][LINE:846]
        rtStreamSynchronize execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:45]
        Call rtStreamSynchronize(stream) fail, ret: 0x7BC8A[FUNC:LaunchKernel][FILE:op_task.cc][LINE:1403]
        [Exec][Op]Execute op failed, ge result = 507018[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:162]
        An exception occurred during AICPU execution, stream_id:411, task_id:21, errcode:21008, msg:inner error[FUNC:ProcessAicpuErrorInfo][FILE:device_error_proc.cc][LINE:667]
        Aicpu kernel execute failed, device_id=0, stream_id=411, task_id=21.[FUNC:PrintAicpuErrorInfo][FILE:task.cc][LINE:747]
        Aicpu kernel execute failed, device_id=0, stream_id=411, task_id=21, fault op_name=NonZero[FUNC:GetError][FILE:stream.cc][LINE:846]
        An exception occurred during AICPU execution, stream_id:411, task_id:24, errcode:21008, msg:inner error[FUNC:ProcessAicpuErrorInfo][FILE:device_error_proc.cc][LINE:667]
        Aicpu kernel execute failed, device_id=0, stream_id=411, task_id=24.[FUNC:PrintAicpuErrorInfo][FILE:task.cc][LINE:747]
        Aicpu kernel execute failed, device_id=0, stream_id=411, task_id=24, fault op_name=NonZero[FUNC:GetError][FILE:stream.cc][LINE:846]

DEVICE[0] PID[27845]: 
EXCEPTION TASK:
  Exception info:TGID=54778, model id=65535, stream id=411, stream phase=3, task id=18, task type=aicpu kernel, recently received task id=19, recently send task id=17, task phase=RUN
  Message info[0]:aicpu=0,slot_id=0,report_mailbox_flag=0x5a5a5a5a,state=0x5210
    Other info[0]:time=2023-09-08-08:56:43.623.893, function=proc_aicpu_task_done, line=972, error code=0x2a 
EXCEPTION TASK:
  Exception info:TGID=54778, model id=65535, stream id=411, stream phase=3, task id=21, task type=aicpu kernel, recently received task id=22, recently send task id=20, task phase=RUN
  Message info[0]:aicpu=0,slot_id=0,report_mailbox_flag=0x5a5a5a5a,state=0x5210
    Other info[0]:time=2023-09-08-08:56:43.628.653, function=proc_aicpu_task_done, line=972, error code=0x2a 
EXCEPTION TASK:
  Exception info:TGID=54778, model id=65535, stream id=411, stream phase=3, task id=24, task type=aicpu kernel, recently received task id=25, recently send task id=23, task phase=RUN
  Message info[0]:aicpu=0,slot_id=0,report_mailbox_flag=0x5a5a5a5a,state=0x5210
    Other info[0]:time=2023-09-08-08:56:43.632.276, function=proc_aicpu_task_done, line=972, error code=0x2a
THPModule_npu_shutdown success.

二、软件版本:
-- CANN 版本:  
package_name=Ascend-cann-toolkit
version=5.1.RC2.1
innerversion=V100R001C82SPC006B220
arch=aarch64
os=linux
--Tensorflow/Pytorch/MindSpore 版本:
--Python 版本 : 3.7.10
-- MindStudio版本 :mindspore1.8.0-cann5.1.2-py3.7-euler2.8.3
--操作系统版本 :linux

Ascend/modelzoo
暂停

内容风险标识

评论 (5)

Ascend/modelzoo暂停 .gitee-modal { width: 500px !important; }

内容风险标识