登录
注册
开源
企业版
高校版
搜索
帮助中心
使用条款
关于我们
开源
企业版
高校版
私有云
模力方舟
AI 队友
登录
注册
Gitee 年度开源项目评选结果正式揭晓,速戳👉
代码拉取完成,页面将自动刷新
仓库状态说明
捐赠
捐赠前请先登录
取消
前往登录
扫描微信二维码支付
取消
支付完成
支付提示
将跳转至支付宝完成支付
确定
取消
Watch
不关注
关注所有动态
仅关注版本发行动态
关注但不提醒动态
68
Star
258
Fork
192
Ascend
/
modelzoo
暂停
代码
Issues
157
Pull Requests
9
Wiki
统计
流水线
服务
JavaDoc
PHPDoc
质量分析
Jenkins for Gitee
腾讯云托管
腾讯云 Serverless
悬镜安全
阿里云 SAE
Codeblitz
SBOM
我知道了,不再自动展开
更新失败,请稍后重试!
移除标识
内容风险标识
本任务被
标识为内容中包含有代码安全 Bug 、隐私泄露等敏感信息,仓库外成员不可访问
NPU error code is:507018,E39999: Inner Error!
DONE
#I7ZTF8
Bug-Report
蒋文
创建于
2023-09-08 17:53
一、问题现象(附报错日志上下文): warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", DeprecationWarning) /home/ma-user/work/Grounded-Segment-Anything-main/GroundingDINO/groundingdino/models/GroundingDINO/ms_deform_attn.py:31: UserWarning: Failed to load custom C++ ops. Running on CPU mode Only! warnings.warn("Failed to load custom C++ ops. Running on CPU mode Only!") /home/ma-user/anaconda3/envs/python-3.7.10/lib/python3.7/site-packages/torch_npu/contrib/transfer_to_npu.py:129: ImportWarning: ************************************************************************************************************* The torch.Tensor.cuda and torch.nn.Module.cuda are replaced with torch.Tensor.npu and torch.nn.Module.npu now.. The torch.cuda.DoubleTensor is replaced with torch.npu.FloatTensor cause the double type is not supported now.. The backend in torch.distributed.init_process_group set to hccl now.. The torch.cuda.* and torch.cuda.amp.* are replaced with torch.npu.* and torch.npu.amp.* now.. The device parameters have been replaced with npu in the function below: torch.logspace, torch.randint, torch.hann_window, torch.rand, torch.full_like, torch.ones_like, torch.rand_like, torch.randperm, torch.arange, torch.frombuffer, torch.normal, torch._empty_per_channel_affine_quantized, torch.empty_strided, torch.empty_like, torch.scalar_tensor, torch.tril_indices, torch.bartlett_window, torch.ones, torch.sparse_coo_tensor, torch.randn, torch.kaiser_window, torch.tensor, torch.triu_indices, torch.as_tensor, torch.zeros, torch.randint_like, torch.full, torch.eye, torch._sparse_csr_tensor_unsafe, torch.empty, torch._sparse_coo_tensor_unsafe, torch.blackman_window, torch.zeros_like, torch.range, torch.sparse_csr_tensor, torch.randn_like, torch.from_file, torch._cudnn_init_dropout_state, torch._empty_affine_quantized, torch.linspace, torch.hamming_window, torch.empty_quantized, torch._pin_memory, torch.device, torch.Tensor.new_empty, torch.Tensor.new_empty_strided, torch.Tensor.new_full, torch.Tensor.new_ones, torch.Tensor.new_tensor, torch.Tensor.new_zeros, torch.Tensor.to, torch.nn.Module.to, torch.nn.Module.to_empty ************************************************************************************************************* warnings.warn(msg, ImportWarning) True npu /home/ma-user/anaconda3/envs/python-3.7.10/lib/python3.7/site-packages/torch/functional.py:568: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /usr1/v1.11.0/pytorch/aten/src/ATen/native/TensorShape.cpp:2228.) return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined] final text_encoder_type: bert-base-uncased Some weights of the model checkpoint at /home/ma-user/work/Grounded-Segment-Anything-main/huggingface were not used when initializing BertModel: ['cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.seq_relationship.weight'] - This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model). - This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). _IncompatibleKeys(missing_keys=[], unexpected_keys=['label_enc.weight']) tensor([[ True, False, True, True]], device='npu:0') NonZero Traceback (most recent call last): File "grounded_sam_demo.py", line 194, in <module> model, image, text_prompt, box_threshold, text_threshold, device=device File "grounded_sam_demo.py", line 64, in get_grounding_output outputs = model(image[None], captions=[caption]) File "/home/ma-user/anaconda3/envs/python-3.7.10/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "/home/ma-user/work/Grounded-Segment-Anything-main/GroundingDINO/groundingdino/models/GroundingDINO/groundingdino.py", line 242, in forward tokenized, self.specical_tokens, self.tokenizer File "/home/ma-user/work/Grounded-Segment-Anything-main/GroundingDINO/groundingdino/models/GroundingDINO/bertwarper.py", line 241, in generate_masks_with_special_tokens_and_transfer_map idxs = torch.nonzero(special_tokens_mask) RuntimeError: Run:/usr1/workspace/FPTA_Daily_open_pytorchv1.11.0-5.0.rc1/CODE/torch_npu/csrc/framework/OpParamMaker.cpp:138 NPU error,NPU error code is:507018 E39999: Inner Error! E39999 An exception occurred during AICPU execution, stream_id:411, task_id:18, errcode:21008, msg:inner error[FUNC:ProcessAicpuErrorInfo][FILE:device_error_proc.cc][LINE:667] Aicpu kernel execute failed, device_id=0, stream_id=411, task_id=18.[FUNC:PrintAicpuErrorInfo][FILE:task.cc][LINE:747] Aicpu kernel execute failed, device_id=0, stream_id=411, task_id=18, fault op_name=NonZero[FUNC:GetError][FILE:stream.cc][LINE:846] rtStreamSynchronize execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:45] Call rtStreamSynchronize(stream) fail, ret: 0x7BC8A[FUNC:LaunchKernel][FILE:op_task.cc][LINE:1403] [Exec][Op]Execute op failed, ge result = 507018[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:162] An exception occurred during AICPU execution, stream_id:411, task_id:21, errcode:21008, msg:inner error[FUNC:ProcessAicpuErrorInfo][FILE:device_error_proc.cc][LINE:667] Aicpu kernel execute failed, device_id=0, stream_id=411, task_id=21.[FUNC:PrintAicpuErrorInfo][FILE:task.cc][LINE:747] Aicpu kernel execute failed, device_id=0, stream_id=411, task_id=21, fault op_name=NonZero[FUNC:GetError][FILE:stream.cc][LINE:846] An exception occurred during AICPU execution, stream_id:411, task_id:24, errcode:21008, msg:inner error[FUNC:ProcessAicpuErrorInfo][FILE:device_error_proc.cc][LINE:667] Aicpu kernel execute failed, device_id=0, stream_id=411, task_id=24.[FUNC:PrintAicpuErrorInfo][FILE:task.cc][LINE:747] Aicpu kernel execute failed, device_id=0, stream_id=411, task_id=24, fault op_name=NonZero[FUNC:GetError][FILE:stream.cc][LINE:846] DEVICE[0] PID[27845]: EXCEPTION TASK: Exception info:TGID=54778, model id=65535, stream id=411, stream phase=3, task id=18, task type=aicpu kernel, recently received task id=19, recently send task id=17, task phase=RUN Message info[0]:aicpu=0,slot_id=0,report_mailbox_flag=0x5a5a5a5a,state=0x5210 Other info[0]:time=2023-09-08-08:56:43.623.893, function=proc_aicpu_task_done, line=972, error code=0x2a EXCEPTION TASK: Exception info:TGID=54778, model id=65535, stream id=411, stream phase=3, task id=21, task type=aicpu kernel, recently received task id=22, recently send task id=20, task phase=RUN Message info[0]:aicpu=0,slot_id=0,report_mailbox_flag=0x5a5a5a5a,state=0x5210 Other info[0]:time=2023-09-08-08:56:43.628.653, function=proc_aicpu_task_done, line=972, error code=0x2a EXCEPTION TASK: Exception info:TGID=54778, model id=65535, stream id=411, stream phase=3, task id=24, task type=aicpu kernel, recently received task id=25, recently send task id=23, task phase=RUN Message info[0]:aicpu=0,slot_id=0,report_mailbox_flag=0x5a5a5a5a,state=0x5210 Other info[0]:time=2023-09-08-08:56:43.632.276, function=proc_aicpu_task_done, line=972, error code=0x2a THPModule_npu_shutdown success. 二、软件版本: -- CANN 版本: package_name=Ascend-cann-toolkit version=5.1.RC2.1 innerversion=V100R001C82SPC006B220 arch=aarch64 os=linux --Tensorflow/Pytorch/MindSpore 版本: --Python 版本 : 3.7.10 -- MindStudio版本 :mindspore1.8.0-cann5.1.2-py3.7-euler2.8.3 --操作系统版本 :linux
一、问题现象(附报错日志上下文): warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", DeprecationWarning) /home/ma-user/work/Grounded-Segment-Anything-main/GroundingDINO/groundingdino/models/GroundingDINO/ms_deform_attn.py:31: UserWarning: Failed to load custom C++ ops. Running on CPU mode Only! warnings.warn("Failed to load custom C++ ops. Running on CPU mode Only!") /home/ma-user/anaconda3/envs/python-3.7.10/lib/python3.7/site-packages/torch_npu/contrib/transfer_to_npu.py:129: ImportWarning: ************************************************************************************************************* The torch.Tensor.cuda and torch.nn.Module.cuda are replaced with torch.Tensor.npu and torch.nn.Module.npu now.. The torch.cuda.DoubleTensor is replaced with torch.npu.FloatTensor cause the double type is not supported now.. The backend in torch.distributed.init_process_group set to hccl now.. The torch.cuda.* and torch.cuda.amp.* are replaced with torch.npu.* and torch.npu.amp.* now.. The device parameters have been replaced with npu in the function below: torch.logspace, torch.randint, torch.hann_window, torch.rand, torch.full_like, torch.ones_like, torch.rand_like, torch.randperm, torch.arange, torch.frombuffer, torch.normal, torch._empty_per_channel_affine_quantized, torch.empty_strided, torch.empty_like, torch.scalar_tensor, torch.tril_indices, torch.bartlett_window, torch.ones, torch.sparse_coo_tensor, torch.randn, torch.kaiser_window, torch.tensor, torch.triu_indices, torch.as_tensor, torch.zeros, torch.randint_like, torch.full, torch.eye, torch._sparse_csr_tensor_unsafe, torch.empty, torch._sparse_coo_tensor_unsafe, torch.blackman_window, torch.zeros_like, torch.range, torch.sparse_csr_tensor, torch.randn_like, torch.from_file, torch._cudnn_init_dropout_state, torch._empty_affine_quantized, torch.linspace, torch.hamming_window, torch.empty_quantized, torch._pin_memory, torch.device, torch.Tensor.new_empty, torch.Tensor.new_empty_strided, torch.Tensor.new_full, torch.Tensor.new_ones, torch.Tensor.new_tensor, torch.Tensor.new_zeros, torch.Tensor.to, torch.nn.Module.to, torch.nn.Module.to_empty ************************************************************************************************************* warnings.warn(msg, ImportWarning) True npu /home/ma-user/anaconda3/envs/python-3.7.10/lib/python3.7/site-packages/torch/functional.py:568: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /usr1/v1.11.0/pytorch/aten/src/ATen/native/TensorShape.cpp:2228.) return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined] final text_encoder_type: bert-base-uncased Some weights of the model checkpoint at /home/ma-user/work/Grounded-Segment-Anything-main/huggingface were not used when initializing BertModel: ['cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.seq_relationship.weight'] - This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model). - This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). _IncompatibleKeys(missing_keys=[], unexpected_keys=['label_enc.weight']) tensor([[ True, False, True, True]], device='npu:0') NonZero Traceback (most recent call last): File "grounded_sam_demo.py", line 194, in <module> model, image, text_prompt, box_threshold, text_threshold, device=device File "grounded_sam_demo.py", line 64, in get_grounding_output outputs = model(image[None], captions=[caption]) File "/home/ma-user/anaconda3/envs/python-3.7.10/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "/home/ma-user/work/Grounded-Segment-Anything-main/GroundingDINO/groundingdino/models/GroundingDINO/groundingdino.py", line 242, in forward tokenized, self.specical_tokens, self.tokenizer File "/home/ma-user/work/Grounded-Segment-Anything-main/GroundingDINO/groundingdino/models/GroundingDINO/bertwarper.py", line 241, in generate_masks_with_special_tokens_and_transfer_map idxs = torch.nonzero(special_tokens_mask) RuntimeError: Run:/usr1/workspace/FPTA_Daily_open_pytorchv1.11.0-5.0.rc1/CODE/torch_npu/csrc/framework/OpParamMaker.cpp:138 NPU error,NPU error code is:507018 E39999: Inner Error! E39999 An exception occurred during AICPU execution, stream_id:411, task_id:18, errcode:21008, msg:inner error[FUNC:ProcessAicpuErrorInfo][FILE:device_error_proc.cc][LINE:667] Aicpu kernel execute failed, device_id=0, stream_id=411, task_id=18.[FUNC:PrintAicpuErrorInfo][FILE:task.cc][LINE:747] Aicpu kernel execute failed, device_id=0, stream_id=411, task_id=18, fault op_name=NonZero[FUNC:GetError][FILE:stream.cc][LINE:846] rtStreamSynchronize execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:45] Call rtStreamSynchronize(stream) fail, ret: 0x7BC8A[FUNC:LaunchKernel][FILE:op_task.cc][LINE:1403] [Exec][Op]Execute op failed, ge result = 507018[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:162] An exception occurred during AICPU execution, stream_id:411, task_id:21, errcode:21008, msg:inner error[FUNC:ProcessAicpuErrorInfo][FILE:device_error_proc.cc][LINE:667] Aicpu kernel execute failed, device_id=0, stream_id=411, task_id=21.[FUNC:PrintAicpuErrorInfo][FILE:task.cc][LINE:747] Aicpu kernel execute failed, device_id=0, stream_id=411, task_id=21, fault op_name=NonZero[FUNC:GetError][FILE:stream.cc][LINE:846] An exception occurred during AICPU execution, stream_id:411, task_id:24, errcode:21008, msg:inner error[FUNC:ProcessAicpuErrorInfo][FILE:device_error_proc.cc][LINE:667] Aicpu kernel execute failed, device_id=0, stream_id=411, task_id=24.[FUNC:PrintAicpuErrorInfo][FILE:task.cc][LINE:747] Aicpu kernel execute failed, device_id=0, stream_id=411, task_id=24, fault op_name=NonZero[FUNC:GetError][FILE:stream.cc][LINE:846] DEVICE[0] PID[27845]: EXCEPTION TASK: Exception info:TGID=54778, model id=65535, stream id=411, stream phase=3, task id=18, task type=aicpu kernel, recently received task id=19, recently send task id=17, task phase=RUN Message info[0]:aicpu=0,slot_id=0,report_mailbox_flag=0x5a5a5a5a,state=0x5210 Other info[0]:time=2023-09-08-08:56:43.623.893, function=proc_aicpu_task_done, line=972, error code=0x2a EXCEPTION TASK: Exception info:TGID=54778, model id=65535, stream id=411, stream phase=3, task id=21, task type=aicpu kernel, recently received task id=22, recently send task id=20, task phase=RUN Message info[0]:aicpu=0,slot_id=0,report_mailbox_flag=0x5a5a5a5a,state=0x5210 Other info[0]:time=2023-09-08-08:56:43.628.653, function=proc_aicpu_task_done, line=972, error code=0x2a EXCEPTION TASK: Exception info:TGID=54778, model id=65535, stream id=411, stream phase=3, task id=24, task type=aicpu kernel, recently received task id=25, recently send task id=23, task phase=RUN Message info[0]:aicpu=0,slot_id=0,report_mailbox_flag=0x5a5a5a5a,state=0x5210 Other info[0]:time=2023-09-08-08:56:43.632.276, function=proc_aicpu_task_done, line=972, error code=0x2a THPModule_npu_shutdown success. 二、软件版本: -- CANN 版本: package_name=Ascend-cann-toolkit version=5.1.RC2.1 innerversion=V100R001C82SPC006B220 arch=aarch64 os=linux --Tensorflow/Pytorch/MindSpore 版本: --Python 版本 : 3.7.10 -- MindStudio版本 :mindspore1.8.0-cann5.1.2-py3.7-euler2.8.3 --操作系统版本 :linux
评论 (
5
)
登录
后才可以发表评论
状态
DONE
TODO
Analysing
ACCEPTED
WIP
Feedback
TEST
DONE
REJECTED
负责人
未设置
张安琪
zhang-anqi11
负责人
协作者
+负责人
+协作者
沈鹏程
shenpengcheng
负责人
协作者
+负责人
+协作者
标签
未设置
项目
未立项任务
未立项任务
里程碑
未关联里程碑
未关联里程碑
Pull Requests
未关联
未关联
关联的 Pull Requests 被合并后可能会关闭此 issue
分支
未关联
未关联
master
开始日期   -   截止日期
-
置顶选项
不置顶
置顶等级:高
置顶等级:中
置顶等级:低
优先级
不指定
严重
主要
次要
不重要
预计工期
(小时)
参与者(2)
1
https://gitee.com/ascend/modelzoo.git
git@gitee.com:ascend/modelzoo.git
ascend
modelzoo
modelzoo
点此查找更多帮助
搜索帮助
Git 命令在线学习
如何在 Gitee 导入 GitHub 仓库
Git 仓库基础操作
企业版和社区版功能对比
SSH 公钥设置
如何处理代码冲突
仓库体积过大,如何减小?
如何找回被删除的仓库数据
Gitee 产品配额说明
GitHub仓库快速导入Gitee及同步更新
什么是 Release(发行版)
将 PHP 项目自动发布到 packagist.org
评论
仓库举报
回到顶部
登录提示
该操作需登录 Gitee 帐号,请先登录后再操作。
立即登录
没有帐号,去注册