910ProB 跑Yolov5_for_PyTorch_v6.0验证报错

一、问题现象（附报错日志上下文）：
https://gitee.com/ascend/modelzoo-GPL/tree/master/built-in/PyTorch/Official/cv/object_detection/Yolov5_for_PyTorch_v6.0 仓库，跑test/train_yolov5s_full_1p.sh（第8行batch_size改为64，数据集改为coco128），训练完成，验证报错，test/output/0/train_acc_1p.log：

```
[W OperatorEntry.cpp:121] Warning: Overriding a previously registered kernel for the same operator and the same dispatch key
  operator: aten::_has_compatible_shallow_copy_type(Tensor self, Tensor from) -> (bool)
    registered at /usr1/v1.8.1/pytorch/build/aten/src/ATen/RegisterSchema.cpp:20
  dispatch key: Math
  previous kernel: registered at /usr1/v1.8.1/pytorch/build/aten/src/ATen/RegisterMath.cpp:5686
       new kernel: registered at /usr1/workspace/FPTA_Daily_Plugin_open_v1.8.1-5.0.rc1/CODE/torch_npu/csrc/aten/ops/HasCompatibleShallowCopyType.cpp:37 (function registerKernel)
/usr/lib/python3.7/site-packages/pytz/__init__.py:516: ResourceWarning: unclosed file <_io.TextIOWrapper name='/usr/share/zoneinfo/zone.tab' mode='r' encoding='UTF-8'>
  for l in open(os.path.join(_tzinfo_dir, 'zone.tab'))
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib64/python3.7/site-packages/torchvision/transforms/functional_pil.py:207: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
  def resize(img, size, interpolation=Image.BILINEAR):
/usr/local/lib64/python3.7/site-packages/torchvision/transforms/functional_pil.py:280: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
  def perspective(img, perspective_coeffs, interpolation=Image.BICUBIC, fill=None):
Fusing layers... 
E89999: Inner Error, Please contact support engineer!
E89999  op[Cast], non_custom inputs shape error![FUNC:GetInputsShapes][FILE:elewise_v3.cc][LINE:316]
        TraceBack (most recent call last):
        op[Cast], elewise tiling inputs get failed.[FUNC:DoTiling][FILE:elewise_v3.cc][LINE:945]
        Autotiling func failed[FUNC:AutoTilingRun][FILE:auto_tiling_rt2.cc][LINE:100]
        [Exec][Op]Execute op failed, ge result = 4294967295[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]

terminate called after throwing an instance of 'c10::Error'
  what():  ReadQueue:/usr1/workspace/FPTA_Daily_Plugin_open_v1.8.1-5.0.rc1/CODE/torch_npu/csrc/core/npu/NPUQueue.cpp:248
Exception raised from ReadQueue at /usr1/workspace/FPTA_Daily_Plugin_open_v1.8.1-5.0.rc1/CODE/torch_npu/csrc/core/npu/NPUQueue.cpp:249 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x8c (0xffff8bdedd0c in /usr/local/lib64/python3.7/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xa0 (0xffff8bdeae08 in /usr/local/lib64/python3.7/site-packages/torch/lib/libc10.so)
frame #2: c10_npu::Repository::ReadQueue() + 0x564 (0xfffeeb837d4c in /usr/local/lib64/python3.7/site-packages/torch_npu/lib/libtorch_npu.so)
frame #3: c10_npu::Repository::Dequeue() + 0x64 (0xfffeeb837e9c in /usr/local/lib64/python3.7/site-packages/torch_npu/lib/libtorch_npu.so)
frame #4: c10_npu::StartConsume(c10_npu::Repository*, signed char) + 0x68 (0xfffeeb835738 in /usr/local/lib64/python3.7/site-packages/torch_npu/lib/libtorch_npu.so)
frame #5: <unknown function> + 0xbe134 (0xffff8bcae134 in /usr/lib64/libstdc++.so.6)
frame #6: <unknown function> + 0x87ac (0xffff9de487ac in /usr/lib64/libpthread.so.0)
frame #7: <unknown function> + 0xd60fc (0xffff9dc560fc in /usr/lib64/libc.so.6)

Process ForkServerProcess-2:
Traceback (most recent call last):
  File "/usr/lib64/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/lib64/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 61, in wrapper
    raise exp
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 58, in wrapper
    func(*args, **kwargs)
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 268, in task_distribute
    key, func_name, detail = resource_proxy[TASK_QUEUE].get()
  File "<string>", line 2, in get
  File "/usr/lib64/python3.7/multiprocessing/managers.py", line 819, in _callmethod
    kind, result = conn.recv()
  File "/usr/lib64/python3.7/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/usr/lib64/python3.7/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/usr/lib64/python3.7/multiprocessing/connection.py", line 383, in _recv
    raise EOFError
EOFError
/usr/lib64/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 91 leaked semaphores to clean up at shutdown
  len(cache))

```

二、软件版本:
-- CANN 版本：
package_name=Ascend-cann-toolkit
version=6.3.RC1.alpha001
innerversion=V100R001C84B121SPC008
arch=aarch64
os=linux
path=/usr/local/Ascend/ascend-toolkit/6.3.RC1.alpha001/aarch64-linux

--Pytorch 版本:
torch                     1.8.1
torch-npu                 1.8.1.post1
--Python 版本 : Python 3.7.9
--操作系统版本 : Linux localhost.localdomain 4.19.90-24.4.v2101.ky10.aarch64

三、测试步骤：
cd modelzoo-GPL/tree/master/built-in/PyTorch/Official/cv/object_detection/Yolov5_for_PyTorch_v6.0
bash test-npu/train_yolov5s_full_1p.sh

四、训练日志
test/output/0/train_1p.log：
```
[W OperatorEntry.cpp:121] Warning: Overriding a previously registered kernel for the same operator and the same dispatch key
  operator: aten::_has_compatible_shallow_copy_type(Tensor self, Tensor from) -> (bool)
    registered at /usr1/v1.8.1/pytorch/build/aten/src/ATen/RegisterSchema.cpp:20
  dispatch key: Math
  previous kernel: registered at /usr1/v1.8.1/pytorch/build/aten/src/ATen/RegisterMath.cpp:5686
       new kernel: registered at /usr1/workspace/FPTA_Daily_Plugin_open_v1.8.1-5.0.rc1/CODE/torch_npu/csrc/aten/ops/HasCompatibleShallowCopyType.cpp:37 (function registerKernel)
/usr/lib/python3.7/site-packages/pytz/__init__.py:516: ResourceWarning: unclosed file <_io.TextIOWrapper name='/usr/share/zoneinfo/zone.tab' mode='r' encoding='UTF-8'>
  for l in open(os.path.join(_tzinfo_dir, 'zone.tab'))
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib64/python3.7/site-packages/torchvision/transforms/functional_pil.py:207: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
  def resize(img, size, interpolation=Image.BILINEAR):
/usr/local/lib64/python3.7/site-packages/torchvision/transforms/functional_pil.py:280: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
  def perspective(img, perspective_coeffs, interpolation=Image.BICUBIC, fill=None):
[34m[1mtrain: [0mweights=yolov5s.pt, cfg=yolov5s.yaml, data=./data/coco128.yaml, hyp=data/hyps/hyp.scratch.yaml, epochs=10, batch_size=64, imgsz=640, rect=False, resume=False, nosave=False, noval=False, noautoanchor=False, evolve=None, bucket=, cache=None, image_weights=False, device=0, multi_scale=False, single_cls=False, adam=False, sync_bn=False, workers=24, project=runs/train, name=exp, exist_ok=False, quad=False, linear_lr=False, label_smoothing=0.0, patience=100, freeze=0, save_period=-1, local_rank=-1, noscale=False, entity=None, upload_dataset=False, bbox_interval=-1, artifact_alias=latest, native_amp=False, FP32=False
[34m[1mgithub: [0mskipping check (not a git repository), for updates see https://github.com/ultralytics/yolov5
[31m[1mrequirements:[0m /root/train_package/hj/modelzoo-GPL-master/built-in/PyTorch/Official/cv/object_detection/Yolov5_for_PyTorch_v6.0/requirements.txt not found, check failed.
[34m[1mhyperparameters: [0mlr0=0.01, lrf=0.1, momentum=0.937, weight_decay=0.0005, warmup_epochs=3.0, warmup_momentum=0.8, warmup_bias_lr=0.1, box=0.05, cls=0.5, cls_pw=1.0, obj=1.0, obj_pw=1.0, iou_t=0.2, anchor_t=4.0, fl_gamma=0.0, hsv_h=0.015, hsv_s=0.7, hsv_v=0.4, degrees=0.0, translate=0.1, scale=0.5, shear=0.0, perspective=0.0, flipud=0.0, fliplr=0.5, mosaic=1.0, mixup=0.0, copy_paste=0.0
[34m[1mWeights & Biases: [0mrun 'pip install wandb' to automatically track and visualize YOLOv5 🚀 runs (RECOMMENDED)
[34m[1mTensorBoard: [0mStart with 'tensorboard --logdir runs/train', view at http://localhost:6006/

from  n    params  module                                  arguments                     
  0                -1  1      3520  models.common.Conv                      [3, 32, 6, 2, 2]              
  1                -1  1     18560  models.common.Conv                      [32, 64, 3, 2]                
  2                -1  1     18816  models.common.C3                        [64, 64, 1]                   
  3                -1  1     73984  models.common.Conv                      [64, 128, 3, 2]               
  4                -1  2    115712  models.common.C3                        [128, 128, 2]                 
  5                -1  1    295424  models.common.Conv                      [128, 256, 3, 2]              
  6                -1  3    625152  models.common.C3                        [256, 256, 3]                 
  7                -1  1   1180672  models.common.Conv                      [256, 512, 3, 2]              
  8                -1  1   1182720  models.common.C3                        [512, 512, 1]                 
  9                -1  1    656896  models.common.SPPF                      [512, 512, 5]                 
 10                -1  1    131584  models.common.Conv                      [512, 256, 1, 1]              
 11                -1  1         0  torch.nn.modules.upsampling.Upsample    [None, 2, 'nearest']          
 12           [-1, 6]  1         0  models.common.Concat                    [1]                           
 13                -1  1    361984  models.common.C3                        [512, 256, 1, False]          
 14                -1  1     33024  models.common.Conv                      [256, 128, 1, 1]              
 15                -1  1         0  torch.nn.modules.upsampling.Upsample    [None, 2, 'nearest']          
 16           [-1, 4]  1         0  models.common.Concat                    [1]                           
 17                -1  1     90880  models.common.C3                        [256, 128, 1, False]          
 18                -1  1    147712  models.common.Conv                      [128, 128, 3, 2]              
 19          [-1, 14]  1         0  models.common.Concat                    [1]                           
 20                -1  1    296448  models.common.C3                        [256, 256, 1, False]          
 21                -1  1    590336  models.common.Conv                      [256, 256, 3, 2]              
 22          [-1, 10]  1         0  models.common.Concat                    [1]                           
 23                -1  1   1182720  models.common.C3                        [512, 512, 1, False]          
 24      [17, 20, 23]  1    229245  models.yolo.Detect                      [80, [[10, 13, 16, 30, 33, 23], [30, 61, 62, 45, 59, 119], [116, 90, 156, 198, 373, 326]], [128, 256, 512]]
Model Summary: 270 layers, 7235389 parameters, 7235389 gradients, 16.6 GFLOPs

Transferred 348/349 items from yolov5s.pt
Scaled weight_decay = 0.0005
[34m[1moptimizer:[0m NpuFusedSGD with parameter groups 57 weight, 60 weight (no decay), 60 bias
Selected optimization level O1:  Insert automatic casts around Pytorch functions and Tensor methods.

Defaults for this optimization level are:
enabled                : True
opt_level              : O1
cast_model_type        : None
patch_torch_functions  : True
keep_batchnorm_fp32    : None
master_weights         : None
loss_scale             : dynamic
combine_grad           : None
combine_ddp            : None
ddp_replica_count      : 4
check_combined_tensors : None
user_cast_preferred    : None
Processing user overrides (additional kwargs that are not None)...
After processing overrides, optimization options are:
enabled                : True
opt_level              : O1
cast_model_type        : None
patch_torch_functions  : True
keep_batchnorm_fp32    : None
master_weights         : None
loss_scale             : 128.0
combine_grad           : True
combine_ddp            : False
ddp_replica_count      : 4
check_combined_tensors : None
user_cast_preferred    : None
Use npu fused optimizer

[34m[1mtrain: [0mScanning 'datasets/coco128/labels/train2017_yolov5_v6.cache' images and labels... 128 found, 0 missing, 2 empty, 0 corrupted: 100%|██████████| 128/128 [00:00<?, ?it/s]
[34m[1mtrain: [0mScanning 'datasets/coco128/labels/train2017_yolov5_v6.cache' images and labels... 128 found, 0 missing, 2 empty, 0 corrupted: 100%|██████████| 128/128 [00:00<?, ?it/s]

[34m[1mval: [0mScanning 'datasets/coco128/labels/train2017_yolov5_v6.cache' images and labels... 128 found, 0 missing, 2 empty, 0 corrupted: 100%|██████████| 128/128 [00:00<?, ?it/s]
[34m[1mval: [0mScanning 'datasets/coco128/labels/train2017_yolov5_v6.cache' images and labels... 128 found, 0 missing, 2 empty, 0 corrupted: 100%|██████████| 128/128 [00:00<?, ?it/s]
Plotting labels...

[34m[1mautoanchor: [0mAnalyzing anchors... anchors/target = 4.27, Best Possible Recall (BPR) = 0.9935
Image sizes 640 train, 640 val
Using 24 dataloader workers
Logging results to [1mruns/train/exp6[0m
Starting training for 10 epochs...
[W OpCommand.cpp:127] Warning: [Check][offset] Check input storage_offset[%ld] = 0 failed, result is untrustworthy33600 (function operator())
group num: 3
Epoch:[ 0][   0/   2][npu:0][FPS:1.7][mTime:37.905][pTime:37.905][dTime:2.895] IoU:0.020 objectness:0.071 classfication:0.008 totalLoss:0.098
Epoch:[ 0][   1/   2][npu:0][FPS:3.1][mTime:20.832][pTime:3.165][dTime:0.000] IoU:0.018 objectness:0.067 classfication:0.007 totalLoss:0.093
```
后面正常结束，省略。

Ascend/modelzoo
暂停

内容风险标识

评论 (3)

Ascend/modelzoo暂停 .gitee-modal { width: 500px !important; }

内容风险标识