torch1.8版本跑8p，使用amp，设置O1为动态，第2个epoch出错

一、问题现象（附报错日志上下文）：
torch1.8版本，8p训练，使用amp，设置O1为动态，第2个epoch出错

[INFO]  Float status is overflow!
Traceback (most recent call last):
  File "train8p.py", line 545, in <module>
    main()
  File "train8p.py", line 168, in main
    mp.spawn(main_worker, nprocs=ngpus_per_node, args=(ngpus_per_node, args))
  File "/root/archiconda3/envs/wch1/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/root/archiconda3/envs/wch1/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes
    while not context.join():
  File "/root/archiconda3/envs/wch1/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 136, in join
    signal_name=name
torch.multiprocessing.spawn.ProcessExitedException: process 2 terminated with signal SIGABRT
Training part end.
E2E Training Duration sec : 1919
=============start running benchmark test.===================
Benchmark test end!
(wch1) [root@localhost SRGAN]# /root/archiconda3/envs/wch1/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 102 leaked semaphores to clean up at shutdown
  len(cache))
/root/archiconda3/envs/wch1/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 102 leaked semaphores to clean up at shutdown
  len(cache))
/root/archiconda3/envs/wch1/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 102 leaked semaphores to clean up at shutdown
  len(cache))
/root/archiconda3/envs/wch1/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 102 leaked semaphores to clean up at shutdown
  len(cache))
/root/archiconda3/envs/wch1/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 102 leaked semaphores to clean up at shutdown
  len(cache))
/root/archiconda3/envs/wch1/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 102 leaked semaphores to clean up at shutdown
  len(cache))
/root/archiconda3/envs/wch1/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 102 leaked semaphores to clean up at shutdown
  len(cache))
/root/archiconda3/envs/wch1/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 102 leaked semaphores to clean up at shutdown
  len(cache))
二、软件版本:
-- CANN 版本 (e.g., CANN 3.0.x，5.x.x):  
--Tensorflow/Pytorch/MindSpore 版本:pytorch
--Python 版本 (e.g., Python 3.7.5):3.7.5
-- MindStudio版本 (e.g., MindStudio 2.0.0 (beta3)):
--操作系统版本 (e.g., Ubuntu 18.04):

三、测试步骤：
环境变量设置HCCL_EXEC_TIMEOUT=1000即可解决

四、日志信息:
xxxx
请根据自己的运行环境参考以下方式搜集日志信息，如果涉及到算子开发相关的问题，建议也提供UT/ST测试和单算子集成测试相关的日志。

日志提供方式:
将日志打包后作为附件上传。若日志大小超出附件限制，则可上传至外部网盘后提供链接。

获取方法请参考wiki：
https://gitee.com/ascend/modelzoo/wikis/%E5%A6%82%E4%BD%95%E8%8E%B7%E5%8F%96%E6%97%A5%E5%BF%97%E5%92%8C%E8%AE%A1%E7%AE%97%E5%9B%BE?sort_id=4097825

Ascend/modelzoo
暂停

内容风险标识

评论 (2)

Ascend/modelzoo暂停 .gitee-modal { width: 500px !important; }

内容风险标识