2.4K Star 8.2K Fork 4.4K

GVPMindSpore / mindspore

 / 详情

【ST】【MS】【2.3】非下沉模式,resnet50网络 ,pynative模式下开启profiler,单卡训练,profiler解析耗时太长

WIP
Bug-Report
创建于  
2024-04-15 16:41
name about labels
Bug Report Use this template for reporting a bug kind/bug

Describe the current behavior / 问题描述 (Mandatory / 必填)

910A环境,非下沉模式,resnet50网络 ,pynative模式下开启profiler,单卡训练,profiler解析耗时太长

Environment / 环境信息 (Mandatory / 必填)

  • Hardware Environment(Ascend/GPU/CPU) / 硬件环境:

Please delete the backend not involved / 请删除不涉及的后端:
/device ascend

  • Software Environment / 软件环境 (Mandatory / 必填):
    -- MindSpore version (e.g., 1.7.0.Bxxx) :
    -- Python version (e.g., Python 3.7.5) :
    -- OS platform and distribution (e.g., Linux Ubuntu 16.04):
    -- GCC/Compiler version (if compiled from source):
    commit_id = '[sha1]:aa16e0b8,[branch]:(HEAD,origin/r2.3,r2.3)'
    runpkg_version:Milan_C17/20240408

  • Excute Mode / 执行模式 (Mandatory / 必填)(PyNative/Graph):

Please delete the mode not involved / 请删除不涉及的模式:
/mode pynative

Related testcase / 关联用例 (Mandatory / 必填)

test_ms_mi_profiler_pynative_resnet50_1p_0001

Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)

1、取mindspore model_zoo中网络resnet50,将训练脚本中的图模式参数设置为pynative,设置为数据非下沉模式,然后开启profiler,执行网络训练,检查生成profiling数据及mindinsight展示情况

用例复现步骤:
source /home/miniconda3/bin/activate ci
export TRAIN_MODE=PYNATIVE_MODE
export DEVICE_TYPE=Ascend_Arm
export ENV_DEVICE=0
source solution_test/env_set.source -e ascend

cd solution_test/cases/03subject_test/02usability/perf_tuning/profiler_pynative
pytest -s test_ms_mi_profiler_pynative_resnet50_1p_0001.py

Describe the expected behavior / 预期结果 (Mandatory / 必填)

用例执行不能超过30分钟,训练需时5分钟左右,profiler解析耗时正常,最好20分钟内完成

Related log / screenshot / 日志 / 截图 (Mandatory / 必填)

训练日志: 如下几个告警步骤都耗时33分钟了
[INFO] PROFILING(5613,python):2024-04-15-14:22:56.234.356 [prof_inner_api.cpp:105] >>> (tid:5613) total_size_report [additional] read size: 924301 bytes, write size: 924301 bytes
[WARNING] ME(5613:281472934203408,MainProcess):2024-04-15-14:22:56.488.414 [mindspore/profiler/parser/ascend_msprof_exporter.py:76] [Profiler]The msprof command was not found. Searching from environment variables...
[WARNING] ME(5613:281472934203408,MainProcess):2024-04-15-14:45:25.226.868 [mindspore/profiler/parser/ascend_timeline_generator.py:168] Could not find cpu op file in /home/jenkins/wenli/solution_test/cases/03subject_test/02usability/perf_tuning/profiler_pynative/test_ms_mi_profiler_pynative_resnet50_1p_0001/pynative_resnet50_1p/profiler
[WARNING] ME(5613:281472934203408,MainProcess):2024-04-15-14:51:50.519.448 [mindspore/profiler/profiling.py:1308] [Profiler]The parameter profile_memory is not supported on Ascend PyNative mode currently.
[WARNING] ME(5613:281472934203408,MainProcess):2024-04-15-14:51:50.526.479 [mindspore/profiler/parser/memory_usage_parser.py:141] The memory file does not exist! Please ignore the warning if you are running heterogeneous training.
[WARNING] ME(5613:281472934203408,MainProcess):2024-04-15-14:51:50.526.972 [mindspore/profiler/profiling.py:1314] The file </home/jenkins/wenli/solution_test/cases/03subject_test/02usability/perf_tuning/profiler_pynative/test_ms_mi_profiler_pynative_resnet50_1p_0001/pynative_resnet50_1p/profiler/memory_usage_0.pb> not found.
[WARNING] ME(5613:281472934203408,MainProcess):2024-04-15-14:55:28.685.133 [mindspore/profiler/parser/msadvisor_parser.py:161] The aicore timeline file does not exist!
[WARNING] ME(5613:281472934203408,MainProcess):2024-04-15-14:55:28.685.508 [mindspore/profiler/profiling.py:1421] MSAdvisor running failed. [ProfilerFileNotFoundException] code: 50546084, msg: The file </home/jenkins/wenli/solution_test/cases/03subject_test/02usability/perf_tuning/profiler_pynative/test_ms_mi_profiler_pynative_resnet50_1p_0001/pynative_resnet50_1p/profiler/output_timeline_data_0.txt> not found.
[WARNING] ME(5613:281472934203408,MainProcess):2024-04-15-14:55:28.686.584 [mindspore/profiler/profiling.py:1425] Pynative mode does not support MSAdvisor analyzer currently.

Special notes for this issue/备注 (Optional / 选填)

走给 臧庆香

评论 (4)

wenli 创建了Bug-Report
wenli 添加了
 
kind/bug
标签
wenli 添加了
 
v2.3.0.rc2
标签
wenli 添加了
 
attr/function
标签
wenli 添加了
 
sig/visualization
标签
wenli 添加了
 
stage/func-debug
标签
wenli 添加协作者zangqx
展开全部操作日志

Please assign maintainer to check this issue.
请为此issue分配处理人。
@wenli

感谢您的提问,您可以评论//mindspore-assistant更快获取帮助:

  1. 如果您刚刚接触MindSpore,或许您可以在教程找到答案
  2. 如果您是资深Pytorch用户,您或许需要:
  1. 如果您遇到动态图问题,可以设置set_context(pynative_synchronize=True)查看报错栈协助定位
  2. 模型精度调优问题可参考官网调优指南
  3. 如果您反馈的是框架BUG,请确认您在ISSUE中提供了MindSpore版本、使用的后端类型(CPU、GPU、Ascend)、环境、训练的代码官方链接以及可以复现报错的代码的启动方式等必要的定位信息
  4. 如果您已经定位出问题根因,欢迎提交PR参与MindSpore开源社区,我们会尽快review
mudongrui 添加协作者mudongrui
mudongrui 负责人mudongrui 修改为zangqx
mudongrui 取消协作者zangqx
wenli 修改了描述
fangwenyi 移除了
 
v2.3.0.rc2
标签
fangwenyi 移除了
 
v2.3.0.rc2
标签
fangwenyi 添加了
 
v2.3.0
标签

ms版本:commit_id = '[sha1]:13fef1e8,[branch]:(HEAD,origin/master,origin/HEAD,master)'
run包:runpkg_version:Milan_C18/20240522
master分支也存在该问题

wenli 添加了
 
master
标签
wenli 添加了
 
device/ascend
标签
fangwenyi 添加了关联分支master 选项
fangwenyi 添加了问题后端类型Ascend 选项
wenli 添加了
 
br_base
标签
zangqx 添加协作者zangqx
zangqx 负责人zangqx 修改为朱家兴
朱家兴 计划截止日期2024-04-22 修改为2024-06-05
朱家兴 计划截止日期2024-06-05 修改为2024-06-06
朱家兴 任务状态TODO 修改为WIP

0606进展:使用测试环境复现用例,发现网络无法跑通
与测试沟通,待此问题解决https://e.gitee.com/mind_spore/issues/list?issue=I9UIIN
报错截图:输入图片说明

i-robot 添加了
 
foruda
标签
lilei 添加了关联分支br_base 选项

登录 后才可以发表评论

状态
负责人
项目
里程碑
Pull Requests
关联的 Pull Requests 被合并后可能会关闭此 issue
分支
开始日期   -   截止日期
-
置顶选项
优先级
预计工期 (小时)
参与者(5)
Python
1
https://gitee.com/mindspore/mindspore.git
git@gitee.com:mindspore/mindspore.git
mindspore
mindspore
mindspore

搜索帮助