76 Star 221 Fork 167

Ascend / modelzoo

 / 详情

【众智】【北京交通大学】【ID2025】Smith模型精度问题

Analysing
Bug-Report
创建于  
2022-05-30 12:21

一、问题现象(附报错日志上下文):
模型精度在NPU上比GPU差很多。通过精度三板斧分析法,检测出有算子溢出。
精度情况说明,由于原始数据集获取的问题,验证集采用训练集,验证结果在GPU上:
输入图片说明

NPU上:
输入图片说明
(截图只是三板斧某一步的指标)

二、软件版本:
-- CANN 版本 (e.g., CANN 3.0.x,5.x.x):
5.1.RC1.alpha005
--Tensorflow/Pytorch/MindSpore 版本:
1.15.0
--Python 版本 (e.g., Python 3.7.5):
3.7.5
-- MindStudio版本 (e.g., MindStudio 2.0.0 (beta3)):
--操作系统版本 (e.g., Ubuntu 18.04):
Ubuntu 18.04

三、测试步骤:
通过如下的检测步骤,第4步分析出有算子溢出。
请按下列步骤提供与GPU的精度对比数据和溢出检测的数据路径,如果哪步精度达标了就不需要再做下一步:
1.默认精度(即不开混合精度):
2.默认精度+关闭融合规则的精度(该步测试完后记得一定要打开融合规则,以免影响其它测试):
3.默认精度+LossScale的精度:
4.混合精度+LossScale下+溢出检测(如精度异常,请提供溢出检测数据OBS路径):
5.使用"force_fp32"+LossScale下的精度:
算子溢出情况:
输入图片说明

溢出数据如下:
输入图片说明

输入图片说明

输入图片说明

四、日志信息:

URL:
https://e-share.obs-website.cn-north-1.myhuaweicloud.com?token=OVWrEUKDR1ADCmJjkBGQ3te9E7FTA135+GMz7Pmx4Qm/2OY04ttw8Kfd9AIM25nBaLabgUyxnFZH9lJcZLKy+y9tgr+Z3e6qcOckRAgSsHeD6HX0AJ+9+q7tUHURCMiZzSGOgR4G8ftQTxCe2emgg7HV896Hfgo93D71ng6K45kTTHCOKHkzgztJTQrddMvNd9I2kNg2ZpIUC/xMvfqHoYKxbg+RpDbzpGGHcgdrWrtnk4pCiDwBhZQIxdEKQ2Ma2jMfnDFOD3TubUb+xiHdben0P3miuzUryQAle6AtzUN75sLFh1Zg+hi3E/ANmIomECwuuMjD5A0Kj7iGWWeg4yiA8PxC3w++KTlPtWGE+cDnlDa2Dj5c+sQf3VwzwetrL36Is8M5B37U6bf75eZ760bOrw/Yzry5LHucH/RVXAf7yp//cgfNNHHU83MWhhrRLCQmBBofNqlNfW057hWHIcrA/JQ0loVHaPmMtOusroJqKsRL4wNvQipP0DAaX6l17uFtrwdY4fUgQVKnQyHchWIhlZaOWxcMpmnxue/rnhQjrZfm+0APTVVzMJOGWLv8

提取码:
111111

*有效期至: 2023/05/25 11:46:02 GMT+08:00

日志提供方式:
将日志打包后作为附件上传。若日志大小超出附件限制,则可上传至外部网盘后提供链接。

获取方法请参考wiki:
https://gitee.com/ascend/modelzoo/wikis/%E5%A6%82%E4%BD%95%E8%8E%B7%E5%8F%96%E6%97%A5%E5%BF%97%E5%92%8C%E8%AE%A1%E7%AE%97%E5%9B%BE?sort_id=4097825

评论 (1)

tianchen2750 创建了Bug-Report
许业涛 负责人设置为chenhu
许业涛 任务状态TODO 修改为Analysing
展开全部操作日志

1.默认精度 (即不开混合精度):
之前用的大模型测的精度。后来统一用的小模型,现在补这步的精度结果。

2.默认精度+关闭融合规则的精度 (该步测试完后记得一定要打开融合规则,以免影响其它测试):
accuracy = 0.46469906
precision = 0.45598847
recall = 0.7870486

3.混合精度+LossScale下+溢出检测(如精度异常,请提供溢出检测数据OBS路径):
URL:
https://e-share.obs-website.cn-north-1.myhuaweicloud.com?token=OVWrEUKDR1ADCmJjkBGQ3t1Ptbx7Jog/2n/YwNovNeoY8ftH2SD79iU7WMCUPH1nqS94EOijfY9yMyzQn6lOWQlOUwvB5q/+gJ9gvn8BRQtHGZfiHatC638bHKeNbAIvfO5qE/lhgukfG+4qdIfh78ekiXnRVta9A5fKryLKsLVLcJOTjjG2NJYZB+uTtF2VyDiBL0YhmBm353BiCguqF4WQK0NJVs6MDQGuN/miRK4oBBfMs3k2J4S4UDi04hv2z44hqzZvey78uxXmXrnr7xuZWT4EmwhrHG1LBmVHgJ4Wqyv94toM1bntODo9aaruj4wwRNQMeo3xBFiTGrJsV9He0NhG6ZLOSIHzRUDgstjjdo4wZo4gvg81LOP7glL4rezQwhT5wt01EIZ5wt2z+G3T/Sr6GdJNK1GHSfaOigJ4q4820lQCGw7dzoFoOCJE68khjR1u358d1Mggm8jCvFPSYl5h03Ia8KcmGy66WAFfNNp/gqnTatthRtsQomE6iVDawU3yqbE+fTsNl2NPT43ezKMvlq+P9has0eX1sNHWkGvyw84DHuWVab9rc+gxSi595BRVjxltWL5B6jsYg2ZPt6WgFLLXSeqvZLNxiwT0dhhYhFQ9h2V7xlGN3RknoweLbNxhddPENfjDxycwqYY0gq4q/uW4Ded13zXWWxYsXW8aO2OF3LgvSsEi4SIBXl/vQeuIGXWs3JkWic8aaLYaBIBsXiHgP611haw3UpG3rMcrkKRg82jUvlITzJQAECcVBb6zWnUIP5AR1Oc7E3bnolWWtNjdNxXJJ2px0aShf+VnUi/XXlBcTqzylZx9WN79sj6MT4m494RwyuIHqw==

提取码:
111111

*有效期至: 2023/06/11 16:06:08 GMT+08:00

备:

混合精度 + lossscale + 黑名单(LayerNorm) + 溢出检测
URL:
https://e-share.obs-website.cn-north-1.myhuaweicloud.com?token=OVWrEUKDR1ADCmJjkBGQ3t1Ptbx7Jog/2n/YwNovNeoY8ftH2SD79iU7WMCUPH1nqS94EOijfY9yMyzQn6lOWQlOUwvB5q/+gJ9gvn8BRQtHGZfiHatC638bHKeNbAIvfO5qE/lhgukfG+4qdIfh78ekiXnRVta9A5fKryLKsLVLcJOTjjG2NJYZB+uTtF2VyDiBL0YhmBm353BiCguqF9AI6YNLFLkLaX120sNW6rSYwycgHBPm4nFhtHIONqWYDpAGZJV+rhLDH/L5JCa9pTiBsI0YUN0DtSSLZwRWxc1Ro3QZpmDwYiGOXUceLl5MId1kmaE/8/cBFjWWThQNJLnA3CgYWzOo1aojOtv+fSfTd+hTdUdpzo+j0oDeC+RUbHKt/9rSyhyRIru118DMaqrdtY5p9xX+4q0mwh+n9/9K8jM+lPDg8C4+6k53vptuPvmiBXKnoG3gzATj8lXb6Gg2Oudh/WkcqdZxdNnIXK5F7shb+Uu6ENOThxzsdDohuW0UMyBV26+Iu8XAMMdaV7o4FcL4Gu8Z2J8Q7F/s+YVTfe6Y1eEGxmuzZvPlaO+hFq3cAldZO6FIK5/sFF12E5xQQl3qSh8sF2WArHnYkvVzaALcBP4wtzZ/RHJ0P9hC7yvlA99vZodK93WAdajFTybUlvLKZL5X8agHDHEzufRtgUosznUSAuCnOsWeMTtuAdDB+Uc3PKviWLuqROyPYixH0FDbsN2obYwIGhox33agq1v6YmAVsfq7ziW5OjxHF/if2GOg87hZKehnFwiFSzLS8XYrE9fpBha5XDK4BNcWbjYK6gOKtW4Uu4ehsprN2j1JpNeStSuq8V72YJSZnU/a9bqMrC8DDRZ3yUnQutBk4AgBKRrDzybQ6OdjQ+5nPVdP70lu8hWkvhEifkSdwZlmE9PmP5i746aizi4aQiPR57yEG7YGMkiIn0YSWF0SaBFoFCgZ1fjFU7gOF0tKxw==

提取码:
111111

*有效期至: 2023/06/11 16:08:14 GMT+08:00

**默认精度+动态lossscale + 溢出 **
URL:
https://e-share.obs-website.cn-north-1.myhuaweicloud.com?token=OVWrEUKDR1ADCmJjkBGQ3te9E7FTA135+GMz7Pmx4Qm/2OY04ttw8Kfd9AIM25nBaLabgUyxnFZH9lJcZLKy+y9tgr+Z3e6qcOckRAgSsHeD6HX0AJ+9+q7tUHURCMiZzSGOgR4G8ftQTxCe2emgg7HV896Hfgo93D71ng6K45kTTHCOKHkzgztJTQrddMvNd9I2kNg2ZpIUC/xMvfqHoZ6Wm51+ouJNwCBbi3JL1Ig1vy3JWcb0oWn1dApB6rvKnUXobYKyIkHtcYemRpyBGEjeuBQJNmtPvZEkkolvrm8vVo926XGllx1kBKnHBKGzk5ezmJMJhb11QcZid4tNYafy2QR60Zk213U4Bx69EmWObD1lwq1u++AYT/+f+OMEeMnfTgWVJ5vAOA55e5P5HlOh6tkUZ8yYxwN2HayDeCgbCHAGc/dsMsv5FRQH2qgYaesIi54NArM03gAAuNcFigdow7RcpXNu3z09b/XSwPM4Qw5F4FO9Uc+/vlznyjYJs3h5RqhNq7fo38JCr1dh1tI9A/emX+xa6JQrK/LQg6OYtdHvj9lc9qeRjB4qRsb4G8yw8JUu6e8OImbDIPPRCOOljhRynbHLDUX18MOjIBjSukiq1o0Zyp4O6XTcdb0a6LSF318osBRQ3cH4bPG7sv31y+ioQsVcuDVWD5wsdXPrJ+98bAWMbRzgxDtBzTCCQ72KOtYOtskkZ0A0A586+8clYhso0yrSvpyWr0RAoWr8MPqz7RPSYWz4pEQcvrXANRdP1jdQTurpl1Om6SdZw0psD5XhbOPQH68J/asrOsh9YSJzUJ3gzET+lwLx8Wgk

提取码:
111111

*有效期至: 2023/06/11 16:08:41 GMT+08:00

4.使用"force_fp32"+LossScale下的精度:
accuracy = 0.5138889
precision = 0.47805455
recall = 0.501868

数据集路径:

URL:
https://e-share.obs-website.cn-north-1.myhuaweicloud.com?token=OVWrEUKDR1ADCmJjkBGQ3te9E7FTA135+GMz7Pmx4Qm/2OY04ttw8Kfd9AIM25nBaLabgUyxnFZH9lJcZLKy+y9tgr+Z3e6qcOckRAgSsHeD6HX0AJ+9+q7tUHURCMiZzSGOgR4G8ftQTxCe2emgg7HV896Hfgo93D71ng6K45kTTHCOKHkzgztJTQrddMvNd9I2kNg2ZpIUC/xMvfqHoZPGllpo6cO6gWo3cdJV/MrGHwuacGibGUbFYoIzLSIkUvMcY0DvbY30Mz/N1ndNLtW3571dkXkqifkfWcoPVGcDoUI7pDjDDG2bM9u64VHcB+EP35rB1sM/uEGnrA7DClo06bJAZRuJ2Kdb24jcb3vBmdAm51CRfSfxt8X9bsyMdVWOKU9XwT7PDh+cwbTFrjovY9R611JzaUZ8/iL6LVUfblNnp4RaEts6Jn9Aw/nkJTzCUBuhKa1NsoFGYENnpMEMVI/I+gQ5w6P9lHtept73Fu0vr8Mm01rqquoGJxpxWxkFg47IFINfv4NROEfu00MJU3hfE7gh7L4FqvSr8IcxtLL2/6EPzSpfi9QdTa+fM2HIdAPiwjTrYgrJiK7XPDB5I3znSZvuMlbhRLT9gBg=

提取码:
111111

*有效期至: 2023/06/11 20:58:38 GMT+08:00

chenhu 添加协作者chenhu
chenhu 负责人chenhu 修改为wubo
chenhu 取消协作者chenhu
颜亚文 负责人wubo 修改为钱泽洪

登录 后才可以发表评论

状态
负责人
项目
里程碑
Pull Requests
关联的 Pull Requests 被合并后可能会关闭此 issue
分支
开始日期   -   截止日期
-
置顶选项
优先级
预计工期 (小时)
参与者(1)
1
https://gitee.com/ascend/modelzoo.git
git@gitee.com:ascend/modelzoo.git
ascend
modelzoo
modelzoo

搜索帮助