登录
注册
开源
企业版
高校版
搜索
帮助中心
使用条款
关于我们
开源
企业版
高校版
私有云
模力方舟
AI 队友
登录
注册
3月21日 深圳|OpenClaw 线下实战沙龙:招聘、资讯、项目协同三大场景实操,VS ZeroClaw 横向对比评测,别再只会装,来现场跑通真实业务!
代码拉取完成,页面将自动刷新
开源项目
>
人工智能
>
大模型
&&
捐赠
捐赠前请先登录
取消
前往登录
扫描微信二维码支付
取消
支付完成
支付提示
将跳转至支付宝完成支付
确定
取消
Watch
不关注
关注所有动态
仅关注版本发行动态
关注但不提醒动态
109
Star
893
Fork
1.4K
MindSpore
/
models
代码
Issues
120
Pull Requests
0
Wiki
统计
流水线
服务
JavaDoc
PHPDoc
质量分析
Jenkins for Gitee
腾讯云托管
腾讯云 Serverless
悬镜安全
阿里云 SAE
Codeblitz
SBOM
我知道了,不再自动展开
更新失败,请稍后重试!
移除标识
内容风险标识
本任务被
标识为内容中包含有代码安全 Bug 、隐私泄露等敏感信息,仓库外成员不可访问
Pangu_alpha模型数据转换报错
DONE
#I72DM1
Bug-Report
AmazingU
创建于
2023-05-11 17:41
环境信息:Atlas800-9000 arm Euler2.8 CANN:CANN6.0.RC1 Python3.7.5 [root@4a03d154aabc Pangu_alpha]# python3 -c "import mindspore as ms;ms.run_check()" MindSpore version: 1.9.0 The result of multiplication calculation is correct, MindSpore has been installed successfull  模型与代码:https://gitee.com/mindspore/models/tree/master/official/nlp/Pangu_alpha 数据目录 [root@4a03d154aabc Pangu_alpha]# ll data/ total 54172 -rw------- 1 root root 4152914 May 11 09:28 153331.txt -rw------- 1 root root 15301163 May 11 09:28 22298.txt -rw------- 1 root root 12125907 May 11 09:28 260878.txt -rw------- 1 root root 23886830 May 11 09:28 2.txt 执行命令: python3 -m src.preprocess --input_glob 'data/*.txt' --tokenizer gpt --eot 50256 --data_column_name input_ids --seq_length 1025 root@4a03d154aabc Pangu_alpha]# python3 -m src.preprocess --input_glob 'data/*.txt' --tokenizer gpt --eot 50256 --data_column_name input_ids --seq_length 1025 [WARNING] ME(8821:281473787678720,MainProcess):2023-05-11-09:28:49.481.980 [mindspore/mindrecord/filewriter.py:278] This interface will be deleted or invisible in the future. [ERROR] CORE(8892,ffffb920d000,python3):2023-05-11-09:34:20.005.200 [mindspore/core/utils/file_utils.cc:253] GetRealPath] Get realpath failed, path[/home/wukong/models/official/nlp/Pangu_alpha/output/transfered_mindrecord_Pages] multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/usr/local/python37/lib/python3.7/multiprocessing/pool.py", line 121, in worker result = (True, func(*args, **kwds)) File "/usr/local/python37/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar return list(map(*args)) File "/home/wukong/models/official/nlp/Pangu_alpha/src/preprocess.py", line 154, in task_unit writer.write_raw_data(data_batch, parallel_writer=parallel_writer) File "/usr/local/python37/lib/python3.7/site-packages/mindspore/mindrecord/filewriter.py", line 323, in write_raw_data return self._writer.write_raw_data(raw_data, True, parallel_writer) File "/usr/local/python37/lib/python3.7/site-packages/mindspore/mindrecord/shardwriter.py", line 178, in write_raw_data ret = self._writer.write_raw_data(raw_data, blob_data, validate, parallel_writer) RuntimeError: Unexpected error. [Internal ERROR] Error raised in FileToPages function. Line of code : 531 File : mindspore/ccsrc/minddata/mindrecord/io/shard_writer.cc """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/python37/lib/python3.7/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/local/python37/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/wukong/models/official/nlp/Pangu_alpha/src/preprocess.py", line 220, in <module> pool.map(map_func, package_file(file_iter, args.file_batch_size)) File "/usr/local/python37/lib/python3.7/multiprocessing/pool.py", line 268, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/usr/local/python37/lib/python3.7/multiprocessing/pool.py", line 657, in get raise self._value RuntimeError: Unexpected error. [Internal ERROR] Error raised in FileToPages function. Line of code : 531 File : mindspore/ccsrc/minddata/mindrecord/io/shard_writer.cc 
环境信息:Atlas800-9000 arm Euler2.8 CANN:CANN6.0.RC1 Python3.7.5 [root@4a03d154aabc Pangu_alpha]# python3 -c "import mindspore as ms;ms.run_check()" MindSpore version: 1.9.0 The result of multiplication calculation is correct, MindSpore has been installed successfull  模型与代码:https://gitee.com/mindspore/models/tree/master/official/nlp/Pangu_alpha 数据目录 [root@4a03d154aabc Pangu_alpha]# ll data/ total 54172 -rw------- 1 root root 4152914 May 11 09:28 153331.txt -rw------- 1 root root 15301163 May 11 09:28 22298.txt -rw------- 1 root root 12125907 May 11 09:28 260878.txt -rw------- 1 root root 23886830 May 11 09:28 2.txt 执行命令: python3 -m src.preprocess --input_glob 'data/*.txt' --tokenizer gpt --eot 50256 --data_column_name input_ids --seq_length 1025 root@4a03d154aabc Pangu_alpha]# python3 -m src.preprocess --input_glob 'data/*.txt' --tokenizer gpt --eot 50256 --data_column_name input_ids --seq_length 1025 [WARNING] ME(8821:281473787678720,MainProcess):2023-05-11-09:28:49.481.980 [mindspore/mindrecord/filewriter.py:278] This interface will be deleted or invisible in the future. [ERROR] CORE(8892,ffffb920d000,python3):2023-05-11-09:34:20.005.200 [mindspore/core/utils/file_utils.cc:253] GetRealPath] Get realpath failed, path[/home/wukong/models/official/nlp/Pangu_alpha/output/transfered_mindrecord_Pages] multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/usr/local/python37/lib/python3.7/multiprocessing/pool.py", line 121, in worker result = (True, func(*args, **kwds)) File "/usr/local/python37/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar return list(map(*args)) File "/home/wukong/models/official/nlp/Pangu_alpha/src/preprocess.py", line 154, in task_unit writer.write_raw_data(data_batch, parallel_writer=parallel_writer) File "/usr/local/python37/lib/python3.7/site-packages/mindspore/mindrecord/filewriter.py", line 323, in write_raw_data return self._writer.write_raw_data(raw_data, True, parallel_writer) File "/usr/local/python37/lib/python3.7/site-packages/mindspore/mindrecord/shardwriter.py", line 178, in write_raw_data ret = self._writer.write_raw_data(raw_data, blob_data, validate, parallel_writer) RuntimeError: Unexpected error. [Internal ERROR] Error raised in FileToPages function. Line of code : 531 File : mindspore/ccsrc/minddata/mindrecord/io/shard_writer.cc """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/python37/lib/python3.7/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/local/python37/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/wukong/models/official/nlp/Pangu_alpha/src/preprocess.py", line 220, in <module> pool.map(map_func, package_file(file_iter, args.file_batch_size)) File "/usr/local/python37/lib/python3.7/multiprocessing/pool.py", line 268, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/usr/local/python37/lib/python3.7/multiprocessing/pool.py", line 657, in get raise self._value RuntimeError: Unexpected error. [Internal ERROR] Error raised in FileToPages function. Line of code : 531 File : mindspore/ccsrc/minddata/mindrecord/io/shard_writer.cc 
评论 (
5
)
登录
后才可以发表评论
状态
DONE
TODO
ACCEPTED
WIP
VALIDATION
DONE
CLOSED
REJECTED
负责人
未设置
guozhijian
jonyguo
负责人
协作者
+负责人
+协作者
luoyang
luoyang42
负责人
协作者
+负责人
+协作者
标签
minddata
未设置
项目
未立项任务
未立项任务
里程碑
B-SIG-Data
未关联里程碑
Pull Requests
未关联
未关联
关联的 Pull Requests 被合并后可能会关闭此 issue
分支
未关联
分支 (17)
标签 (5)
master
r2.7.rc1
r2.3
r2.2
r2.1
r2.0
r1.8
r1.9
r2.0.0-alpha
r1.10
r1.7
r1.6
dynamic_shape
r1.5
r1.3
r1.4
r1.2
v2.2.11
v2.2.10
v2.2.1
v1.6.0
v1.5.0
开始日期   -   截止日期
-
置顶选项
不置顶
置顶等级:高
置顶等级:中
置顶等级:低
优先级
不指定
严重
主要
次要
不重要
预计工期
(小时)
参与者(6)
1
https://gitee.com/mindspore/models.git
git@gitee.com:mindspore/models.git
mindspore
models
models
点此查找更多帮助
搜索帮助
Git 命令在线学习
如何在 Gitee 导入 GitHub 仓库
Git 仓库基础操作
企业版和社区版功能对比
SSH 公钥设置
如何处理代码冲突
仓库体积过大,如何减小?
如何找回被删除的仓库数据
Gitee 产品配额说明
GitHub仓库快速导入Gitee及同步更新
什么是 Release(发行版)
将 PHP 项目自动发布到 packagist.org
仓库举报
回到顶部
登录提示
该操作需登录 Gitee 帐号,请先登录后再操作。
立即登录
没有帐号,去注册