登录
注册
开源
企业版
高校版
搜索
帮助中心
使用条款
关于我们
开源
企业版
高校版
私有云
模力方舟
AI 队友
登录
注册
轻量养虾,开箱即用!低 Token + 稳定算力,Gitee & 模力方舟联合出品的 PocketClaw 正式开售!点击了解详情~
代码拉取完成,页面将自动刷新
开源项目
>
人工智能
>
大模型
&&
捐赠
捐赠前请先登录
取消
前往登录
扫描微信二维码支付
取消
支付完成
支付提示
将跳转至支付宝完成支付
确定
取消
Watch
不关注
关注所有动态
仅关注版本发行动态
关注但不提醒动态
109
Star
894
Fork
1.4K
MindSpore
/
models
代码
Issues
120
Pull Requests
0
Wiki
统计
流水线
服务
JavaDoc
PHPDoc
质量分析
Jenkins for Gitee
腾讯云托管
腾讯云 Serverless
悬镜安全
阿里云 SAE
Codeblitz
SBOM
开发画像分析
我知道了,不再自动展开
更新失败,请稍后重试!
移除标识
内容风险标识
本任务被
标识为内容中包含有代码安全 Bug 、隐私泄露等敏感信息,仓库外成员不可访问
[Bug]:使用model-zoo上的slowfast模型跑训练,对于小批量数据单卡和8卡都可以,但是对于大批量150G的数据8卡训练报错
DONE
#I8NY66
奥德彪
创建于
2023-12-13 10:40
### 问题描述 使用model-zoo上的slowfast模型跑训练,对于小批量数据单卡和8卡都可以,但是对于大批量150G的数据8卡训练报错 ### 环境信息 - **Hardware Environment(`Ascend`/`GPU`/`CPU`) / 硬件环境**: Ascend > Please delete the backend not involved / 请删除不涉及的后端: > /device Ascend - **Software Environment / 软件环境 (Mandatory / 必填)**: -- MindSpore version (e.g., 2.0.0) :2.1.0 -- Python version (e.g., Python 3.7.5) :3.9.5 -- OS platform and distribution (e.g., Linux Ubuntu 16.04):Euler2.8.3 -- GCC/Compiler version (if compiled from source): - **Execute Mode / 执行模式 (Mandatory / 必填)(`PyNative`/`Graph`)**: PyNative > Please delete the mode not involved / 请删除不涉及的模式: > /mode pynative ### 关联用例 https://gitee.com/mindspore/models/tree/master/research/cv/slowfast#%E8%AE%AD%E7%BB%83%E8%BF%87%E7%A8%8B ### 重现步骤 1.代码为model-zoo上代码,https://gitee.com/mindspore/models/tree/master/research/cv/slowfast#%E8%AE%AD%E7%BB%83%E8%BF%87%E7%A8%8B 2.数据集为客户自定义数据据,大小由150G 3.参考的文档是model-zoo上的文档,按步骤来操作的 4.使用分布式8卡训练,报错EI0002: ### 预期结果 8卡分布式训练能够正常运行,生成相应的权重文件 ### 日志/截图 [图片上传中…(image-LjUtxE0Mi031aprP4eZ7)] [12/12 11:33:15][INFO] train.py: 96: AUG: AA_TYPE: rand-m9-mstd0.5-inc1 COLOR_JITTER: 0.4 ENABLE: False INTERPOLATION: bicubic NUM_SAMPLE: 1 RE_COUNT: 1 RE_MODE: pixel RE_PROB: 0.25 RE_SPLIT: False AVA: ANNOTATION_DIR: /home/ma-user/work/slowfast/data/ava/ava_annotations BGR: False DETECTION_SCORE_THRESH: 0.8 EXCLUSION_FILE: ava_val_excluded_timestamps_v2.2.csv FRAME_DIR: /home/ma-user/work/slowfast/data/ava/rawframes FRAME_LIST_DIR: /home/ma-user/work/slowfast/data/ava/ava_annotations FULL_TEST_ON_VAL: False GROUNDTRUTH_FILE: ava_val_v2.2.csv IMG_PROC_BACKEND: cv2 LABEL_MAP_FILE: ava_action_list_v2.1_for_activitynet_2018.pbtxt TEST_FORCE_FLIP: False TEST_LISTS: ['val.csv'] TEST_PREDICT_BOX_LISTS: ['ava_detection_val_boxes_and_labels.csv'] TRAIN_GT_BOX_LISTS: ['ava_train_v2.2.csv'] TRAIN_LISTS: ['train.csv'] TRAIN_PCA_JITTER_ONLY: True TRAIN_PREDICT_BOX_LISTS: ['ava_train_v2.2.csv', 'ava_detection_train_boxes_and_labels_include_negative_v2.2.csv'] TRAIN_USE_COLOR_AUGMENTATION: False BENCHMARK: LOG_PERIOD: 100 NUM_EPOCHS: 5 SHUFFLE: True BN: NORM_TYPE: batchnorm NUM_BATCHES_PRECISE: 200 NUM_SPLITS: 1 NUM_SYNC_DEVICES: 1 USE_PRECISE_STATS: False WEIGHT_DECAY: 0.0 DATA: DECODING_BACKEND: pyav ENSEMBLE_METHOD: sum INPUT_CHANNEL_NUM: [3, 3] INV_UNIFORM_SAMPLE: False MAX_NUM_BOXES_PER_FRAME: 28 MEAN: [0.45, 0.45, 0.45] MULTI_LABEL: False NUM_FRAMES: 32 PATH_LABEL_SEPARATOR: PATH_PREFIX: PATH_TO_DATA_DIR: PATH_TO_PRELOAD_IMDB: RANDOM_FLIP: True REVERSE_INPUT_CHANNEL: False SAMPLING_RATE: 2 STD: [0.225, 0.225, 0.225] TARGET_FPS: 25 TEST_CROP_SIZE: 224 TEST_SCALE_HEIGHT: 256 TEST_SCALE_WIDTH: 384 TRAIN_CROP_SIZE: 224 TRAIN_JITTER_ASPECT_RELATIVE: [] TRAIN_JITTER_MOTION_SHIFT: False TRAIN_JITTER_SCALES: [256, 320] TRAIN_JITTER_SCALES_RELATIVE: [] TRAIN_PCA_EIGVAL: [0.225, 0.224, 0.229] TRAIN_PCA_EIGVEC: [[-0.5675, 0.7192, 0.4009], [-0.5808, -0.0045, -0.814], [-0.5836, -0.6948, 0.4203]] USE_OFFSET_SAMPLING: False DATA_LOADER: ENABLE_MULTI_THREAD_DECODE: False NUM_WORKERS: 2 PIN_MEMORY: True DEMO: BUFFER_SIZE: 0 CLIP_VIS_SIZE: 10 COMMON_CLASS_NAMES: ['watch (a person)', 'talk to (e.g., self, a person, a group)', 'listen to (a person)', 'touch (an object)', 'carry/hold (an object)', 'walk', 'sit', 'lie/sleep', 'bend/bow (at the waist)'] COMMON_CLASS_THRES: 0.7 DETECTRON2_CFG: COCO-Detection/faster_rcnn_R_50_FPN_3x.yaml DETECTRON2_THRESH: 0.9 DETECTRON2_WEIGHTS: detectron2://COCO-Detection/faster_rcnn_R_50_FPN_3x/137849458/model_final_280758.pkl DISPLAY_HEIGHT: 0 DISPLAY_WIDTH: 0 ENABLE: False FPS: 25 GT_BOXES: INPUT_FORMAT: BGR INPUT_VIDEO: LABEL_FILE_PATH: NUM_CLIPS_SKIP: 0 NUM_VIS_INSTANCES: 2 OUTPUT_FILE: OUTPUT_FPS: -1 PREDS_BOXES: SLOWMO: 1 STARTING_SECOND: 0 THREAD_ENABLE: False UNCOMMON_CLASS_THRES: 0.3 VIS_MODE: thres WEBCAM: -1 DETECTION: ALIGNED: True ENABLE: True ROI_XFORM_RESOLUTION: 7 SPATIAL_SCALE_FACTOR: 16 LOG_MODEL_INFO: True LOG_PERIOD: 1 MIXUP: ALPHA: 0.8 CUTMIX_ALPHA: 1.0 ENABLE: False LABEL_SMOOTH_VALUE: 0.1 PROB: 1.0 SWITCH_PROB: 0.5 MODEL: ACT_CHECKPOINT: False ARCH: slowfast DROPCONNECT_RATE: 0.0 DROPOUT_RATE: 0.5 FC_INIT_STD: 0.01 HEAD_ACT: sigmoid LOSS_FUNC: bce MODEL_NAME: SlowFast MULTI_PATHWAY_ARCH: ['slowfast'] NUM_CLASSES: 80 SINGLE_PATHWAY_ARCH: ['2d', 'c2d', 'i3d', 'slow', 'x3d', 'mvit'] MULTIGRID: BN_BASE_SIZE: 8 DEFAULT_B: 0 DEFAULT_S: 0 DEFAULT_T: 0 EPOCH_FACTOR: 1.5 EVAL_FREQ: 3 LONG_CYCLE: False LONG_CYCLE_FACTORS: [(0.25, 0.7071067811865476), (0.5, 0.7071067811865476), (0.5, 1), (1, 1)] LONG_CYCLE_SAMPLING_RATE: 0 SHORT_CYCLE: False SHORT_CYCLE_FACTORS: [0.5, 0.7071067811865476] MVIT: CLS_EMBED_ON: True DEPTH: 16 DIM_MUL: [] DROPOUT_RATE: 0.0 DROPPATH_RATE: 0.1 EMBED_DIM: 96 HEAD_MUL: [] MLP_RATIO: 4.0 MODE: conv NORM: layernorm NORM_STEM: False NUM_HEADS: 1 PATCH_2D: False PATCH_KERNEL: [3, 7, 7] PATCH_PADDING: [2, 4, 4] PATCH_STRIDE: [2, 4, 4] POOL_FIRST: False POOL_KVQ_KERNEL: None POOL_KV_STRIDE: None POOL_KV_STRIDE_ADAPTIVE: None POOL_Q_STRIDE: [] QKV_BIAS: True SEP_POS_EMBED: False ZERO_DECAY_POS_CLS: True NONLOCAL: GROUP: [[1, 1], [1, 1], [1, 1], [1, 1]] INSTANTIATION: dot_product LOCATION: [[[], []], [[], []], [[], []], [[], []]] POOL: [[[1, 2, 2], [1, 2, 2]], [[1, 2, 2], [1, 2, 2]], [[1, 2, 2], [1, 2, 2]], [[1, 2, 2], [1, 2, 2]]] NUM_GPUS: 8 NUM_SHARDS: 1 OUTPUT_DIR: . RESNET: DEPTH: 50 INPLACE_RELU: True NUM_BLOCK_TEMP_KERNEL: [[3, 3], [4, 4], [6, 6], [3, 3]] NUM_GROUPS: 1 SPATIAL_DILATIONS: [[1, 1], [1, 1], [1, 1], [2, 2]] SPATIAL_STRIDES: [[1, 1], [2, 2], [2, 2], [1, 1]] STRIDE_1X1: False TRANS_FUNC: bottleneck_transform WIDTH_PER_GROUP: 64 ZERO_INIT_FINAL_BN: True RNG_SEED: 0 SHARD_ID: 0 SLOWFAST: ALPHA: 4 BETA_INV: 8 FUSION_CONV_CHANNEL_RATIO: 2 FUSION_KERNEL_SZ: 7 SOLVER: BASE_LR: 0.15 BASE_LR_SCALE_NUM_SHARDS: False CLIP_GRAD_L2NORM: None CLIP_GRAD_VAL: None COSINE_AFTER_WARMUP: False COSINE_END_LR: 0.0 DAMPENING: 0.0 GAMMA: 0.1 LRS: [1, 0.1, 0.01, 0.001] LR_POLICY: steps_with_relative_lrs MAX_EPOCH: 20 MOMENTUM: 0.9 NESTEROV: False OPTIMIZING_METHOD: sgd STEPS: [0, 10, 15, 20] STEP_SIZE: 1 WARMUP_EPOCHS: 5.0 WARMUP_FACTOR: 0.1 WARMUP_START_LR: 0.000125 WEIGHT_DECAY: 1e-07 ZERO_WD_1D_PARAM: False TENSORBOARD: CATEGORIES_PATH: CLASS_NAMES_PATH: CONFUSION_MATRIX: ENABLE: False FIGSIZE: [8, 8] SUBSET_PATH: ENABLE: False HISTOGRAM: ENABLE: False FIGSIZE: [8, 8] SUBSET_PATH: TOPK: 10 LOG_DIR: MODEL_VIS: ACTIVATIONS: False COLORMAP: Pastel2 ENABLE: False GRAD_CAM: COLORMAP: viridis ENABLE: True LAYER_LIST: [] USE_TRUE_LABEL: False INPUT_VIDEO: False LAYER_LIST: [] MODEL_WEIGHTS: False TOPK_PREDS: 1 PREDICTIONS_PATH: WRONG_PRED_VIS: ENABLE: False SUBSET_PATH: TAG: Incorrectly classified videos. TEST: BATCH_SIZE: 8 CHECKPOINT_FILE_PATH: CHECKPOINT_TYPE: pytorch DATASET: ava ENABLE: True NUM_ENSEMBLE_VIEWS: 10 NUM_SPATIAL_CROPS: 3 SAVE_BINS_RESULTS_PATH: ./preprocess_Result SAVE_RESULTS_PATH: TRAIN: AUTO_RESUME: True BATCH_SIZE: 8 CHECKPOINT_CLEAR_NAME_PATTERN: () CHECKPOINT_EPOCH_RESET: False CHECKPOINT_FILE_PATH: /home/ma-user/work/slowfast/SLOWFAST_8x8_R50.pkl.ckpt CHECKPOINT_INFLATE: False CHECKPOINT_PERIOD: 1 CHECKPOINT_TYPE: caffe2 DATASET: ava ENABLE: True EVAL_PERIOD: 5 MIXED_PRECISION: False X3D: BN_LIN5: False BOTTLENECK_FACTOR: 1.0 CHANNELWISE_3x3x3: True DEPTH_FACTOR: 1.0 DIM_C1: 12 DIM_C5: 2048 SCALE_RES2: False WIDTH_FACTOR: 1.0 [12/12 11:33:18][INFO] ava_helper.py: 74: Finished loading image paths from: ['/home/ma-user/work/slowfast/data/ava/ava_annotations/train.csv'],len is 346 ******************** ['/home/ma-user/work/slowfast/data/ava/ava_annotations/ava_train_v2.2.csv', '/home/ma-user/work/slowfast/data/ava/ava_annotations/ava_train_v2.2.csv', '/home/ma-user/work/slowfast/data/ava/ava_annotations/ava_detection_train_boxes_and_labels_include_negative_v2.2.csv'] [True, False, False] 0.8 1 [12/12 11:33:19][INFO] ava_helper.py: 126: Finished loading annotations from: ['/home/ma-user/work/slowfast/data/ava/ava_annotations/ava_train_v2.2.csv', '/home/ma-user/work/slowfast/data/ava/ava_annotations/ava_train_v2.2.csv', '/home/ma-user/work/slowfast/data/ava/ava_annotations/ava_detection_train_boxes_and_labels_include_negative_v2.2.csv'], [12/12 11:33:19][INFO] ava_helper.py: 129: Detection threshold: 0.8 [12/12 11:33:19][INFO] ava_helper.py: 130: Number of unique boxes: 20357 [12/12 11:33:19][INFO] ava_helper.py: 131: Number of annotations: 59989 [12/12 11:33:19][INFO] ava_helper.py: 176: 20253 keyframes used. [12/12 11:33:19][INFO] ava_dataset.py: 111: === AVA dataset summary === [12/12 11:33:19][INFO] ava_dataset.py: 112: Split: train [12/12 11:33:19][INFO] ava_dataset.py: 113: Number of videos: 346 [12/12 11:33:19][INFO] ava_dataset.py: 117: Number of frames: 533209 [12/12 11:33:19][INFO] ava_dataset.py: 118: Number of key frames: <src.datasets.ava_dataset.Ava object at 0xffff41857340> [12/12 11:33:19][INFO] ava_dataset.py: 119: Number of boxes: 20357. loading /home/ma-user/work/slowfast/SLOWFAST_8x8_R50.pkl.ckpt [WARNING] ME(299854:281473426987936,MainProcess):2023-12-12-11:33:22.584.587 [mindspore/train/serialization.py:1237] For 'load_param_into_net', 2 parameters in the 'net' are not loaded, because they are not in the 'parameter_dict', please check whether the network structure is consistent when training and loading checkpoint. [WARNING] ME(299854:281473426987936,MainProcess):2023-12-12-11:33:22.584.805 [mindspore/train/serialization.py:1242] net.head.projection.weight is not loaded. [WARNING] ME(299854:281473426987936,MainProcess):2023-12-12-11:33:22.584.874 [mindspore/train/serialization.py:1242] net.head.projection.bias is not loaded. [12/12 11:33:23][INFO] train.py: 134: ============== Starting Training ============== [12/12 11:33:23][INFO] train.py: 135: total_epoch=20, steps_per_epoch=317 Training.20 <mindspore.dataset.engine.datasets.BatchDataset object at 0xffff41895610> [WARNING] CORE(299854,ffffa3a11ba0,python):2023-12-12-11:33:23.545.398 [mindspore/core/utils/numa_interface.cc:145] NumaBind] Try to bind numa id: 0, but execute set_mempolicy failed, errno: Operation not permitted. Please use mindspore.dataset.config.set_numa_enable(False) to disable numa bind. [WARNING] CORE(299854,ffffa3a11ba0,python):2023-12-12-11:33:26.596.314 [mindspore/core/utils/numa_interface.cc:145] NumaBind] Try to bind numa id: 0, but execute set_mempolicy failed, errno: Operation not permitted. Please use mindspore.dataset.config.set_numa_enable(False) to disable numa bind. [WARNING] CORE(299854,ffffa3a11ba0,python):2023-12-12-11:33:57.176.550 [mindspore/core/utils/numa_interface.cc:145] NumaBind] Try to bind numa id: 0, but execute set_mempolicy failed, errno: Operation not permitted. Please use mindspore.dataset.config.set_numa_enable(False) to disable numa bind. [WARNING] MD(299854,fffba3fff1e0,python):2023-12-12-11:38:20.901.966 [mindspore/ccsrc/minddata/dataset/engine/datasetops/data_queue_op.cc:912] DetectPerBatchTime] Bad performance attention, it takes more than 25 seconds to fetch a batch of data from dataset pipeline, which might result `GetNext` timeout problem. You may test dataset processing performance(with creating dataset iterator) and optimize it. [WARNING] DEVICE(299854,fffd2cff91e0,python):2023-12-12-12:04:07.125.519 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:801] GetDumpPath] The environment variable 'MS_OM_PATH' is not set, the files will save to the process local path, as ./rank_id/node_dump/... [ERROR] DEVICE(299854,fffd2cff91e0,python):2023-12-12-12:04:07.125.659 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:819] PrintDebugInfoAndDumpFailNode] Task fail infos, rt task_id: 14, rt stream_id: 7, tid: 299854, device_id: 0, retcode: 507011 ( model execute failed) [WARNING] DEVICE(299854,fffd2cff91e0,python):2023-12-12-12:04:07.129.962 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:801] GetDumpPath] The environment variable 'MS_OM_PATH' is not set, the files will save to the process local path, as ./rank_id/exec_order/... [ERROR] DEVICE(299854,fffd2cff91e0,python):2023-12-12-12:04:07.220.624 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:906] DumpDebugInfoFile] Execute order has saved at /home/ma-user/work/slowfast/train_parallel0/rank_0/exec_order/kernel_graph_1.csv Traceback (most recent call last): File "/home/ma-user/work/slowfast/train_parallel0/train.py", line 139, in <module> train() File "/home/ma-user/work/slowfast/train_parallel0/train.py", line 136, in train model.train(cfg.SOLVER.MAX_EPOCH, dataset, callbacks=callbacks, dataset_sink_mode=bool(args.dataset_sink_mode)) File "/home/ma-user/anaconda3/envs/python3.9/lib/python3.9/site-packages/mindspore/train/model.py", line 1063, in train self._train(epoch, File "/home/ma-user/anaconda3/envs/python3.9/lib/python3.9/site-packages/mindspore/train/model.py", line 113, in wrapper func(self, *args, **kwargs) File "/home/ma-user/anaconda3/envs/python3.9/lib/python3.9/site-packages/mindspore/train/model.py", line 621, in _train self._train_dataset_sink_process(epoch, train_dataset, list_callback, File "/home/ma-user/anaconda3/envs/python3.9/lib/python3.9/site-packages/mindspore/train/model.py", line 705, in _train_dataset_sink_process outputs = train_network(*inputs) File "/home/ma-user/anaconda3/envs/python3.9/lib/python3.9/site-packages/mindspore/nn/cell.py", line 639, in __call__ out = self.compile_and_run(*args, **kwargs) File "/home/ma-user/anaconda3/envs/python3.9/lib/python3.9/site-packages/mindspore/nn/cell.py", line 966, in compile_and_run return _cell_graph_executor(self, *new_args, phase=self.phase) File "/home/ma-user/anaconda3/envs/python3.9/lib/python3.9/site-packages/mindspore/common/api.py", line 1674, in __call__ return self.run(obj, *args, phase=phase) File "/home/ma-user/anaconda3/envs/python3.9/lib/python3.9/site-packages/mindspore/common/api.py", line 1713, in run return self._exec_pip(obj, *args, phase=phase_real) File "/home/ma-user/anaconda3/envs/python3.9/lib/python3.9/site-packages/mindspore/common/api.py", line 106, in wrapper results = fn(*arg, **kwargs) File "/home/ma-user/anaconda3/envs/python3.9/lib/python3.9/site-packages/mindspore/common/api.py", line 1693, in _exec_pip return self._graph_executor(args, phase) RuntimeError: Run task for graph:kernel_graph_1 error! The details refer to 'Ascend Error Message'. ---------------------------------------------------- - Ascend Error Message: ---------------------------------------------------- EI0002: The wait execution of the Notify register times out. Reason: The Notify register has not received the Notify record from remote rank [4].base information: [streamID:[7], taskID[14], taskType[Notify Wait], tag[HcomAllReduce_6629421139219749105_0].] task information: [notify id:[0x00000000000000a0], stage:[ffffffff], remote rank:[4].] Possible Cause: 1. An exception occurs during the execution on some NPUs in the cluster. As a result, collective communication operation failed.2. The execution speed on some NPU in the cluster is too slow to complete a communication operation within the timeout interval. (default 1800s, You can set the interval by using HCCL_EXEC_TIMEOUT.)3. The number of training samples of each NPU is inconsistent.4. Packet loss or other connectivity problems occur on the communication link. Solution: 1. If this error is reported on part of these ranks, check other ranks to see whether other errors have been reported earlier.2. If this error is reported for all ranks, check whether the error reporting time is consistent (the maximum difference must not exceed 1800s). If not, locate the cause or adjust the locate the cause or set the HCCL_EXEC_TIMEOUT environment variable to a larger value.3. Check whether the completion queue element (CQE) of the error exists in the plog(grep -rn 'error cqe'). If so, check the network connection status. (For details, see the TLS command and HCCN connectivity check examples.)4. Ensure that the number of training samples of each NPU is consistent. For details:https://www.hiascend.com/document TraceBack (most recent call last): Notify wait execute failed, device_id=0, stream_id=7, task_id=14, flip_num=0, notify_id=20[FUNC:GetError][FILE:stream.cc][LINE:1418] rtStreamSynchronize execute failed, reason=[the model stream execute failed][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:50] (Please search "Ascend Error Message" at https://www.mindspore.cn for error code description) ---------------------------------------------------- - C++ Call Stack: (For framework developers) ---------------------------------------------------- mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_graph_executor.cc:296 RunGraph [WARNING] MD(299854,ffffa3a11ba0,python):2023-12-12-12:04:09.257.359 [mindspore/ccsrc/minddata/dataset/engine/datasetops/data_queue_op.cc:115] ~DataQueueOp] preprocess_batch: 28; batch_queue: 0, 0, 0, 0, 0, 0, 0, 0, 0, 0; push_start_time -> push_end_time 2023-12-12-11:38:49.190.646 -> 2023-12-12-11:38:49.206.462 2023-12-12-11:38:49.704.597 -> 2023-12-12-11:38:49.717.272 2023-12-12-11:38:53.740.464 -> 2023-12-12-11:38:53.753.396 2023-12-12-11:38:54.816.448 -> 2023-12-12-11:38:54.838.486 2023-12-12-11:38:58.286.482 -> 2023-12-12-11:38:58.301.204 2023-12-12-11:38:59.418.431 -> 2023-12-12-11:38:59.432.061 2023-12-12-11:39:02.766.988 -> 2023-12-12-11:39:02.783.474 2023-12-12-11:39:03.989.076 -> 2023-12-12-11:39:04.003.052 2023-12-12-11:39:06.062.494 -> 2023-12-12-11:39:06.085.283 2023-12-12-11:39:07.613.613 -> 2023-12-12-11:39:07.626.919 For more details, please refer to the FAQ at https://www.mindspore.cn/docs/en/master/faq/data_processing.html. ### 备注 麻烦帮忙看一下,谢谢
### 问题描述 使用model-zoo上的slowfast模型跑训练,对于小批量数据单卡和8卡都可以,但是对于大批量150G的数据8卡训练报错 ### 环境信息 - **Hardware Environment(`Ascend`/`GPU`/`CPU`) / 硬件环境**: Ascend > Please delete the backend not involved / 请删除不涉及的后端: > /device Ascend - **Software Environment / 软件环境 (Mandatory / 必填)**: -- MindSpore version (e.g., 2.0.0) :2.1.0 -- Python version (e.g., Python 3.7.5) :3.9.5 -- OS platform and distribution (e.g., Linux Ubuntu 16.04):Euler2.8.3 -- GCC/Compiler version (if compiled from source): - **Execute Mode / 执行模式 (Mandatory / 必填)(`PyNative`/`Graph`)**: PyNative > Please delete the mode not involved / 请删除不涉及的模式: > /mode pynative ### 关联用例 https://gitee.com/mindspore/models/tree/master/research/cv/slowfast#%E8%AE%AD%E7%BB%83%E8%BF%87%E7%A8%8B ### 重现步骤 1.代码为model-zoo上代码,https://gitee.com/mindspore/models/tree/master/research/cv/slowfast#%E8%AE%AD%E7%BB%83%E8%BF%87%E7%A8%8B 2.数据集为客户自定义数据据,大小由150G 3.参考的文档是model-zoo上的文档,按步骤来操作的 4.使用分布式8卡训练,报错EI0002: ### 预期结果 8卡分布式训练能够正常运行,生成相应的权重文件 ### 日志/截图 [图片上传中…(image-LjUtxE0Mi031aprP4eZ7)] [12/12 11:33:15][INFO] train.py: 96: AUG: AA_TYPE: rand-m9-mstd0.5-inc1 COLOR_JITTER: 0.4 ENABLE: False INTERPOLATION: bicubic NUM_SAMPLE: 1 RE_COUNT: 1 RE_MODE: pixel RE_PROB: 0.25 RE_SPLIT: False AVA: ANNOTATION_DIR: /home/ma-user/work/slowfast/data/ava/ava_annotations BGR: False DETECTION_SCORE_THRESH: 0.8 EXCLUSION_FILE: ava_val_excluded_timestamps_v2.2.csv FRAME_DIR: /home/ma-user/work/slowfast/data/ava/rawframes FRAME_LIST_DIR: /home/ma-user/work/slowfast/data/ava/ava_annotations FULL_TEST_ON_VAL: False GROUNDTRUTH_FILE: ava_val_v2.2.csv IMG_PROC_BACKEND: cv2 LABEL_MAP_FILE: ava_action_list_v2.1_for_activitynet_2018.pbtxt TEST_FORCE_FLIP: False TEST_LISTS: ['val.csv'] TEST_PREDICT_BOX_LISTS: ['ava_detection_val_boxes_and_labels.csv'] TRAIN_GT_BOX_LISTS: ['ava_train_v2.2.csv'] TRAIN_LISTS: ['train.csv'] TRAIN_PCA_JITTER_ONLY: True TRAIN_PREDICT_BOX_LISTS: ['ava_train_v2.2.csv', 'ava_detection_train_boxes_and_labels_include_negative_v2.2.csv'] TRAIN_USE_COLOR_AUGMENTATION: False BENCHMARK: LOG_PERIOD: 100 NUM_EPOCHS: 5 SHUFFLE: True BN: NORM_TYPE: batchnorm NUM_BATCHES_PRECISE: 200 NUM_SPLITS: 1 NUM_SYNC_DEVICES: 1 USE_PRECISE_STATS: False WEIGHT_DECAY: 0.0 DATA: DECODING_BACKEND: pyav ENSEMBLE_METHOD: sum INPUT_CHANNEL_NUM: [3, 3] INV_UNIFORM_SAMPLE: False MAX_NUM_BOXES_PER_FRAME: 28 MEAN: [0.45, 0.45, 0.45] MULTI_LABEL: False NUM_FRAMES: 32 PATH_LABEL_SEPARATOR: PATH_PREFIX: PATH_TO_DATA_DIR: PATH_TO_PRELOAD_IMDB: RANDOM_FLIP: True REVERSE_INPUT_CHANNEL: False SAMPLING_RATE: 2 STD: [0.225, 0.225, 0.225] TARGET_FPS: 25 TEST_CROP_SIZE: 224 TEST_SCALE_HEIGHT: 256 TEST_SCALE_WIDTH: 384 TRAIN_CROP_SIZE: 224 TRAIN_JITTER_ASPECT_RELATIVE: [] TRAIN_JITTER_MOTION_SHIFT: False TRAIN_JITTER_SCALES: [256, 320] TRAIN_JITTER_SCALES_RELATIVE: [] TRAIN_PCA_EIGVAL: [0.225, 0.224, 0.229] TRAIN_PCA_EIGVEC: [[-0.5675, 0.7192, 0.4009], [-0.5808, -0.0045, -0.814], [-0.5836, -0.6948, 0.4203]] USE_OFFSET_SAMPLING: False DATA_LOADER: ENABLE_MULTI_THREAD_DECODE: False NUM_WORKERS: 2 PIN_MEMORY: True DEMO: BUFFER_SIZE: 0 CLIP_VIS_SIZE: 10 COMMON_CLASS_NAMES: ['watch (a person)', 'talk to (e.g., self, a person, a group)', 'listen to (a person)', 'touch (an object)', 'carry/hold (an object)', 'walk', 'sit', 'lie/sleep', 'bend/bow (at the waist)'] COMMON_CLASS_THRES: 0.7 DETECTRON2_CFG: COCO-Detection/faster_rcnn_R_50_FPN_3x.yaml DETECTRON2_THRESH: 0.9 DETECTRON2_WEIGHTS: detectron2://COCO-Detection/faster_rcnn_R_50_FPN_3x/137849458/model_final_280758.pkl DISPLAY_HEIGHT: 0 DISPLAY_WIDTH: 0 ENABLE: False FPS: 25 GT_BOXES: INPUT_FORMAT: BGR INPUT_VIDEO: LABEL_FILE_PATH: NUM_CLIPS_SKIP: 0 NUM_VIS_INSTANCES: 2 OUTPUT_FILE: OUTPUT_FPS: -1 PREDS_BOXES: SLOWMO: 1 STARTING_SECOND: 0 THREAD_ENABLE: False UNCOMMON_CLASS_THRES: 0.3 VIS_MODE: thres WEBCAM: -1 DETECTION: ALIGNED: True ENABLE: True ROI_XFORM_RESOLUTION: 7 SPATIAL_SCALE_FACTOR: 16 LOG_MODEL_INFO: True LOG_PERIOD: 1 MIXUP: ALPHA: 0.8 CUTMIX_ALPHA: 1.0 ENABLE: False LABEL_SMOOTH_VALUE: 0.1 PROB: 1.0 SWITCH_PROB: 0.5 MODEL: ACT_CHECKPOINT: False ARCH: slowfast DROPCONNECT_RATE: 0.0 DROPOUT_RATE: 0.5 FC_INIT_STD: 0.01 HEAD_ACT: sigmoid LOSS_FUNC: bce MODEL_NAME: SlowFast MULTI_PATHWAY_ARCH: ['slowfast'] NUM_CLASSES: 80 SINGLE_PATHWAY_ARCH: ['2d', 'c2d', 'i3d', 'slow', 'x3d', 'mvit'] MULTIGRID: BN_BASE_SIZE: 8 DEFAULT_B: 0 DEFAULT_S: 0 DEFAULT_T: 0 EPOCH_FACTOR: 1.5 EVAL_FREQ: 3 LONG_CYCLE: False LONG_CYCLE_FACTORS: [(0.25, 0.7071067811865476), (0.5, 0.7071067811865476), (0.5, 1), (1, 1)] LONG_CYCLE_SAMPLING_RATE: 0 SHORT_CYCLE: False SHORT_CYCLE_FACTORS: [0.5, 0.7071067811865476] MVIT: CLS_EMBED_ON: True DEPTH: 16 DIM_MUL: [] DROPOUT_RATE: 0.0 DROPPATH_RATE: 0.1 EMBED_DIM: 96 HEAD_MUL: [] MLP_RATIO: 4.0 MODE: conv NORM: layernorm NORM_STEM: False NUM_HEADS: 1 PATCH_2D: False PATCH_KERNEL: [3, 7, 7] PATCH_PADDING: [2, 4, 4] PATCH_STRIDE: [2, 4, 4] POOL_FIRST: False POOL_KVQ_KERNEL: None POOL_KV_STRIDE: None POOL_KV_STRIDE_ADAPTIVE: None POOL_Q_STRIDE: [] QKV_BIAS: True SEP_POS_EMBED: False ZERO_DECAY_POS_CLS: True NONLOCAL: GROUP: [[1, 1], [1, 1], [1, 1], [1, 1]] INSTANTIATION: dot_product LOCATION: [[[], []], [[], []], [[], []], [[], []]] POOL: [[[1, 2, 2], [1, 2, 2]], [[1, 2, 2], [1, 2, 2]], [[1, 2, 2], [1, 2, 2]], [[1, 2, 2], [1, 2, 2]]] NUM_GPUS: 8 NUM_SHARDS: 1 OUTPUT_DIR: . RESNET: DEPTH: 50 INPLACE_RELU: True NUM_BLOCK_TEMP_KERNEL: [[3, 3], [4, 4], [6, 6], [3, 3]] NUM_GROUPS: 1 SPATIAL_DILATIONS: [[1, 1], [1, 1], [1, 1], [2, 2]] SPATIAL_STRIDES: [[1, 1], [2, 2], [2, 2], [1, 1]] STRIDE_1X1: False TRANS_FUNC: bottleneck_transform WIDTH_PER_GROUP: 64 ZERO_INIT_FINAL_BN: True RNG_SEED: 0 SHARD_ID: 0 SLOWFAST: ALPHA: 4 BETA_INV: 8 FUSION_CONV_CHANNEL_RATIO: 2 FUSION_KERNEL_SZ: 7 SOLVER: BASE_LR: 0.15 BASE_LR_SCALE_NUM_SHARDS: False CLIP_GRAD_L2NORM: None CLIP_GRAD_VAL: None COSINE_AFTER_WARMUP: False COSINE_END_LR: 0.0 DAMPENING: 0.0 GAMMA: 0.1 LRS: [1, 0.1, 0.01, 0.001] LR_POLICY: steps_with_relative_lrs MAX_EPOCH: 20 MOMENTUM: 0.9 NESTEROV: False OPTIMIZING_METHOD: sgd STEPS: [0, 10, 15, 20] STEP_SIZE: 1 WARMUP_EPOCHS: 5.0 WARMUP_FACTOR: 0.1 WARMUP_START_LR: 0.000125 WEIGHT_DECAY: 1e-07 ZERO_WD_1D_PARAM: False TENSORBOARD: CATEGORIES_PATH: CLASS_NAMES_PATH: CONFUSION_MATRIX: ENABLE: False FIGSIZE: [8, 8] SUBSET_PATH: ENABLE: False HISTOGRAM: ENABLE: False FIGSIZE: [8, 8] SUBSET_PATH: TOPK: 10 LOG_DIR: MODEL_VIS: ACTIVATIONS: False COLORMAP: Pastel2 ENABLE: False GRAD_CAM: COLORMAP: viridis ENABLE: True LAYER_LIST: [] USE_TRUE_LABEL: False INPUT_VIDEO: False LAYER_LIST: [] MODEL_WEIGHTS: False TOPK_PREDS: 1 PREDICTIONS_PATH: WRONG_PRED_VIS: ENABLE: False SUBSET_PATH: TAG: Incorrectly classified videos. TEST: BATCH_SIZE: 8 CHECKPOINT_FILE_PATH: CHECKPOINT_TYPE: pytorch DATASET: ava ENABLE: True NUM_ENSEMBLE_VIEWS: 10 NUM_SPATIAL_CROPS: 3 SAVE_BINS_RESULTS_PATH: ./preprocess_Result SAVE_RESULTS_PATH: TRAIN: AUTO_RESUME: True BATCH_SIZE: 8 CHECKPOINT_CLEAR_NAME_PATTERN: () CHECKPOINT_EPOCH_RESET: False CHECKPOINT_FILE_PATH: /home/ma-user/work/slowfast/SLOWFAST_8x8_R50.pkl.ckpt CHECKPOINT_INFLATE: False CHECKPOINT_PERIOD: 1 CHECKPOINT_TYPE: caffe2 DATASET: ava ENABLE: True EVAL_PERIOD: 5 MIXED_PRECISION: False X3D: BN_LIN5: False BOTTLENECK_FACTOR: 1.0 CHANNELWISE_3x3x3: True DEPTH_FACTOR: 1.0 DIM_C1: 12 DIM_C5: 2048 SCALE_RES2: False WIDTH_FACTOR: 1.0 [12/12 11:33:18][INFO] ava_helper.py: 74: Finished loading image paths from: ['/home/ma-user/work/slowfast/data/ava/ava_annotations/train.csv'],len is 346 ******************** ['/home/ma-user/work/slowfast/data/ava/ava_annotations/ava_train_v2.2.csv', '/home/ma-user/work/slowfast/data/ava/ava_annotations/ava_train_v2.2.csv', '/home/ma-user/work/slowfast/data/ava/ava_annotations/ava_detection_train_boxes_and_labels_include_negative_v2.2.csv'] [True, False, False] 0.8 1 [12/12 11:33:19][INFO] ava_helper.py: 126: Finished loading annotations from: ['/home/ma-user/work/slowfast/data/ava/ava_annotations/ava_train_v2.2.csv', '/home/ma-user/work/slowfast/data/ava/ava_annotations/ava_train_v2.2.csv', '/home/ma-user/work/slowfast/data/ava/ava_annotations/ava_detection_train_boxes_and_labels_include_negative_v2.2.csv'], [12/12 11:33:19][INFO] ava_helper.py: 129: Detection threshold: 0.8 [12/12 11:33:19][INFO] ava_helper.py: 130: Number of unique boxes: 20357 [12/12 11:33:19][INFO] ava_helper.py: 131: Number of annotations: 59989 [12/12 11:33:19][INFO] ava_helper.py: 176: 20253 keyframes used. [12/12 11:33:19][INFO] ava_dataset.py: 111: === AVA dataset summary === [12/12 11:33:19][INFO] ava_dataset.py: 112: Split: train [12/12 11:33:19][INFO] ava_dataset.py: 113: Number of videos: 346 [12/12 11:33:19][INFO] ava_dataset.py: 117: Number of frames: 533209 [12/12 11:33:19][INFO] ava_dataset.py: 118: Number of key frames: <src.datasets.ava_dataset.Ava object at 0xffff41857340> [12/12 11:33:19][INFO] ava_dataset.py: 119: Number of boxes: 20357. loading /home/ma-user/work/slowfast/SLOWFAST_8x8_R50.pkl.ckpt [WARNING] ME(299854:281473426987936,MainProcess):2023-12-12-11:33:22.584.587 [mindspore/train/serialization.py:1237] For 'load_param_into_net', 2 parameters in the 'net' are not loaded, because they are not in the 'parameter_dict', please check whether the network structure is consistent when training and loading checkpoint. [WARNING] ME(299854:281473426987936,MainProcess):2023-12-12-11:33:22.584.805 [mindspore/train/serialization.py:1242] net.head.projection.weight is not loaded. [WARNING] ME(299854:281473426987936,MainProcess):2023-12-12-11:33:22.584.874 [mindspore/train/serialization.py:1242] net.head.projection.bias is not loaded. [12/12 11:33:23][INFO] train.py: 134: ============== Starting Training ============== [12/12 11:33:23][INFO] train.py: 135: total_epoch=20, steps_per_epoch=317 Training.20 <mindspore.dataset.engine.datasets.BatchDataset object at 0xffff41895610> [WARNING] CORE(299854,ffffa3a11ba0,python):2023-12-12-11:33:23.545.398 [mindspore/core/utils/numa_interface.cc:145] NumaBind] Try to bind numa id: 0, but execute set_mempolicy failed, errno: Operation not permitted. Please use mindspore.dataset.config.set_numa_enable(False) to disable numa bind. [WARNING] CORE(299854,ffffa3a11ba0,python):2023-12-12-11:33:26.596.314 [mindspore/core/utils/numa_interface.cc:145] NumaBind] Try to bind numa id: 0, but execute set_mempolicy failed, errno: Operation not permitted. Please use mindspore.dataset.config.set_numa_enable(False) to disable numa bind. [WARNING] CORE(299854,ffffa3a11ba0,python):2023-12-12-11:33:57.176.550 [mindspore/core/utils/numa_interface.cc:145] NumaBind] Try to bind numa id: 0, but execute set_mempolicy failed, errno: Operation not permitted. Please use mindspore.dataset.config.set_numa_enable(False) to disable numa bind. [WARNING] MD(299854,fffba3fff1e0,python):2023-12-12-11:38:20.901.966 [mindspore/ccsrc/minddata/dataset/engine/datasetops/data_queue_op.cc:912] DetectPerBatchTime] Bad performance attention, it takes more than 25 seconds to fetch a batch of data from dataset pipeline, which might result `GetNext` timeout problem. You may test dataset processing performance(with creating dataset iterator) and optimize it. [WARNING] DEVICE(299854,fffd2cff91e0,python):2023-12-12-12:04:07.125.519 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:801] GetDumpPath] The environment variable 'MS_OM_PATH' is not set, the files will save to the process local path, as ./rank_id/node_dump/... [ERROR] DEVICE(299854,fffd2cff91e0,python):2023-12-12-12:04:07.125.659 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:819] PrintDebugInfoAndDumpFailNode] Task fail infos, rt task_id: 14, rt stream_id: 7, tid: 299854, device_id: 0, retcode: 507011 ( model execute failed) [WARNING] DEVICE(299854,fffd2cff91e0,python):2023-12-12-12:04:07.129.962 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:801] GetDumpPath] The environment variable 'MS_OM_PATH' is not set, the files will save to the process local path, as ./rank_id/exec_order/... [ERROR] DEVICE(299854,fffd2cff91e0,python):2023-12-12-12:04:07.220.624 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:906] DumpDebugInfoFile] Execute order has saved at /home/ma-user/work/slowfast/train_parallel0/rank_0/exec_order/kernel_graph_1.csv Traceback (most recent call last): File "/home/ma-user/work/slowfast/train_parallel0/train.py", line 139, in <module> train() File "/home/ma-user/work/slowfast/train_parallel0/train.py", line 136, in train model.train(cfg.SOLVER.MAX_EPOCH, dataset, callbacks=callbacks, dataset_sink_mode=bool(args.dataset_sink_mode)) File "/home/ma-user/anaconda3/envs/python3.9/lib/python3.9/site-packages/mindspore/train/model.py", line 1063, in train self._train(epoch, File "/home/ma-user/anaconda3/envs/python3.9/lib/python3.9/site-packages/mindspore/train/model.py", line 113, in wrapper func(self, *args, **kwargs) File "/home/ma-user/anaconda3/envs/python3.9/lib/python3.9/site-packages/mindspore/train/model.py", line 621, in _train self._train_dataset_sink_process(epoch, train_dataset, list_callback, File "/home/ma-user/anaconda3/envs/python3.9/lib/python3.9/site-packages/mindspore/train/model.py", line 705, in _train_dataset_sink_process outputs = train_network(*inputs) File "/home/ma-user/anaconda3/envs/python3.9/lib/python3.9/site-packages/mindspore/nn/cell.py", line 639, in __call__ out = self.compile_and_run(*args, **kwargs) File "/home/ma-user/anaconda3/envs/python3.9/lib/python3.9/site-packages/mindspore/nn/cell.py", line 966, in compile_and_run return _cell_graph_executor(self, *new_args, phase=self.phase) File "/home/ma-user/anaconda3/envs/python3.9/lib/python3.9/site-packages/mindspore/common/api.py", line 1674, in __call__ return self.run(obj, *args, phase=phase) File "/home/ma-user/anaconda3/envs/python3.9/lib/python3.9/site-packages/mindspore/common/api.py", line 1713, in run return self._exec_pip(obj, *args, phase=phase_real) File "/home/ma-user/anaconda3/envs/python3.9/lib/python3.9/site-packages/mindspore/common/api.py", line 106, in wrapper results = fn(*arg, **kwargs) File "/home/ma-user/anaconda3/envs/python3.9/lib/python3.9/site-packages/mindspore/common/api.py", line 1693, in _exec_pip return self._graph_executor(args, phase) RuntimeError: Run task for graph:kernel_graph_1 error! The details refer to 'Ascend Error Message'. ---------------------------------------------------- - Ascend Error Message: ---------------------------------------------------- EI0002: The wait execution of the Notify register times out. Reason: The Notify register has not received the Notify record from remote rank [4].base information: [streamID:[7], taskID[14], taskType[Notify Wait], tag[HcomAllReduce_6629421139219749105_0].] task information: [notify id:[0x00000000000000a0], stage:[ffffffff], remote rank:[4].] Possible Cause: 1. An exception occurs during the execution on some NPUs in the cluster. As a result, collective communication operation failed.2. The execution speed on some NPU in the cluster is too slow to complete a communication operation within the timeout interval. (default 1800s, You can set the interval by using HCCL_EXEC_TIMEOUT.)3. The number of training samples of each NPU is inconsistent.4. Packet loss or other connectivity problems occur on the communication link. Solution: 1. If this error is reported on part of these ranks, check other ranks to see whether other errors have been reported earlier.2. If this error is reported for all ranks, check whether the error reporting time is consistent (the maximum difference must not exceed 1800s). If not, locate the cause or adjust the locate the cause or set the HCCL_EXEC_TIMEOUT environment variable to a larger value.3. Check whether the completion queue element (CQE) of the error exists in the plog(grep -rn 'error cqe'). If so, check the network connection status. (For details, see the TLS command and HCCN connectivity check examples.)4. Ensure that the number of training samples of each NPU is consistent. For details:https://www.hiascend.com/document TraceBack (most recent call last): Notify wait execute failed, device_id=0, stream_id=7, task_id=14, flip_num=0, notify_id=20[FUNC:GetError][FILE:stream.cc][LINE:1418] rtStreamSynchronize execute failed, reason=[the model stream execute failed][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:50] (Please search "Ascend Error Message" at https://www.mindspore.cn for error code description) ---------------------------------------------------- - C++ Call Stack: (For framework developers) ---------------------------------------------------- mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_graph_executor.cc:296 RunGraph [WARNING] MD(299854,ffffa3a11ba0,python):2023-12-12-12:04:09.257.359 [mindspore/ccsrc/minddata/dataset/engine/datasetops/data_queue_op.cc:115] ~DataQueueOp] preprocess_batch: 28; batch_queue: 0, 0, 0, 0, 0, 0, 0, 0, 0, 0; push_start_time -> push_end_time 2023-12-12-11:38:49.190.646 -> 2023-12-12-11:38:49.206.462 2023-12-12-11:38:49.704.597 -> 2023-12-12-11:38:49.717.272 2023-12-12-11:38:53.740.464 -> 2023-12-12-11:38:53.753.396 2023-12-12-11:38:54.816.448 -> 2023-12-12-11:38:54.838.486 2023-12-12-11:38:58.286.482 -> 2023-12-12-11:38:58.301.204 2023-12-12-11:38:59.418.431 -> 2023-12-12-11:38:59.432.061 2023-12-12-11:39:02.766.988 -> 2023-12-12-11:39:02.783.474 2023-12-12-11:39:03.989.076 -> 2023-12-12-11:39:04.003.052 2023-12-12-11:39:06.062.494 -> 2023-12-12-11:39:06.085.283 2023-12-12-11:39:07.613.613 -> 2023-12-12-11:39:07.626.919 For more details, please refer to the FAQ at https://www.mindspore.cn/docs/en/master/faq/data_processing.html. ### 备注 麻烦帮忙看一下,谢谢
评论 (
4
)
登录
后才可以发表评论
状态
DONE
TODO
ACCEPTED
WIP
VALIDATION
DONE
CLOSED
REJECTED
负责人
未设置
yxx
yangxixin
负责人
协作者
+负责人
+协作者
lyq
liu-yongqi-63
负责人
协作者
+负责人
+协作者
标签
minddata
未设置
项目
未立项任务
未立项任务
里程碑
未关联里程碑
未关联里程碑
Pull Requests
未关联
未关联
关联的 Pull Requests 被合并后可能会关闭此 issue
分支
未关联
分支 (
-
)
标签 (
-
)
开始日期   -   截止日期
-
置顶选项
不置顶
置顶等级:高
置顶等级:中
置顶等级:低
优先级
不指定
严重
主要
次要
不重要
预计工期
(小时)
参与者(5)
1
https://gitee.com/mindspore/models.git
git@gitee.com:mindspore/models.git
mindspore
models
models
点此查找更多帮助
搜索帮助
Git 命令在线学习
如何在 Gitee 导入 GitHub 仓库
Git 仓库基础操作
企业版和社区版功能对比
SSH 公钥设置
如何处理代码冲突
仓库体积过大,如何减小?
如何找回被删除的仓库数据
Gitee 产品配额说明
GitHub仓库快速导入Gitee及同步更新
什么是 Release(发行版)
将 PHP 项目自动发布到 packagist.org
仓库举报
回到顶部
登录提示
该操作需登录 Gitee 帐号,请先登录后再操作。
立即登录
没有帐号,去注册