2.3K Star 8K Fork 4.2K

GVPMindSpore / mindspore

 / 详情

[CT][MS] cases core dump at cpu backend

DONE
Bug-Report
创建于  
2021-07-08 18:19
name about labels
Bug Report Use this template for reporting a bug kind/bug

Environment

  • Hardware Environment(Ascend/GPU/CPU):

Uncomment only one /device <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:
/device cpu

  • Software Environment:
    -- MindSpore version (source or binary):
    -- Python version (e.g., Python 3.7.5):
    -- OS platform and distribution (e.g., Linux Ubuntu 16.04):
    -- GCC/Compiler version (if compiled from source):

Related testcase

Steps to reproduce the issue

  1. pytest -s -v operations/test_avgpool.py::test_avgpool_input_32x64x112x112_ksize_3_stride_2_padding_same
    pytest -s -v operations/test_avgpool2d.py::test_avgpool2d_input_32x64x112x112_ksize_3_stride_2_padding_same
    pytest -s -v operations/test_batchnorm3d.py::test_batchnorm3d_input_1x3x4x27x27_fp32
    pytest -s -v operations/test_maxpool2d.py::test_maxpool_input_32x64x112x112_dtype_fp16
    pytest -s -v operations/test_rmsprop.py::test_rmsprop_forward_input_shape_4d_epoch_3_lr_0001_centered_true_dtype_float32
    pytest -s -v operations/test_tanh.py::test_tanh_input_0
    pytest -s -v side_effect_expression/test_side_effect_expression02.py::test_side_effect_grad_two_addn_switch
    pytest -s -v parse/test_parser_tuple_index.py::test_parser_tuple_minus_index_004

Describe the current behavior

cpu + master + graph/pynative +python3.7

  1. pytest -s -v operations/test_avgpool.py::test_avgpool_input_32x64x112x112_ksize_3_stride_2_padding_same
INFO] DEVICE(17604,7f09f4c36700,python):2021-07-08-17:29:58.036.857 [mindspore/ccsrc/runtime/hardware/cpu/cpu_memory_pool.cc:75] AllocDeviceMem] Current alloc size[1073741824], total used size[1073741824].
Fatal Python error: Segmentation fault

Thread 0x00007f0bb4602740 (most recent call first):
Segmentation fault (core dumped)

  1. pytest -s -v operations/test_avgpool2d.py::test_avgpool2d_input_32x64x112x112_ksize_3_stride_2_padding_same
[INFO] VM(62337,7fc03eee4740,python):2021-07-08-17:32:03.564.425 [mindspore/ccsrc/vm/backend.cc:830] RunGraph] Run actor end, actor name: kernel_graph_0
double free or corruption (out)
Fatal Python error: Aborted

Current thread 0x00007fc03eee4740 (most recent call first):
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/mindspore/common/api.py", line 643 in del_net_res
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/mindspore/nn/cell.py", line 293 in __del__
  File "/home/wangchunai/code0707/MindSporeTest/share/ops/nn/avgpool2d_ops.py", line 93 in forward_cmp
  File "/home/wangchunai/code0707/MindSporeTest/operations/test_avgpool2d.py", line 515 in test_avgpool2d_input_32x64x112x112_ksize_3_stride_2_padding_same
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/python.py", line 167 in pytest_pyfunc_call
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/python.py", line 1445 in runtest
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 134 in pytest_runtest_call
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 210 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 237 in from_call
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 210 in call_runtest_hook
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 185 in call_and_report
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 99 in runtestprotocol
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 84 in pytest_runtest_protocol
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/main.py", line 271 in pytest_runtestloop
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/main.py", line 247 in _main
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/main.py", line 197 in wrap_session
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/main.py", line 240 in pytest_cmdline_main
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/config/__init__.py", line 93 in main
  File "/root/miniconda3/envs/ci3.7/bin/pytest", line 8 in <module>
Aborted (core dumped)

  1. pytest -s -v operations/test_batchnorm3d.py::test_batchnorm3d_input_1x3x4x27x27_fp32
[INFO] VM(38173,7f63860c6740,python):2021-07-08-17:27:35.727.008 [mindspore/ccsrc/vm/backend.cc:830] RunGraph] Run actor end, actor name: kernel_graph_1
corrupted double-linked list
Fatal Python error: Aborted

Current thread 0x00007f63860c6740 (most recent call first):
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/mindspore/context.py", line 734 in get_context
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/mindspore/nn/cell.py", line 290 in __del__
  File "/home/wangchunai/code0707/MindSporeTest/share/ops/nn/batchnorm3d_ops.py", line 223 in grad_cmp
  File "/home/wangchunai/code0707/MindSporeTest/operations/test_batchnorm3d.py", line 30 in test_batchnorm3d_input_1x3x4x27x27_fp32
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/python.py", line 167 in pytest_pyfunc_call
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/python.py", line 1445 in runtest
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 134 in pytest_runtest_call
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 210 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 237 in from_call
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 210 in call_runtest_hook
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 185 in call_and_report
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 99 in runtestprotocol
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 84 in pytest_runtest_protocol
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/main.py", line 271 in pytest_runtestloop
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/main.py", line 247 in _main
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/main.py", line 197 in wrap_session
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/main.py", line 240 in pytest_cmdline_main
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/config/__init__.py", line 93 in main
  File "/root/miniconda3/envs/ci3.7/bin/pytest", line 8 in <module>
Aborted (core dumped)

  1. pytest -s -v operations/test_maxpool2d.py::test_maxpool_input_32x64x112x112_dtype_fp16
[INFO] ME(61093,7fbd0f14b740,python):2021-07-08-17:33:38.018.572 [mindspore/core/mindrt/src/actor/actormgr.cc:113] Finalize] mindrt IOMGRS finish exiting.
[INFO] DEVICE(61093,7fbd0f14b740,python):2021-07-08-17:33:38.030.693 [mindspore/ccsrc/runtime/hardware/device_context_manager.cc:29] ClearDeviceContexts] Release device CPU_0
double free or corruption (out)
Fatal Python error: Aborted

Current thread 0x00007fbd0f14b740 (most recent call first):
Aborted (core dumped)

  1. pytest -s -v operations/test_rmsprop.py::test_rmsprop_forward_input_shape_4d_epoch_3_lr_0001_centered_true_dtype_float32
[INFO] PARSER(2206,7fe3daadc740,python):2021-07-08-17:35:50.274.121 [mindspore/ccsrc/pipeline/jit/parse/function_block.cc:475] AttachIsolatedNodesBeforeReturn] Attached for side-effect nodes, depend_node: construct.102:[CNode]108{[0]: ValueNode<Primitive> Depend, [1]: [CNode]109, [2]: [CNode]110}, state: construct.102:[CNode]103{[0]: construct.102:[CNode]104{[0]: ValueNode<Primitive> resolve, [1]: ValueNode<NameSpace> SymbolStr, [2]: ValueNode<Symbol> _shape_check_bn}, [1]: construct.102:[CNode]105{[0]: [CNode]106, [1]: Φx}, [2]: construct.102:[CNode]107{[0]: ValueNode<Primitive> resolve, [1]: ValueNode<NameSpace> ClassMember, [2]: ValueNode<Symbol> input_dims}}
python: malloc.c:3857: _int_malloc: Assertion `chunk_main_arena (fwd)' failed.
Fatal Python error: Aborted

Current thread 0x00007fe3daadc740 (most recent call first):
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/mindspore/common/api.py", line 531 in compile
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/mindspore/nn/cell.py", line 630 in compile
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/mindspore/nn/cell.py", line 643 in compile_and_run
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/mindspore/nn/cell.py", line 385 in __call__
  File "/home/wangchunai/code0707/MindSporeTest/share/ops/nn/rmsprop_ops.py", line 89 in forward_mindspore_impl
  File "/home/wangchunai/code0707/MindSporeTest/share/ops/nn/rmsprop_ops.py", line 124 in forward_cmp
  File "/home/wangchunai/code0707/MindSporeTest/operations/test_rmsprop.py", line 79 in test_rmsprop_forward_input_shape_4d_epoch_3_lr_0001_centered_true_dtype_float32
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/python.py", line 167 in pytest_pyfunc_call
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/python.py", line 1445 in runtest
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 134 in pytest_runtest_call
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 210 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 237 in from_call
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 210 in call_runtest_hook
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 185 in call_and_report
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 99 in runtestprotocol
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 84 in pytest_runtest_protocol
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/main.py", line 271 in pytest_runtestloop
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/main.py", line 247 in _main
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/main.py", line 197 in wrap_session
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/main.py", line 240 in pytest_cmdline_main
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/config/__init__.py", line 93 in main
  File "/root/miniconda3/envs/ci3.7/bin/pytest", line 8 in <module>
Aborted (core dumped)

  1. pytest -s -v operations/test_tanh.py::test_tanh_input_0
[INFO] DEVICE(96702,7fc315ffb700,python):2021-07-08-17:37:42.376.555 [mindspore/ccsrc/runtime/device/cpu/cpu_device_address.cc:83] SyncHostToDevice] No need sync, host size: 0, device size: 0
[INFO] DEVICE(96702,7fc315ffb700,python):2021-07-08-17:37:42.376.581 [mindspore/ccsrc/runtime/device/cpu/cpu_device_address.cc:83] SyncHostToDevice] No need sync, host size: 0, device size: 0
Fatal Python error: Segmentation fault

Thread 0x00007fcabca3e740 (most recent call first):
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/mindspore/common/api.py", line 622 in _exec_pip
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/mindspore/common/api.py", line 75 in wrapper
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/mindspore/common/api.py", line 639 in run
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/mindspore/common/api.py", line 611 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/mindspore/nn/cell.py", line 659 in compile_and_run
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/mindspore/nn/cell.py", line 385 in __call__
  File "/home/wangchunai/code0707/MindSporeTest/share/ops/primitive/tanh_ops.py", line 48 in grad_mindspore_impl
  File "/home/wangchunai/code0707/MindSporeTest/share/ops/primitive/tanh_ops.py", line 67 in grad_cmp
  File "/home/wangchunai/code0707/MindSporeTest/operations/test_tanh.py", line 121 in test_tanh_input_0
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/python.py", line 167 in pytest_pyfunc_call
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/python.py", line 1445 in runtest
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 134 in pytest_runtest_call
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 210 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 237 in from_call
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 210 in call_runtest_hook
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 185 in call_and_report
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 99 in runtestprotocol
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 84 in pytest_runtest_protocol
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/main.py", line 271 in pytest_runtestloop
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/main.py", line 247 in _main
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/main.py", line 197 in wrap_session
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/main.py", line 240 in pytest_cmdline_main
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/config/__init__.py", line 93 in main
  File "/root/miniconda3/envs/ci3.7/bin/pytest", line 8 in <module>
Segmentation fault (core dumped)

7.pytest -s -v side_effect_expression/test_side_effect_expression02.py::test_side_effect_grad_two_addn_switch

[INFO] DEVICE(60882,7f9afc249700,python):2021-07-08-17:39:27.696.663 [mindspore/ccsrc/runtime/hardware/cpu/cpu_memory_pool.cc:75] AllocDeviceMem] Current alloc size[1073741824], total used size[1073741824].
[ERROR] RUNTIME_FRAMEWORK(60882,7f9afe24d700,python):2021-07-08-17:39:27.715.288 [mindspore/ccsrc/runtime/framework/actor/switch_actor.cc:357] FetchInputDeviceTensor] 101_96_87_19_1_construct_wrapper.55:[CNode]42{[0]: ValueNode<Primitive> Switch, [1]: equiv[CNode]58, [2]: [CNode]42, [3]: [CNode]42} get device tensor store failed: Default/data-0, device type:2
[ERROR] RUNTIME_FRAMEWORK(60882,7f9afda4c700,python):2021-07-08-17:39:27.715.294 [mindspore/ccsrc/runtime/framework/actor/switch_actor.cc:357] FetchInputDeviceTensor] 101_96_87_19_1_construct_wrapper.55:[CNode]39{[0]: ValueNode<Primitive> Switch, [1]: equiv[CNode]58, [2]: [CNode]39, [3]: [CNode]39} get device tensor store failed: Default/data-0, device type:2
[ERROR] VM(60882,7f9cb94fb740,python):2021-07-08-17:39:27.715.382 [mindspore/ccsrc/vm/backend.cc:807] RunGraph] The actor runs failed, actor name: kernel_graph_0_1
[ERROR] DEBUG(60882,7f9cb94fb740,python):2021-07-08-17:39:27.715.418 [mindspore/ccsrc/debug/trace.cc:122] TraceGraphEval] Length of analysis graph stack is empty.
[INFO] DEBUG(60882,7f9cb94fb740,python):2021-07-08-17:39:27.715.453 [mindspore/ccsrc/debug/trace.cc:505] GetEvalStackInfo] Get graph analysis information begin
[ERROR] DEBUG(60882,7f9cb94fb740,python):2021-07-08-17:39:27.715.478 [mindspore/ccsrc/debug/trace.cc:508] GetEvalStackInfo] Length of analysis information stack is empty.
terminate called after throwing an instance of 'std::future_error'
  what():  std::future_error: Promise already satisfied
Fatal Python error: Aborted

Thread 0x00007f9cb94fb740 (most recent call first):
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/mindspore/common/api.py", line 622 in _exec_pip
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/mindspore/common/api.py", line 75 in wrapper
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/mindspore/common/api.py", line 639 in run
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/mindspore/common/api.py", line 611 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/mindspore/nn/cell.py", line 659 in compile_and_run
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/mindspore/nn/cell.py", line 385 in __call__
  File "/home/wangchunai/code0707/MindSporeTest/side_effect_expression/test_side_effect_expression02.py", line 410 in grad_mindspore_impl
  File "/home/wangchunai/code0707/MindSporeTest/side_effect_expression/test_side_effect_expression02.py", line 456 in test_side_effect_grad_two_addn_switch
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/python.py", line 167 in pytest_pyfunc_call
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 208 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/python.py", line 1445 in runtest
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 144 in pytest_runtest_call
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 208 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 210 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 244 in from_call
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 210 in call_runtest_hook
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 187 in call_and_report
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 99 in runtestprotocol
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 84 in pytest_runtest_protocol
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/main.py", line 271 in pytest_runtestloop
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/main.py", line 247 in _main
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/main.py", line 197 in wrap_session
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/main.py", line 240 in pytest_cmdline_main
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/config/__init__.py", line 93 in main
  File "/root/miniconda3/envs/ci3.7/bin/pytest", line 8 in <module>
Aborted (core dumped)
  1. pytest -s -v parse/test_parser_tuple_index.py::test_parser_tuple_minus_index_004
[INFO] RUNTIME_FRAMEWORK(6027,7f59dfed0700,python):2021-07-08-19:13:09.330.084 [mindspore/ccsrc/runtime/framework/actor/data_source_actor.cc:39] FetchData] Data source actor(kernel_graph_0_1_2_HostDSActor) fetches data.
[ERROR] RUNTIME_FRAMEWORK(6027,7f59e06d1700,python):2021-07-08-19:13:09.330.108 [mindspore/ccsrc/runtime/framework/actor/data_source_actor.cc:245] OnMemoryAllocFinish] Host data queue is empty.
[ERROR] RUNTIME_FRAMEWORK(6027,7f59e06d1700,python):2021-07-08-19:13:09.330.142 [mindspore/ccsrc/runtime/framework/actor/data_source_actor.cc:245] OnMemoryAllocFinish] Host data queue is empty.
[INFO] RUNTIME_FRAMEWORK(6027,7f59e16d3700,python):2021-07-08-19:13:09.330.223 [mindspore/ccsrc/runtime/framework/actor/output_actor.cc:25] CreateOutputTensor] Create output tensor, output node: Default/relu-ReLU/ReLU-op20, output index: 0, output position: 0
terminate called after throwing an instance of 'std::future_error'
  what():  std::future_error: Promise already satisfied
Fatal Python error: Aborted

Thread 0x00007f5b06ca2740 (most recent call first):
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/mindspore/common/api.py", line 622 in _exec_pip
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/mindspore/common/api.py", line 75 in wrapper
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/mindspore/common/api.py", line 639 in run
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/mindspore/common/api.py", line 611 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/mindspore/nn/cell.py", line 659 in compile_and_run
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/mindspore/nn/cell.py", line 385 in __call__
  File "/home/wangchunai/code0707/MindSporeTest/parse/test_parser_construct_input_no_tensor01.py", line 60 in forward_mindspore_impl
  File "/home/wangchunai/code0707/MindSporeTest/parse/test_parser_construct_input_no_tensor01.py", line 51 in __init__
  File "/home/wangchunai/code0707/MindSporeTest/parse/test_parser_tuple_index.py", line 216 in test_parser_tuple_minus_index_004
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/python.py", line 167 in pytest_pyfunc_call
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/python.py", line 1445 in runtest
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 134 in pytest_runtest_call
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 210 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 237 in from_call
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 210 in call_runtest_hook
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 185 in call_and_report
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 99 in runtestprotocol
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 84 in pytest_runtest_protocol
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/main.py", line 271 in pytest_runtestloop
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/main.py", line 247 in _main
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/main.py", line 197 in wrap_session
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/main.py", line 240 in pytest_cmdline_main
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/config/__init__.py", line 93 in main
  File "/root/miniconda3/envs/ci3.7/bin/pytest", line 8 in <module>
Aborted (core dumped)

  1. pytest -s -v checkpoint/test_checkpoint_specified.py::test_checkpoint_config_saved_network_none

[INFO] MD(109629,7f5eb8984740,python):2021-07-08-19:15:12.970.319 [mindspore/ccsrc/minddata/dataset/util/intrp_service.cc:26] ~IntrpService] Number of registered resources is 181.
[INFO] ME(109629:140044800640832,MainProcess):2021-07-08-19:15:12.970.465 [/home/wangchunai/code0707/MindSporeTest/checkpoint/test_checkpoint_specified.py:71] ----finish model train-----
Fatal Python error: Segmentation fault

Current thread 0x00007f5eb8984740 (most recent call first):
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/mindspore/common/api.py", line 643 in del_net_res
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/mindspore/nn/cell.py", line 293 in __del__
  File "/home/wangchunai/code0707/MindSporeTest/checkpoint/test_checkpoint_specified.py", line 147 in test_checkpoint_config_saved_network_none
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/python.py", line 167 in pytest_pyfunc_call
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/python.py", line 1445 in runtest
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 134 in pytest_runtest_call
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 210 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 237 in from_call
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 210 in call_runtest_hook
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 185 in call_and_report
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 99 in runtestprotocol
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 84 in pytest_runtest_protocol
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/main.py", line 271 in pytest_runtestloop
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/main.py", line 247 in _main
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/main.py", line 197 in wrap_session
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/main.py", line 240 in pytest_cmdline_main
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/config/__init__.py", line 93 in main
  File "/root/miniconda3/envs/ci3.7/bin/pytest", line 8 in <module>
Segmentation fault (core dumped)

  1. pytest -s -v checkpoint/test_checkpoint_resnet50_train.py::test_resnet50_train_image_checkpoint_check_loss_value[True]
[INFO] RUNTIME_FRAMEWORK(101506,7fb221336740,python):2021-07-08-19:17:07.877.820 [mindspore/ccsrc/runtime/framework/graph_scheduler.cc:203] PrepareDataForWeightNode] Prepare device data for weight node:moments.bn1.gamma, device type:2
[INFO] RUNTIME_FRAMEWORK(101506,7fb221336740,python):2021-07-08-19:17:07.877.832 [mindspore/ccsrc/runtime/framework/graph_scheduler.cc:203] PrepareDataForWeightNode] Prepare device data for weight node:moments.conv1.weight, device type:2
[INFO] RUNTIME_FRAMEWORK(101506,7fafecff9700,python):2021-07-08-19:17:07.878.032 [mindspore/ccsrc/runtime/framework/actor/data_source_actor.cc:39] FetchData] Data source actor(kernel_graph_1_HostDSActor) fetches data.
Fatal Python error: Segmentation fault

Thread 0x00007fb221336740 (most recent call first):
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/mindspore/common/api.py", line 622 in _exec_pip
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/mindspore/common/api.py", line 75 in wrapper
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/mindspore/common/api.py", line 639 in run
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/mindspore/common/api.py", line 611 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/mindspore/nn/cell.py", line 659 in compile_and_run
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/mindspore/nn/cell.py", line 385 in __call__
  File "/home/wangchunai/code0707/MindSporeTest/checkpoint/test_checkpoint_resnet50_train.py", line 158 in me_train_dataset_load_ckpt_get_loss
  File "/home/wangchunai/code0707/MindSporeTest/checkpoint/test_checkpoint_resnet50_train.py", line 261 in test_resnet50_train_image_checkpoint_check_loss_value
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/python.py", line 167 in pytest_pyfunc_call
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/python.py", line 1445 in runtest
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 134 in pytest_runtest_call
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 210 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 237 in from_call
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 210 in call_runtest_hook
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 185 in call_and_report
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 99 in runtestprotocol
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/runner.py", line 84 in pytest_runtest_protocol
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/main.py", line 271 in pytest_runtestloop
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/main.py", line 247 in _main
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/main.py", line 197 in wrap_session
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/main.py", line 240 in pytest_cmdline_main
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/root/miniconda3/envs/ci3.7/lib/python3.7/site-packages/_pytest/config/__init__.py", line 93 in main
  File "/root/miniconda3/envs/ci3.7/bin/pytest", line 8 in <module>
Segmentation fault (core dumped)

Describe the expected behavior

Related log / screenshot

Special notes for this issue

评论 (4)

si_chasing 创建了Bug-Report
si_chasing 负责人设置为limingqi107
si_chasing 关联仓库设置为MindSpore/mindspore
si_chasing 计划开始日期设置为2020-04-01
si_chasing 计划截止日期设置为2021-12-31
si_chasing 里程碑设置为B-MIT
si_chasing 关联分支设置为master
si_chasing 优先级设置为主要
si_chasing 添加了
 
kind/bug
标签
si_chasing 添加了device/cpu(已删除)标签
展开全部操作日志

Please add labels (comp or sig), also you can visit "https://gitee.com/mindspore/community/blob/master/sigs/dx/docs/labels.md" to find more.
为了让问题更快得到响应,请您为该issue打上**组件(comp)或兴趣组(sig)**标签,打上标签的问题可以直接推送给责任人进行处理。更多的标签可以查看 https://gitee.com/mindspore/community/blob/master/sigs/dx/docs/labels.md"
以组件问题为例,如果你发现问题是data组件造成的,你可以这样评论:
//comp/data
当然你也可以向data SIG组求助,可以这样写:
//comp/data
//sig/data
如果是一个简单的问题,你可以留给刚进入社区的小伙伴来回答,这时候你可以这样写:
//good-first-issue
恭喜你,你已经学会了使用命令来打标签,接下来就在下面的评论里打上标签吧!

si_chasing 修改了描述
zsj_mind 添加了
 
v1.3.0
标签
limingqi107 添加协作者limingqi107
limingqi107 负责人limingqi107 修改为zyli2020

7.9 CPU graph r1.3 python3.8
pytest -s -v operations/test_avgpool.py::test_avgpool_input_32x64x112x112_ksize_3_stride_2_padding_same

[INFO] RUNTIME_FRAMEWORK(53821,7f10d2432740,python):2021-07-09-18:12:08.217.279 [mindspore/ccsrc/runtime/framework/graph_scheduler.cc:540] Transform] Graph(kernel_graph_0) transforms actor end.
[INFO] PIPELINE(53821,7f10d2432740,python):2021-07-09-18:12:08.217.309 [mindspore/ccsrc/pipeline/jit/pipeline.cc:175] SetGpuLoopSink] Change gpu_loopsink_flag_ to 1, set loopsink size to 1
[INFO] PIPELINE(53821,7f10d2432740,python):2021-07-09-18:12:08.217.321 [mindspore/ccsrc/pipeline/jit/pipeline.cc:889] Run] End
[INFO] PIPELINE(53821,7f10d2432740,python):2021-07-09-18:12:08.217.334 [mindspore/ccsrc/pipeline/jit/pipeline.cc:579] SaveCompiledGraph] Save compiled func graph(3_1_construct_wrapper.2) phase(0train.1625825528188246272)!
[INFO] PIPELINE(53821,7f10d2432740,python):2021-07-09-18:12:08.217.342 [mindspore/ccsrc/pipeline/jit/pipeline.cc:592] SaveCompiledGraph] End save compiled func graph!
[INFO] PIPELINE(53821,7f10d2432740,python):2021-07-09-18:12:08.217.435 [mindspore/ccsrc/pipeline/jit/pipeline.cc:715] CompileInner] End ExecutorPy compile!
[INFO] PIPELINE(53821,7f10d2432740,python):2021-07-09-18:12:08.217.730 [mindspore/ccsrc/pipeline/jit/pipeline.cc:1011] Run] VM loop size 1, loopsink size 1
[INFO] PIPELINE(53821,7f10d2432740,python):2021-07-09-18:12:08.217.741 [mindspore/ccsrc/pipeline/jit/action.cc:84] operator()] Execute args size 1
[INFO] VM(53821,7f10d2432740,python):2021-07-09-18:12:08.217.750 [mindspore/ccsrc/vm/backend.cc:749] RunGraph] Run actor begin, actor name: kernel_graph_0
[INFO] RUNTIME_FRAMEWORK(53821,7f0eeb359700,python):2021-07-09-18:12:08.217.872 [mindspore/ccsrc/runtime/framework/actor/data_source_actor.cc:39] FetchData] Data source actor(kernel_graph_0_HostDSActor) fetches data.
[INFO] DEVICE(53821,7f0eea357700,python):2021-07-09-18:12:08.218.072 [mindspore/ccsrc/runtime/hardware/cpu/cpu_memory_pool.cc:47] GetSystemMemorySize] Get system memory(MemAvailable): 259468548 kB
[INFO] DEVICE(53821,7f0eea357700,python):2021-07-09-18:12:08.218.106 [mindspore/ccsrc/runtime/hardware/cpu/cpu_memory_pool.cc:75] AllocDeviceMem] Current alloc size[1073741824], total used size[1073741824].
Fatal Python error: Segmentation fault

Thread 0x00007f10d2432740 (most recent call first):
  File Segmentation fault (core dumped)

CPU 部分算子调用三方库,不支持初始化和launch在不同的线程,出现core dump,修复方案:在launch线程里reinitialize kernel,详见pr:
https://gitee.com/mind_spore/dashboard/projects/mindspore/mindspore/pulls/19855
https://gitee.com/mind_spore/dashboard/projects/mindspore/mindspore/pulls/19922

zyli2020 任务状态TODO 修改为VALIDATION
zyli2020 添加协作者zyli2020
zyli2020 负责人zyli2020 修改为si_chasing
zyli2020 里程碑B-MIT 修改为B-ComponentTest

CPU r1.3 python3.8 python3.7 graph pynative
all case passed
pytest -s -v operations/test_avgpool.py::test_avgpool_input_32x64x112x112_ksize_3_stride_2_padding_same
pytest -s -v operations/test_avgpool2d.py::test_avgpool2d_input_32x64x112x112_ksize_3_stride_2_padding_same
pytest -s -v operations/test_batchnorm3d.py::test_batchnorm3d_input_1x3x4x27x27_fp32
pytest -s -v operations/test_maxpool2d.py::test_maxpool_input_32x64x112x112_dtype_fp16
pytest -s -v operations/test_rmsprop.py::test_rmsprop_forward_input_shape_4d_epoch_3_lr_0001_centered_true_dtype_float32
pytest -s -v operations/test_tanh.py::test_tanh_input_0
pytest -s -v side_effect_expression/test_side_effect_expression02.py::test_side_effect_grad_two_addn_switch

si_chasing 任务状态VALIDATION 修改为DONE
si_chasing 移除了device/cpu(已删除)标签

登录 后才可以发表评论

状态
负责人
项目
里程碑
Pull Requests
关联的 Pull Requests 被合并后可能会关闭此 issue
分支
开始日期   -   截止日期
-
置顶选项
优先级
预计工期 (小时)
参与者(4)
8777557 test bot 1617846881
Python
1
https://gitee.com/mindspore/mindspore.git
git@gitee.com:mindspore/mindspore.git
mindspore
mindspore
mindspore

搜索帮助