[ERROR] KERNEL(2671083,ffff951de480,python):2024-03-19-11:53:08.037.101 [mindspore/ccsrc/plugin/factory/ms_factory.h:53] Register] Kernel NotEqual is already registered!
[WARNING] GE_ADPT(2671083,ffff951de480,python):2024-03-19-11:53:08.103.617 [mindspore/ccsrc/transform/symbol/acl_tdt_symbol.cc:49] LoadAcltdtApiSymbol] Dlopen /usr/local/Ascend/latest/lib64/libacl_tdt_channel.so failed!libacl_tdt_queue.so: cannot open shared object file: No such file or directory
2024-03-19 11:53:14,103 - mindformers[mindformers/tools/utils.py:153] - INFO - set output path to '/home/jenkins0/MindFormers_Test/cases/llama2/13b/train/test_mf_llama2_13b_train_alpaca_16p_0001/output'
[WARNING] DEVICE(2671083,ffff951de480,python):2024-03-19-11:53:14.284.288 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_memory_adapter.cc:95] Initialize] Reserved memory size for other components(3187671040) is less than recommend size(4073724928), It may lead to Out Of Memory in HCCL or other components, Please double check context key 'variable_memory_max_size'/'max_device_memory'
2024-03-19 11:53:31,012 - mindformers[mindformers/tools/cloud_adapter/cloud_monitor.py:43] - ERROR - Traceback (most recent call last):
 File "/home/jenkins0/MindFormers_Test/cases/llama2/13b/train/test_mf_llama2_13b_train_alpaca_16p_0001/scripts/mf_parallel15/mindformers/core/context/build_context.py", line 116, in init_context
   init()
 File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindspore/communication/management.py", line 188, in init
   init_hccl()
RuntimeError: acltdtCreateChannelWithCapacity is null.

----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/transform/symbol/symbol_utils.h:26 RunAscendApi


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 File "/home/jenkins0/MindFormers_Test/cases/llama2/13b/train/test_mf_llama2_13b_train_alpaca_16p_0001/scripts/mf_parallel15/mindformers/tools/cloud_adapter/cloud_monitor.py", line 34, in wrapper
   result = run_func(*args, **kwargs)
 File "run_mindformer.py", line 35, in main
   build_context(config)
 File "/home/jenkins0/MindFormers_Test/cases/llama2/13b/train/test_mf_llama2_13b_train_alpaca_16p_0001/scripts/mf_parallel15/mindformers/core/context/build_context.py", line 44, in build_context
   context_config=config.context, parallel_config=config.parallel)
 File "/home/jenkins0/MindFormers_Test/cases/llama2/13b/train/test_mf_llama2_13b_train_alpaca_16p_0001/scripts/mf_parallel15/mindformers/core/context/build_context.py", line 118, in init_context
   raise RuntimeError("Notice: if you are trying to run with a single device, please set "
RuntimeError: Notice: if you are trying to run with a single device, please set use_parallel=False. If not, please check the error message above.

Traceback (most recent call last):
 File "/home/jenkins0/MindFormers_Test/cases/llama2/13b/train/test_mf_llama2_13b_train_alpaca_16p_0001/scripts/mf_parallel15/mindformers/core/context/build_context.py", line 116, in init_context
   init()
 File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindspore/communication/management.py", line 188, in init
   init_hccl()
RuntimeError: acltdtCreateChannelWithCapacity is null.

----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/transform/symbol/symbol_utils.h:26 RunAscendApi

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 File "run_mindformer.py", line 267, in <module>
   main(config_)
 File "/home/jenkins0/MindFormers_Test/cases/llama2/13b/train/test_mf_llama2_13b_train_alpaca_16p_0001/scripts/mf_parallel15/mindformers/tools/cloud_adapter/cloud_monitor.py", line 44, in wrapper
   raise exc
 File "/home/jenkins0/MindFormers_Test/cases/llama2/13b/train/test_mf_llama2_13b_train_alpaca_16p_0001/scripts/mf_parallel15/mindformers/tools/cloud_adapter/cloud_monitor.py", line 34, in wrapper
   result = run_func(*args, **kwargs)
 File "run_mindformer.py", line 35, in main
   build_context(config)
 File "/home/jenkins0/MindFormers_Test/cases/llama2/13b/train/test_mf_llama2_13b_train_alpaca_16p_0001/scripts/mf_parallel15/mindformers/core/context/build_context.py", line 44, in build_context
   context_config=config.context, parallel_config=config.parallel)
 File "/home/jenkins0/MindFormers_Test/cases/llama2/13b/train/test_mf_llama2_13b_train_alpaca_16p_0001/scripts/mf_parallel15/mindformers/core/context/build_context.py", line 118, in init_context
   raise RuntimeError("Notice: if you are trying to run with a single device, please set "
RuntimeError: Notice: if you are trying to run with a single device, please set use_parallel=False. If not, please check the error message above.
Error in atexit._run_exitfuncs:
RuntimeError: The pointer[runtime_instance_] is null.

----------------------------------------------------
- Framework Unexpected Exception Raised:
----------------------------------------------------
This exception is caused by framework's unexpected error. Please create an issue at https://gitee.com/mindspore/mindspore/issues to get help.

----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ge_device_res_manager.h:82 GetStream

free(): double free detected in tcache 2