使用DeepSpeed+torch+torch_npu训练Qwen1.5-7B-Chat模型报错，ninja编译cpu_adam报错

一、问题现象（附报错日志上下文）：
在910A上使用DeepSpeed全量训练Qwen1.5-7B-Chat模型，ninja在编译cpu_adam报错，无法训练，看日志bug错误是torch_npu acl层所报。

二、软件版本:
--CANN 版本:  7.0.0
--GCC版本：9.3.0
--transformers版本：4.41.0
--Pytorch 版本: 2.1.0
--torch-npu 版本：2.1.0.post3
--Python 版本: 3.8.13
--DeepSpeed版本: 0.14.2
--ninja版本：1.11.1.git.kitware.jobserver-1
--操作系统版本：Ubuntu 18.04

三、测试步骤：
下载Firefly仓https://github.com/yangjianxin1/Firefly.git，根据样例Firefly/train_args/sft/full/qwen-7b-sft-full.json编写Qwen1.5-7B-Chat的训练json配置文件，用已下载好的Qwen1.5-7B-Chat模型，之后，运行其Firefly/train.py脚本：运行命令：
deepspeed --num_gpus=4 train.py --train_args_file train_args/sft/full/qwen1.5-7b-sft-full.json

四、日志信息:
(hf-dev) root@root:/workspace/code/hf-test/Firefly# deepspeed --num_gpus=4 train.py --train_args_file train_args/sft/full/qwen1.5-7b-sft-full.json
[2024-05-21 12:19:31,896] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to npu (auto detect)
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
[2024-05-21 12:19:32,811] [WARNING] [runner.py:202:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2024-05-21 12:19:32,811] [INFO] [runner.py:568:main] cmd = /usr/local/conda/envs/hf-dev/bin/python -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMSwgMiwgM119 --master_addr=127.0.0.1 --master_port=29500 --enable_each_rank_log=None train.py --train_args_file train_args/sft/full/qwen1.5-7b-sft-full.json
[2024-05-21 12:19:37,996] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to npu (auto detect)
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
[2024-05-21 12:19:38,918] [INFO] [launch.py:146:main] WORLD INFO DICT: {'localhost': [0, 1, 2, 3]}
[2024-05-21 12:19:38,918] [INFO] [launch.py:152:main] nnodes=1, num_local_procs=4, node_rank=0
[2024-05-21 12:19:38,918] [INFO] [launch.py:163:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1, 2, 3]})
[2024-05-21 12:19:38,918] [INFO] [launch.py:164:main] dist_world_size=4
[2024-05-21 12:19:38,918] [INFO] [launch.py:168:main] Setting ASCEND_RT_VISIBLE_DEVICES=0,1,2,3
[2024-05-21 12:19:38,937] [INFO] [launch.py:256:main] process 684505 spawned with command: ['/usr/local/conda/envs/hf-dev/bin/python', '-u', 'train.py', '--local_rank=0', '--train_args_file', 'train_args/sft/full/qwen2-7b-sft-full.json']
[2024-05-21 12:19:38,949] [INFO] [launch.py:256:main] process 684506 spawned with command: ['/usr/local/conda/envs/hf-dev/bin/python', '-u', 'train.py', '--local_rank=1', '--train_args_file', 'train_args/sft/full/qwen2-7b-sft-full.json']
[2024-05-21 12:19:38,962] [INFO] [launch.py:256:main] process 684507 spawned with command: ['/usr/local/conda/envs/hf-dev/bin/python', '-u', 'train.py', '--local_rank=2', '--train_args_file', 'train_args/sft/full/qwen2-7b-sft-full.json']
[2024-05-21 12:19:38,975] [INFO] [launch.py:256:main] process 684508 spawned with command: ['/usr/local/conda/envs/hf-dev/bin/python', '-u', 'train.py', '--local_rank=3', '--train_args_file', 'train_args/sft/full/qwen2-7b-sft-full.json']
/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
  warn("The installed version of bitsandbytes was compiled without GPU support. "
'NoneType' object has no attribute 'cadam32bit_grad_fp32'
/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
  warn("The installed version of bitsandbytes was compiled without GPU support. "
'NoneType' object has no attribute 'cadam32bit_grad_fp32'
/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
  warn("The installed version of bitsandbytes was compiled without GPU support. "
'NoneType' object has no attribute 'cadam32bit_grad_fp32'
/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
  warn("The installed version of bitsandbytes was compiled without GPU support. "
'NoneType' object has no attribute 'cadam32bit_grad_fp32'
[2024-05-21 12:19:46,406] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to npu (auto detect)
[2024-05-21 12:19:46,511] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to npu (auto detect)
[2024-05-21 12:19:46,523] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to npu (auto detect)
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
[2024-05-21 12:19:46,661] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to npu (auto detect)
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
Using /root/.cache/torch_extensions/py38_cpu as PyTorch extensions root...
Emitting ninja build file /root/.cache/torch_extensions/py38_cpu/cpu_adam/build.ninja...
Building extension module cpu_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
Using /root/.cache/torch_extensions/py38_cpu as PyTorch extensions root...
Using /root/.cache/torch_extensions/py38_cpu as PyTorch extensions root...
Using /root/.cache/torch_extensions/py38_cpu as PyTorch extensions root...
[1/3] c++ -MMD -MF cpu_adam.o.d -DTORCH_EXTENSION_NAME=cpu_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/usr/local/Ascend/ascend-toolkit/latest/include -I/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch_npu/include -I/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/deepspeed/ops/csrc/includes -isystem /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include -isystem /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/TH -isystem /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/conda/envs/hf-dev/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O3 -std=c++17 -g -Wno-reorder -fopenmp -fstack-protector-all -Wl,-z,relro,-z,now,-z,noexecstack -Wl,--disable-new-dtags,--rpath -D__ENABLE_CANN__ -march=native -D__SCALAR__ -L/usr/local/Ascend/ascend-toolkit/latest/lib64 -L/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch_npu/lib -c /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/deepspeed/ops/csrc/adam/cpu_adam.cpp -o cpu_adam.o
FAILED: cpu_adam.o
c++ -MMD -MF cpu_adam.o.d -DTORCH_EXTENSION_NAME=cpu_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/usr/local/Ascend/ascend-toolkit/latest/include -I/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch_npu/include -I/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/deepspeed/ops/csrc/includes -isystem /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include -isystem /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/TH -isystem /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/conda/envs/hf-dev/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O3 -std=c++17 -g -Wno-reorder -fopenmp -fstack-protector-all -Wl,-z,relro,-z,now,-z,noexecstack -Wl,--disable-new-dtags,--rpath -D__ENABLE_CANN__ -march=native -D__SCALAR__ -L/usr/local/Ascend/ascend-toolkit/latest/lib64 -L/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch_npu/lib -c /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/deepspeed/ops/csrc/adam/cpu_adam.cpp -o cpu_adam.o
In file included from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch_npu/include/torch_npu/csrc/core/npu/NPUException.h:16:0,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch_npu/include/torch_npu/csrc/core/npu/NPUStream.h:10,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/deepspeed/ops/csrc/includes/cpu_adam.h:24,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/deepspeed/ops/csrc/adam/cpu_adam.cpp:6:
/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch_npu/include/torch_npu/csrc/core/npu/interface/AclInterface.h:157:40: error: ‘aclCannAttr’ does not name a type; did you mean ‘aclopAttr’?
 aclError AclGetCannAttributeList(const aclCannAttr **cannAttrList, size_t *num);
                                        ^~~~~~~~~~~
                                        aclopAttr
/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch_npu/include/torch_npu/csrc/core/npu/interface/AclInterface.h:159:30: error: ‘aclCannAttr’ was not declared in this scope
 aclError AclGetCannAttribute(aclCannAttr cannAttr, int32_t *value);
                              ^~~~~~~~~~~
/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch_npu/include/torch_npu/csrc/core/npu/interface/AclInterface.h:159:30: note: suggested alternative: ‘aclopAttr’
 aclError AclGetCannAttribute(aclCannAttr cannAttr, int32_t *value);
                              ^~~~~~~~~~~
                              aclopAttr
/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch_npu/include/torch_npu/csrc/core/npu/interface/AclInterface.h:159:60: error: expected primary-expression before ‘*’ token
 aclError AclGetCannAttribute(aclCannAttr cannAttr, int32_t *value);
                                                            ^
/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch_npu/include/torch_npu/csrc/core/npu/interface/AclInterface.h:159:61: error: ‘value’ was not declared in this scope
 aclError AclGetCannAttribute(aclCannAttr cannAttr, int32_t *value);
                                                             ^~~~~
/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch_npu/include/torch_npu/csrc/core/npu/interface/AclInterface.h:159:61: note: suggested alternatives:
In file included from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/torch/csrc/jit/ir/ir.h:18:0,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/torch/csrc/jit/api/function_impl.h:4,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/torch/csrc/jit/api/method.h:7,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/torch/csrc/jit/api/object.h:6,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/torch/csrc/jit/api/module.h:4,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/serialize/input-archive.h:6,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/serialize/archive.h:3,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/samplers/serialize.h:4,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/samplers.h:8,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/datasets/chunk.h:7,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/datasets.h:4,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data.h:4,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/all.h:9,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/torch/extension.h:5,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/deepspeed/ops/csrc/includes/cpu_adam.h:12,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/deepspeed/ops/csrc/adam/cpu_adam.cpp:6:
/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/ATen/core/interned_strings.h:354:1: note:   ‘c10::attr::value’
 FORALL_NS_SYMBOLS(DEFINE_SYMBOL)
 ^
/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/ATen/core/interned_strings.h:354:1: note:   ‘c10::attr::value’
In file included from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch_npu/include/torch_npu/csrc/core/npu/NPUException.h:16:0,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch_npu/include/torch_npu/csrc/core/npu/NPUStream.h:10,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/deepspeed/ops/csrc/includes/cpu_adam.h:24,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/deepspeed/ops/csrc/adam/cpu_adam.cpp:6:
/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch_npu/include/torch_npu/csrc/core/npu/interface/AclInterface.h:159:66: error: expression list treated as compound expression in initializer [-fpermissive]
 aclError AclGetCannAttribute(aclCannAttr cannAttr, int32_t *value);
                                                                  ^
/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch_npu/include/torch_npu/csrc/core/npu/interface/AclInterface.h:161:52: error: ‘aclDeviceInfo’ has not been declared
 aclError AclGetDeviceCapability(uint32_t deviceId, aclDeviceInfo deviceInfo, int64_t *value);
                                                    ^~~~~~~~~~~~~
[2/3] c++ -MMD -MF cpu_adam_impl.o.d -DTORCH_EXTENSION_NAME=cpu_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/usr/local/Ascend/ascend-toolkit/latest/include -I/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch_npu/include -I/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/deepspeed/ops/csrc/includes -isystem /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include -isystem /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/TH -isystem /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/conda/envs/hf-dev/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O3 -std=c++17 -g -Wno-reorder -fopenmp -fstack-protector-all -Wl,-z,relro,-z,now,-z,noexecstack -Wl,--disable-new-dtags,--rpath -D__ENABLE_CANN__ -march=native -D__SCALAR__ -L/usr/local/Ascend/ascend-toolkit/latest/lib64 -L/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch_npu/lib -c /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/deepspeed/ops/csrc/adam/cpu_adam_impl.cpp -o cpu_adam_impl.o
FAILED: cpu_adam_impl.o
c++ -MMD -MF cpu_adam_impl.o.d -DTORCH_EXTENSION_NAME=cpu_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/usr/local/Ascend/ascend-toolkit/latest/include -I/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch_npu/include -I/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/deepspeed/ops/csrc/includes -isystem /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include -isystem /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/TH -isystem /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/conda/envs/hf-dev/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O3 -std=c++17 -g -Wno-reorder -fopenmp -fstack-protector-all -Wl,-z,relro,-z,now,-z,noexecstack -Wl,--disable-new-dtags,--rpath -D__ENABLE_CANN__ -march=native -D__SCALAR__ -L/usr/local/Ascend/ascend-toolkit/latest/lib64 -L/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch_npu/lib -c /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/deepspeed/ops/csrc/adam/cpu_adam_impl.cpp -o cpu_adam_impl.o
In file included from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch_npu/include/torch_npu/csrc/core/npu/NPUException.h:16:0,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch_npu/include/torch_npu/csrc/core/npu/NPUStream.h:10,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/deepspeed/ops/csrc/includes/cpu_adam.h:24,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/deepspeed/ops/csrc/adam/cpu_adam_impl.cpp:12:
/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch_npu/include/torch_npu/csrc/core/npu/interface/AclInterface.h:157:40: error: ‘aclCannAttr’ does not name a type; did you mean ‘aclopAttr’?
 aclError AclGetCannAttributeList(const aclCannAttr **cannAttrList, size_t *num);
                                        ^~~~~~~~~~~
                                        aclopAttr
/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch_npu/include/torch_npu/csrc/core/npu/interface/AclInterface.h:159:30: error: ‘aclCannAttr’ was not declared in this scope
 aclError AclGetCannAttribute(aclCannAttr cannAttr, int32_t *value);
                              ^~~~~~~~~~~
/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch_npu/include/torch_npu/csrc/core/npu/interface/AclInterface.h:159:30: note: suggested alternative: ‘aclopAttr’
 aclError AclGetCannAttribute(aclCannAttr cannAttr, int32_t *value);
                              ^~~~~~~~~~~
                              aclopAttr
/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch_npu/include/torch_npu/csrc/core/npu/interface/AclInterface.h:159:60: error: expected primary-expression before ‘*’ token
 aclError AclGetCannAttribute(aclCannAttr cannAttr, int32_t *value);
                                                            ^
/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch_npu/include/torch_npu/csrc/core/npu/interface/AclInterface.h:159:61: error: ‘value’ was not declared in this scope
 aclError AclGetCannAttribute(aclCannAttr cannAttr, int32_t *value);
                                                             ^~~~~
/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch_npu/include/torch_npu/csrc/core/npu/interface/AclInterface.h:159:61: note: suggested alternatives:
In file included from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/torch/csrc/jit/ir/ir.h:18:0,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/torch/csrc/jit/api/function_impl.h:4,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/torch/csrc/jit/api/method.h:7,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/torch/csrc/jit/api/object.h:6,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/torch/csrc/jit/api/module.h:4,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/serialize/input-archive.h:6,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/serialize/archive.h:3,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/samplers/serialize.h:4,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/samplers.h:8,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/datasets/chunk.h:7,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/datasets.h:4,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data.h:4,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/all.h:9,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/torch/extension.h:5,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/deepspeed/ops/csrc/adam/cpu_adam_impl.cpp:6:
/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/ATen/core/interned_strings.h:354:1: note:   ‘c10::attr::value’
 FORALL_NS_SYMBOLS(DEFINE_SYMBOL)
 ^
/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/include/ATen/core/interned_strings.h:354:1: note:   ‘c10::attr::value’
In file included from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch_npu/include/torch_npu/csrc/core/npu/NPUException.h:16:0,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch_npu/include/torch_npu/csrc/core/npu/NPUStream.h:10,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/deepspeed/ops/csrc/includes/cpu_adam.h:24,
                 from /usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/deepspeed/ops/csrc/adam/cpu_adam_impl.cpp:12:
/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch_npu/include/torch_npu/csrc/core/npu/interface/AclInterface.h:159:66: error: expression list treated as compound expression in initializer [-fpermissive]
 aclError AclGetCannAttribute(aclCannAttr cannAttr, int32_t *value);
                                                                  ^
/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch_npu/include/torch_npu/csrc/core/npu/interface/AclInterface.h:161:52: error: ‘aclDeviceInfo’ has not been declared
 aclError AclGetDeviceCapability(uint32_t deviceId, aclDeviceInfo deviceInfo, int64_t *value);
                                                    ^~~~~~~~~~~~~
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 2100, in _run_ninja_build
    subprocess.run(
  File "/usr/local/conda/envs/hf-dev/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "train.py", line 43, in <module>
    deepspeed.ops.op_builder.CPUAdamBuilder().load()
  File "/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/deepspeed/ops/op_builder/builder.py", line 480, in load
    return self.jit_load(verbose)
  File "/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/deepspeed/ops/op_builder/builder.py", line 524, in jit_load
    op_module = load(name=self.name,
  File "/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1308, in load
    return _jit_compile(
  File "/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1710, in _jit_compile
    _write_ninja_file_and_build_library(
  File "/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1823, in _write_ninja_file_and_build_library
    _run_ninja_build(
  File "/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 2116, in _run_ninja_build
    raise RuntimeError(message) from e
RuntimeError: Error building extension 'cpu_adam'
Loading extension module cpu_adam...
Traceback (most recent call last):
  File "train.py", line 43, in <module>
    deepspeed.ops.op_builder.CPUAdamBuilder().load()
  File "/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/deepspeed/ops/op_builder/builder.py", line 480, in load
    return self.jit_load(verbose)
  File "/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/deepspeed/ops/op_builder/builder.py", line 524, in jit_load
    op_module = load(name=self.name,
  File "/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1308, in load
    return _jit_compile(
  File "/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1736, in _jit_compile
    return _import_module_from_library(name, build_directory, is_python_module)
  File "/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 2136, in _import_module_from_library
    module = importlib.util.module_from_spec(spec)
  File "<frozen importlib._bootstrap>", line 556, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1166, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: /root/.cache/torch_extensions/py38_cpu/cpu_adam/cpu_adam.so: cannot open shared object file: No such file or directory
Loading extension module cpu_adam...
Traceback (most recent call last):
  File "train.py", line 43, in <module>
    deepspeed.ops.op_builder.CPUAdamBuilder().load()
  File "/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/deepspeed/ops/op_builder/builder.py", line 480, in load
    return self.jit_load(verbose)
  File "/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/deepspeed/ops/op_builder/builder.py", line 524, in jit_load
    op_module = load(name=self.name,
  File "/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1308, in load
    return _jit_compile(
  File "/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1736, in _jit_compile
    return _import_module_from_library(name, build_directory, is_python_module)
  File "/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 2136, in _import_module_from_library
    module = importlib.util.module_from_spec(spec)
  File "<frozen importlib._bootstrap>", line 556, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1166, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: /root/.cache/torch_extensions/py38_cpu/cpu_adam/cpu_adam.so: cannot open shared object file: No such file or directory
Loading extension module cpu_adam...
Traceback (most recent call last):
  File "train.py", line 43, in <module>
    deepspeed.ops.op_builder.CPUAdamBuilder().load()
  File "/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/deepspeed/ops/op_builder/builder.py", line 480, in load
    return self.jit_load(verbose)
  File "/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/deepspeed/ops/op_builder/builder.py", line 524, in jit_load
    op_module = load(name=self.name,
  File "/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1308, in load
    return _jit_compile(
  File "/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1736, in _jit_compile
    return _import_module_from_library(name, build_directory, is_python_module)
  File "/usr/local/conda/envs/hf-dev/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 2136, in _import_module_from_library
    module = importlib.util.module_from_spec(spec)
  File "<frozen importlib._bootstrap>", line 556, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1166, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: /root/.cache/torch_extensions/py38_cpu/cpu_adam/cpu_adam.so: cannot open shared object file: No such file or directory
[2024-05-21 12:20:20,020] [INFO] [launch.py:319:sigkill_handler] Killing subprocess 684505
[2024-05-21 12:20:20,041] [INFO] [launch.py:319:sigkill_handler] Killing subprocess 684506
[2024-05-21 12:20:20,137] [INFO] [launch.py:319:sigkill_handler] Killing subprocess 684507
[2024-05-21 12:20:20,141] [INFO] [launch.py:319:sigkill_handler] Killing subprocess 684508
[2024-05-21 12:20:20,141] [ERROR] [launch.py:325:sigkill_handler] ['/usr/local/conda/envs/hf-dev/bin/python', '-u', 'train.py', '--local_rank=3', '--train_args_file', 'train_args/sft/full/qwen2-7b-sft-full.json'] exits with return code = 1

日志提供方式:
将日志打包后作为附件上传。若日志大小超出附件限制，则可上传至外部网盘后提供链接。

获取方法请参考wiki：
https://gitee.com/ascend/modelzoo/wikis/%E5%A6%82%E4%BD%95%E8%8E%B7%E5%8F%96%E6%97%A5%E5%BF%97%E5%92%8C%E8%AE%A1%E7%AE%97%E5%9B%BE?sort_id=4097825

deepspeed使用问题建议在deepspeed仓提问

Ascend/pytorch

内容风险标识

评论 (1)

Ascend/pytorch .gitee-modal { width: 500px !important; }

内容风险标识