使用atexit调用aclFinalize会出现segmentation fault的问题

一、问题现象（附报错日志上下文）：

***注***：目前paddle-lite使用此方法在程序退出前调用aclFinalize()，也有[相关的issue](https://github.com/PaddlePaddle/Paddle-Lite/issues/9087)指出遇到同样的segmetnation fault问题。此问题在[opencv dnn中实现cann后端](https://github.com/opencv/opencv/pull/22258)时发现。

多次执行会有不同的报错，总的来说观察到有以下三种（或更多）：
1. Segmentation fault (core dumped)
2. double free or corruption (!prev)
3. bus error

二、软件版本:
-- CANN 版本 (e.g., CANN 3.0.x，5.x.x):  5.1.RC2.alpha007
--Tensorflow/Pytorch/MindSpore 版本: 无
--Python 版本 (e.g., Python 3.7.5): 3.7.5
-- MindStudio版本 (e.g., MindStudio 2.0.0 (beta3)): 无
--操作系统版本 (e.g., Ubuntu 18.04): Ubuntu 18.04.1 LTS

三、测试步骤：
1. 源代码中使用atexit调用aclFinalize：
    ```cpp
    static void finalizeAscend() {
        std::cout << "Exiting ... " << std::endl;
        aclFinalize(); // can only be called once in one process
    }

void initACL()
    {
        static std::mutex mtx;
        static bool initialized = false;
        mtx.lock();
        if (!initialized)
        {
            aclInit(NULL);
            atexit(finalizeAscend);
            initialized = true;
        }
        mtx.unlock();
    }

int main()
    {
        // 申请管理资源
        initACL();
        // 其他代码...
    }
    ```
2. 编译，运行，可以得到正确结果，但在程序退出时会触发segmentation fault错误。错误日志在第四部分。

四、日志信息:
```
// 忽略前面日志
Exiting ...
double free or corruption (!prev)
[1]    22843 abort (core dumped)  ./build/matmuldebug
$ [ERROR] TBE(22963,python3):2022-09-15-14:42:30.311.177 [../../../../../../latest/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:29][log] [../../../../../../latest/python/site-packages/tbe/common/repository_manager/utils/common.py:97][repository_manager] The main process does not exist. We would kill multiprocess manager process: 22854.
[ERROR] TBE(22959,python3):2022-09-15-14:42:30.358.406 [../../../../../../latest/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:29][log] [../../../../../../latest/python/site-packages/tbe/common/repository_manager/route.py:72][repository_manager] Subprocess[task_distribute] raise error[]
Process ForkServerProcess-3:
Traceback (most recent call last):
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 73, in wrapper
    raise exp
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 70, in wrapper
    func(*args, **kwargs)
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 371, in task_distribute
    key, func_name, detail = task_queue.get()
  File "<string>", line 2, in get
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/managers.py", line 819, in _callmethod
    kind, result = conn.recv()
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 383, in _recv
    raise EOFError
EOFError
[ERROR] TBE(22958,python3):2022-09-15-14:42:31.184.824 [../../../../../../latest/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:29][log] [../../../../../../latest/python/site-packages/tbe/common/repository_manager/route.py:72][repository_manager] Subprocess[server] raise error[[Errno 32] Broken pipe]
Process ForkServerProcess-2:
Traceback (most recent call last):
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 73, in wrapper
    raise exp
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 70, in wrapper
    func(*args, **kwargs)
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 300, in server
    while not server_event.is_set():
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/managers.py", line 1088, in is_set
    return self._callmethod('is_set')
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/managers.py", line 818, in _callmethod
    conn.send((self._id, methodname, args, kwds))
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes
    self._send(header + buf)
  File "/usr/local/python3.7.5/lib/python3.7/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
/usr/local/python3.7.5/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 91 leaked semaphores to clean up at shutdown
  len(cache))
```

<!--
xxxx
请根据自己的运行环境参考以下方式搜集日志信息，如果涉及到算子开发相关的问题，建议也提供UT/ST测试和单算子集成测试相关的日志。

日志提供方式:
将日志打包后作为附件上传。若日志大小超出附件限制，则可上传至外部网盘后提供链接。

获取方法请参考wiki：
https://gitee.com/ascend/modelzoo/wikis/%E5%A6%82%E4%BD%95%E8%8E%B7%E5%8F%96%E6%97%A5%E5%BF%97%E5%92%8C%E8%AE%A1%E7%AE%97%E5%9B%BE?sort_id=4097825
-->

Ascend/samples

内容风险标识

评论 (6)

Ascend/samples .gitee-modal { width: 500px !important; }

内容风险标识