1 Star 0 Fork 0

ylxy123 / pkuseg-python

Create your Gitee Account
Explore and code with more than 12 million developers,Free private repositories !:)
Sign up
Clone or Download
multiprocess.md 831 Bytes
AI 代码解读
Copy Edit Raw Blame History
jingjingxupku authored 2019-04-09 15:47 . update readme

多进程分词

当将以上代码示例置于文件中运行时,如涉及多进程功能,请务必使用if __name__ == '__main__'保护全局语句,如:
mp.py文件

import pkuseg

if __name__ == '__main__':
    pkuseg.test('input.txt', 'output.txt', nthread=20)
    pkuseg.train('msr_training.utf8', 'msr_test_gold.utf8', './models', nthread=20)	

运行

python3 mp.py

详见无法使用多进程分词和训练功能,提示RuntimeError和BrokenPipeError

在Windows平台上,请当文件足够大时再使用多进程分词功能,详见关于多进程速度问题

1
https://gitee.com/ylxy123/pkuseg-python.git
git@gitee.com:ylxy123/pkuseg-python.git
ylxy123
pkuseg-python
pkuseg-python
master

Search