What type of PR is this?
Uncomment only one
/kind <>line, hit enter to put that in a new line, and remove leading whitespaces from that line:
What does this PR do / why do we need it:
Add multiprocessing support for Pyfunc
Which issue(s) this PR fixes:
Special notes for your reviewers:
This add multiprocess support for Pyfunc.
The following pieces are involved.
- Add new option python_multiprocess (default to false) to mapOp.
- On iterator construction, create process pool in python. (pool will be GCed along with iterator)
- Wrap each user pyfunc to make it executed in process pool.