425 Star 4.3K Fork 424

GVPPaddlePaddle / Paddle

 / 详情

se_resnext run with fluid get error

已完成
创建于  
2021-03-27 01:10

源自github用户seiriosPlus:
detail error is:

E0402 06:16:52.417307   862 grpc_client.cc:189] proc param error:name:[batch_norm_0.b_0@GRAD.trainer_0] ep:[127.0.0.1:6170] grpc error:Connect Failed
Traceback (most recent call last):
  File "/models/image_classification/se_resnext_cluster.py", line 316, in <module>
    layers=layers)
  File "/models/image_classification/se_resnext_cluster.py", line 241, in train
    fetch_list=[avg_cost, acc_top1, acc_top5])
  File "/usr/local/lib/python2.7/dist-packages/paddle/fluid/executor.py", line 373, in run
    self.executor.run(program.desc, scope, 0, True, True)
paddle.fluid.core.EnforceNotMet:  at [/paddle/paddle/fluid/operators/send_op.cc:82]
PaddlePaddle Call Stacks:
0       0x7f4a4928f542p paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int) + 482
1       0x7f4a498b61d8p paddle::operators::SendOp::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) const + 4488
2       0x7f4a49907574p paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) + 52
3       0x7f4a49314252p paddle::framework::Executor::RunPreparedContext(paddle::framework::ExecutorPrepareContext*, paddle::framework::Scope*, bool, bool) + 1138
4       0x7f4a49315119p paddle::framework::Executor::Run(paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool) + 89
5       0x7f4a492a2491p void pybind11::cpp_function::initialize<pybind11::cpp_function::initialize<void, paddle::framework::Executor, paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool, pybind11::name, pybind11::is_method, pybind11::sibling>(void (paddle::framework::Executor::*)(paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&)::{lambda(paddle::framework::Executor*, paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool)#1}, void, paddle::framework::Executor*, paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool, pybind11::name, pybind11::is_method, pybind11::sibling>(pybind11::cpp_function::initialize<void, paddle::framework::Executor, paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool, pybind11::name, pybind11::is_method, pybind11::sibling>(void (paddle::framework::Executor::*)(paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&)::{lambda(paddle::framework::Executor*, paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool)#1}&&, void (*)(paddle::framework::Executor*, paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call) + 545
6       0x7f4a4929c452p pybind11::cpp_function::dispatcher(_object*, _object*, _object*) + 2338
7             0x4c37edp PyEval_EvalFrameEx + 31165
8             0x4b9ab6p PyEval_EvalCodeEx + 774
9             0x4c16e7p PyEval_EvalFrameEx + 22711
10            0x4b9ab6p PyEval_EvalCodeEx + 774
11            0x4c16e7p PyEval_EvalFrameEx + 22711
12            0x4b9ab6p PyEval_EvalCodeEx + 774
13            0x4eb30fp
14            0x4e5422p PyRun_FileExFlags + 130
15            0x4e3cd6p PyRun_SimpleFileExFlags + 390
16            0x493ae2p Py_Main + 1554
17      0x7f4a92ff5830p __libc_start_main + 240
18            0x4933e9p _start + 41

OS: docker ubuntu 16.04
PaddlePaddle: GPU-latest
Run Fluid with 4 Trainer and 4 Pserver
Code: https://github.com/seiriosPlus/fluid_benchmark/blob/master/image_classification/se_resnext_cluster.py

评论 (1)

PaddlePaddle-Gardener 创建了任务
PaddlePaddle-Coordinator 任务状态待办的 修改为已完成
展开全部操作日志

登录 后才可以发表评论

状态
负责人
里程碑
Pull Requests
关联的 Pull Requests 被合并后可能会关闭此 issue
分支
开始日期   -   截止日期
-
置顶选项
优先级
参与者(1)
Python
1
https://gitee.com/paddlepaddle/Paddle.git
git@gitee.com:paddlepaddle/Paddle.git
paddlepaddle
Paddle
Paddle

搜索帮助