[ST][MS] 基于标签的分布式并行训练失败

| name       | about	                             | labels   |
| ---------- | ------------------------------------- | -------- |
| Bug Report | Use this template for reporting a bug | kind/bug |

### Describe the current behavior / 问题描述 (Mandatory / 必填)
Lenet网络，设置其中的三个算子分别在MS_WORKER上1,2,3号进程执行，其余算子在0号，启动4个worker进程训练，无法训练
 
### Environment / 环境信息 (Mandatory / 必填)
- **Hardware Environment(`Ascend`/`GPU`/`CPU`)  / 硬件环境**: 
> Please delete the backend not involved / 请删除不涉及的后端:
> /device ascend/GPU/CPU

- **Software Environment / 软件环境 (Mandatory / 必填)**:
-- MindSpore version (e.g., 1.7.0.Bxxx) :2.0.0alpha，__commit_id__ = '[sha1]:1ca41893,[branch]:(HEAD,origin/r2.0.0-alpha,r2.0.0-alpha)'
-- Python version (e.g., Python 3.7.5) :
-- OS platform and distribution (e.g., Linux Ubuntu 16.04):
-- GCC/Compiler version (if compiled from source):

- **Excute Mode / 执行模式 (Mandatory / 必填)(`PyNative`/`Graph`)**: 
> Please delete the mode not involved / 请删除不涉及的模式:
> /mode graph

### Related testcase / 关联用例 (Mandatory / 必填)
 test_ms_distributed_graph_partition_001
### Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)
1. Lenet网络，设置其中的三个算子分别在MS_WORKER上1,2,3号进程执行，其余算子在0号，启动4个worker进程训练
2.  
3.

### Describe the expected behavior / 预期结果 (Mandatory / 必填)
 训练成功
### Related log / screenshot / 日志 / 截图 (Mandatory / 必填)
 worker日志：
```
[WARNING] DISTRIBUTED(91354,7fbec8519740,python):2023-01-15-17:38:13.607.378 [mindspore/ccsrc/distributed/rpc/tcp/tcp_client.cc:66] Connect] Failed to connect to the tcp server : 127.0.0.1:6667, retry to reconnect(1/1)...
[WARNING] DISTRIBUTED(91354,7fbec8519740,python):2023-01-15-17:38:15.607.670 [mindspore/ccsrc/distributed/cluster/topology/compute_graph_node.cc:162] Register] Failed to connect to the meta server node url: 127.0.0.1:6667
[WARNING] DISTRIBUTED(91354,7fbec8519740,python):2023-01-15-17:38:15.607.722 [mindspore/ccsrc/distributed/cluster/topology/compute_graph_node.cc:310] ReconnectIfNeeded] Failed to register and try to reconnect to the meta server.
[WARNING] DISTRIBUTED(91354,7fbec8519740,python):2023-01-15-17:38:18.608.798 [mindspore/ccsrc/distributed/cluster/cluster_context.cc:190] BuildCluster] Topology build timed out., retry(1/60).
[WARNING] PS(91354,7fbec8519740,python):2023-01-15-17:38:21.609.014 [mindspore/ccsrc/ps/core/file_configuration.cc:24] Initialize] The file: is not exist.
[WARNING] DEVICE(91354,7fbec8519740,python):2023-01-15-17:38:21.609.041 [mindspore/ccsrc/plugin/device/cpu/hal/hardware/ms_collective_node.cc:33] Start] Failed to initialize the configuration for this mccl collective node.
[WARNING] DEVICE(91354,7fbec8519740,python):2023-01-15-17:38:30.093.624 [mindspore/ccsrc/plugin/device/cpu/hal/hardware/ms_collective_comm_lib.cc:204] QueryUniqueID] Retry to lookup the unique id for group nccl_world_group from the meta server node...
[WARNING] PRE_ACT(91354,7fbec8519740,python):2023-01-15-17:38:34.845.025 [mindspore/ccsrc/backend/common/pass/convert_const_input_to_attr.cc:41] Process] primitive target does not match backend: GPU, primitive_target: CPU
[WARNING] PRE_ACT(91354,7fbec8519740,python):2023-01-15-17:38:34.939.164 [mindspore/ccsrc/backend/common/pass/convert_const_input_to_attr.cc:41] Process] primitive target does not match backend: GPU, primitive_target: CPU
[WARNING] PRE_ACT(91354,7fbec8519740,python):2023-01-15-17:38:34.939.190 [mindspore/ccsrc/backend/common/pass/convert_const_input_to_attr.cc:41] Process] primitive target does not match backend: GPU, primitive_target: CPU
[WARNING] PRE_ACT(91354,7fbec8519740,python):2023-01-15-17:38:34.939.219 [mindspore/ccsrc/backend/common/pass/convert_const_input_to_attr.cc:41] Process] primitive target does not match backend: GPU, primitive_target: CPU
[WARNING] PRE_ACT(91354,7fbec8519740,python):2023-01-15-17:38:34.939.234 [mindspore/ccsrc/backend/common/pass/convert_const_input_to_attr.cc:41] Process] primitive target does not match backend: GPU, primitive_target: CPU
[WARNING] DISTRIBUTED(91354,7fbec8519740,python):2023-01-15-17:38:37.942.672 [mindspore/ccsrc/distributed/cluster/actor_route_table_proxy.cc:60] LookupRoute] Retry to get the address of actor Default/network-TrainOneStepCell/network-WithLossCell/_backbone-LeNet5/conv1-Conv2d/Conv2D-op9_3_MS_WORKER->Default/network-TrainOneStepCell/optimizer-Momentum/UpdateState-op65_0_MS_WORKER
```
sched日志：
```
[WARNING] DISTRIBUTED(91347,7fd900563740,python):2023-01-15-17:38:13.633.753 [mindspore/ccsrc/distributed/cluster/cluster_context.cc:190] BuildCluster] Topology build timed out., retry(1/210).
[WARNING] DISTRIBUTED(91347,7fd900563740,python):2023-01-15-17:38:16.633.974 [mindspore/ccsrc/distributed/cluster/cluster_context.cc:190] BuildCluster] Topology build timed out., retry(2/210).
[WARNING] DISTRIBUTED(91347,7fd900563740,python):2023-01-15-17:38:20.398.959 [mindspore/ccsrc/distributed/cluster/topology/meta_server_node.cc:79] Finalize] The meta server node can not be finalized because there are still 4 alive nodes.
[WARNING] DISTRIBUTED(91347,7fd900563740,python):2023-01-15-17:38:20.398.980 [mindspore/ccsrc/distributed/cluster/cluster_context.cc:109] Finalize] Retry to finalize the node...
[WARNING] DISTRIBUTED(91347,7fd900563740,python):2023-01-15-17:38:25.399.103 [mindspore/ccsrc/distributed/cluster/topology/meta_server_node.cc:79] Finalize] The meta server node can not be finalized because there are still 4 alive nodes.
[WARNING] DISTRIBUTED(91347,7fd900563740,python):2023-01-15-17:38:25.399.143 [mindspore/ccsrc/distributed/cluster/cluster_context.cc:109] Finalize] Retry to finalize the node...
[WARNING] DISTRIBUTED(91347,7fd900563740,python):2023-01-15-17:38:30.399.276 [mindspore/ccsrc/distributed/cluster/topology/meta_server_node.cc:79] Finalize] The meta server node can not be finalized because there are still 4 alive nodes.
```

### Special notes for this issue/备注 (Optional / 选填)

GVP MindSpore/mindspore

内容风险标识

评论 (4)

GVPMindSpore/mindspore

内容风险标识