# MigrationX

**Repository Path**: gaoqinglong/migration-x

## Basic Information

- **Project Name**: MigrationX
- **Description**: HDFS(Hive)集群数据迁移到新集群通用解决方案
- **Primary Language**: Unknown
- **License**: MIT
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2024-10-16
- **Last Updated**: 2024-10-20

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

### MigrationX
```text
该项目是多套集群之间，hdfs/hive数据迁移的通用解决方案。
集群之间不需要配置互信，传输稳定。
```
### 1.项目优点：
- 集群之间不需要配置互信
- hdfs 数据下载增加重试
- scp 传输文件增加文件校验
- scp 文件传输增加重试

- 注意点：
    - 该项目基于pyhton2编写
    - 具体特殊情况处理有待优化

### 2.使用说明
```text
1) 安装命令：sshpass
yum install sshpass -y

2) ssh测试网络联通，一定要做
[gql@hive cores]$ ssh root@192.168.110.11
The authenticity of host '192.168.110.11 (192.168.110.11)' can't be established.
ECDSA key fingerprint is SHA256:aeNfHS9xGs1cCHgg895a4RW/qDOaiOByaYHh8RMOjhg.
ECDSA key fingerprint is MD5:11:35:7c:3a:6b:fd:9e:24:02:74:eb:25:d3:1f:d2:05.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.110.11' (ECDSA) to the list of known hosts.
root@192.168.110.11's password: 
[root@tdh-11 ~]# exit

3) sshpass测试网络联通
[gql@hive cores]$ sshpass -p 1 ssh -p 22 root@192.168.110.11 hostname
tdh-11
```

### 3.目录说明
```text
bin/      # MigrationX管理脚本
conf/     # 配置文件，清单文件
cores/    # 核心代码
tests/    # 测试
```

### 4.快速开始
参考连接：[quickStart](docs/quickStart.md)
```shell script
## hdfs_get and scp
sh migration_x.sh start hdfs_get 1001 batch.txt - 20201011
sh migration_x.sh status hdfs_get
sh migration_x.sh stop hdfs_get

## hdfs_put
sh migration_x.sh start hdfs_put
sh migration_x.sh status hdfs_put
sh migration_x.sh stop hdfs_put
```