From c8134a5e640d2bac82619a271661be77b0c4cca3 Mon Sep 17 00:00:00 2001 From: gitee-bot Date: Mon, 14 Jul 2025 03:46:48 +0000 Subject: [PATCH] Update README.md --- README.en.md | 73 ++++++++++++++++++++++++++++++++++++++++++++++++ README.md | 79 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 152 insertions(+) create mode 100644 README.en.md create mode 100644 README.md diff --git a/README.en.md b/README.en.md new file mode 100644 index 0000000..b278982 --- /dev/null +++ b/README.en.md @@ -0,0 +1,73 @@ +# OssMove - OSS Migration Tool + +## Introduction + +OssMove is a tool for migrating files from Alibaba Cloud OSS to MinIO or other OSS instances. It supports multi-threaded migration, resumable uploads, and retry mechanisms for failed files, ensuring the migration process is efficient and reliable. + +## Features + +- **OSS to MinIO Migration**: Supports migrating files from Alibaba Cloud OSS to MinIO. +- **OSS to OSS Migration**: Supports migrating files within the same or different OSS instances. +- **Multi-threaded Processing**: Utilizes a thread pool to improve migration efficiency. +- **Resumable Uploads**: Avoids re-migrating already processed files by recording migration progress. +- **Retry Mechanism for Failures**: Automatically logs failed files and supports retry functionality. +- **Logging**: Records failed files and their error messages during migration for easy troubleshooting. + +## Usage Instructions + +### Configuration Parameters + +Before use, modify the following static constants according to your actual setup: + +- `OSS_ENDPOINT`: The endpoint address of Alibaba Cloud OSS. +- `OSS_ACCESS_KEY_ID`: The access ID of Alibaba Cloud OSS. +- `OSS_ACCESS_KEY_SECRET`: The access key of Alibaba Cloud OSS. +- `OSS_BUCKET_NAME`: The bucket name of Alibaba Cloud OSS. +- `MINIO_ENDPOINT`: The endpoint address of the MinIO service. +- `MINIO_ACCESS_KEY`: The access key ID of MinIO. +- `MINIO_SECRET_KEY`: The access key of MinIO. +- `MINIO_BUCKET_NAME`: The bucket name of MinIO. +- `MINIO_REGION`: The region name of MinIO. +- `THREAD_POOL_SIZE`: The size of the thread pool, which controls the number of concurrent migration threads. +- `PAGE_SIZE`: The number of files retrieved from OSS at each batch. +- `MAX_RETRY`: The maximum number of retries for each file. +- `PROGRESS_FILE`: The file path used to store migration progress. +- `FAILED_LOG_FILE`: The file path used to log files that failed during migration. + +### Running the Program + +1. **OSS to MinIO Migration**: Run the `main` method of the `OssToMinioMigratorPrintTxt` class. +2. **OSS to OSS Migration**: Run the `main` method of the `OssToOssMigrator` class. + +### Migration Process + +- **Load Progress**: Load the list of already migrated files and the current marker from a local file. +- **Migrate Files**: Migrate files using multiple threads and record migration progress. +- **Retry Failures**: After migration completes, call the `retryFailedFiles` method to retry failed files. +- **Release Resources**: After migration completes, close the OSS and MinIO clients as well as the thread pool. + +## Notes + +- Ensure that the access keys and endpoint addresses for OSS and MinIO are configured correctly. +- It is recommended to perform large-scale file migrations during off-peak hours to minimize service impact. +- Do not delete or modify the progress file or failure log file during migration, as this may cause loss of migration state. + +## Dependencies + +This project is built on the following libraries: + +- Alibaba Cloud OSS SDK +- MinIO Java SDK +- Java 8 or higher + +## License + +This project uses the MIT License. For details, please refer to the [LICENSE](LICENSE) file. + +## Contact + +If you have any questions or suggestions, feel free to contact the developer. + +--- + +Please adjust the relevant parameters according to your actual project requirements and environment configuration to ensure smooth execution of the migration task! \ No newline at end of file diff --git a/README.md b/README.md new file mode 100644 index 0000000..a7eb150 --- /dev/null +++ b/README.md @@ -0,0 +1,79 @@ + + +看起来该项目主要是实现将文件从阿里云OSS迁移到MinIO的服务，同时也提供了OSS到OSS的迁移功能。以下是该项目的README.md内容： + +--- + +# OssMove - OSS 迁移工具 + +## 简介 + +OssMove 是一个用于将文件从阿里云OSS迁移到MinIO或其他OSS实例的工具。它支持多线程迁移、断点续传以及失败文件的重试机制，确保迁移过程高效且可靠。 + +## 功能特点 + +- **OSS 到 MinIO 迁移**：支持将文件从阿里云OSS迁移到MinIO。 +- **OSS 到 OSS 迁移**：支持在同一或不同OSS实例之间迁移文件。 +- **多线程处理**：利用线程池提高迁移效率。 +- **断点续传**：通过记录迁移进度，避免重复迁移已处理文件。 +- **失败重试机制**：自动记录失败文件，并支持重试功能。 +- **日志记录**：记录迁移过程中失败的文件及其错误信息，便于排查问题。 + +## 使用说明 + +### 配置参数 + +在使用前，请根据实际情况修改以下静态常量： + +- `OSS_ENDPOINT`: 阿里云OSS的端点地址。 +- `OSS_ACCESS_KEY_ID`: 阿里云OSS的访问ID。 +- `OSS_ACCESS_KEY_SECRET`: 阿里云OSS的访问密钥。 +- `OSS_BUCKET_NAME`: 阿里云OSS的Bucket名称。 +- `MINIO_ENDPOINT`: MinIO服务的端点地址。 +- `MINIO_ACCESS_KEY`: MinIO的访问密钥ID。 +- `MINIO_SECRET_KEY`: MinIO的访问密钥。 +- `MINIO_BUCKET_NAME`: MinIO的Bucket名称。 +- `MINIO_REGION`: MinIO的Region名称。 +- `THREAD_POOL_SIZE`: 线程池的大小，控制并发迁移的线程数。 +- `PAGE_SIZE`: 每次从OSS获取的文件数量。 +- `MAX_RETRY`: 每个文件的最大重试次数。 +- `PROGRESS_FILE`: 用于存储迁移进度的文件路径。 +- `FAILED_LOG_FILE`: 用于记录迁移失败文件的文件路径。 + +### 运行程序 + +1. **OSS 到 MinIO 迁移**：运行 `OssToMinioMigratorPrintTxt` 类的 `main` 方法。 +2. **OSS 到 OSS 迁移**：运行 `OssToOssMigrator` 类的 `main` 方法。 + +### 迁移流程 + +- **加载进度**：从本地文件加载已迁移的文件列表和当前的marker。 +- **迁移文件**：通过多线程迁移文件，并记录迁移进度。 +- **失败重试**：迁移完成后，可以调用 `retryFailedFiles` 方法对失败的文件进行重试。 +- **关闭资源**：迁移完成后，关闭OSS和MinIO客户端，以及线程池。 + +## 注意事项 + +- 请确保OSS和MinIO的访问密钥和端点地址配置正确。 +- 建议在低峰期进行大规模文件迁移，以减少对服务的影响。 +- 迁移过程中请勿删除或修改进度文件和失败日志文件，以免导致迁移状态丢失。 + +## 依赖 + +该项目基于以下依赖库： + +- 阿里云OSS SDK +- MinIO Java SDK +- Java 8或更高版本 + +## 许可证 + +本项目使用 MIT 许可证。有关详细信息，请参阅 [LICENSE](LICENSE) 文件。 + +## 联系方式 + +如有任何问题或建议，欢迎联系开发者。 + +--- + +请根据实际的项目需求和环境配置调整相关参数，确保迁移任务顺利执行！ \ No newline at end of file -- Gitee