Score
0
Watch 32 Star 113 Fork 24

WeBank / ExchangisJavaApache-2.0

Join us
Explore and code with more than 5 million developers,Free private repositories !:)
Sign up
Exchangis是微众银行开源的轻量级数据交换平台,用于数据在不同存储介质中快速传递,并解决数据交换过程中面临的复杂性、兼容性等问题。设计上采取微服务架构形态,上下层服务松耦合,易于进行个性化的,高拓展性的迭代开发。 spread retract

Clone or download
Cancel
Notice: Creating folder will generate an empty file .keep, because not support in Git
Loading...
README.md

License

English | 中文

Introduction

Exchangis is a lightweight,highly extensible data exchange platform that supports data transmission between structured and unstructured heterogeneous data sources. On the application layer, it has business features such as data permission management and control, high availability of node services and multi-tenant resource isolation. On the data layer, it also has architectural characteristics such as diversified transmission architecture, module plug-in and low coupling of components.

Exchnagis's transmission and exchange capabilities depend on its underlying aggregated transmission engines. It defines a unified parameter model for various data sources on the top layer. It maps and configures the parameter model for each transmission engine, and then converts it into the engine's input model. Each type of engine will add Exchangis features, and the enhancement of certain engine features will improve the Exchangis features. Exchangis's default engine aggregated and enhanced is Alibaba's DataX transmission engine.

Features

  • Data Source Management
    Share your own data source in a bound project;
    Set the external authority of the data source to control the inflow and outflow of data。

  • Muti-transport Engine Support
    Transmission engine scales horizontally;
    The current version fully aggregates the offline batch engine DataX and partially aggregates the big data batch derivative engine SQOOP

  • Near Real-time Task Control
    Quickly capture the transmission task log, transmission rate and other information, close the task in real time;
    Dynamically limit transmission rate based on bandwidth

  • Support Unstructured Transmission
    Transform the DataX framework and build a binary stream fast channel separately, suitable for pure data synchronization scenarios without data conversion。

  • Task Status Self-check
    Monitor long-running tasks and tasks with abnormal status, release occupied resources in time and issue alarms。

Comparison With Existing Systems

Comparison of some existing data exchange tools and platforms:

Function module Description Exchangis DataX Sqoop DataLink DBus
UI Integrated the convenient management interface and monitoring window Integrated None None Integrated Integrated
Installation and deployment Ease of deployment and third-party dependencies One-click deployment, no dependencies No dependencies Rely on Hadoop environment Rely on Zookeeper Rely on a large number of third-party components
Data authority management Multi-tenant permission configuration and data source permission control Support Not support Not support Not support Support
Dynamic limit transmission Support Partially supported, unable to adjust dynamically Partially supported, unable to adjust dynamically Support Support,with Kafka
Data transmission Unstructured data binary transmission Support, fast channel Not support Not support Not support,only transport record Not support,need to be converted to a unified message format
Embed processing code Support,dynamic compilation Not support Not support Not support Partial support
Transmission breakpoint recovery Support(Not open source) Not support Not support Support Support
High availability Mutiple services, failure does not affect the use Application high availability, transmission single point(Distributed architecture planning) Single point service(Open source version) Multipoint transmission Application、transmission high availability Application、transmission high availability
System Management Nodes、resources management Support Not support Not support Support Support

Overall Design

Architecture

Architecture

Documents

Quick Deploy
User Manual

Communication

If you desire immediate response, please kindly raise issues to us or scan the below QR code by WeChat and QQ to join our group:

Communication

License

Exchangis is under the Apache 2.0 License. See the License file for details.

Comments ( 0 )

Sign in for post a comment

Java
1
https://gitee.com/WeBank/Exchangis.git
git@gitee.com:WeBank/Exchangis.git
WeBank
Exchangis
Exchangis
master

Help Search

205735 778617b6 1899542 205747 347fc4a1 1899542