13 Star 81 Fork 24

dromara / hodor

加入 Gitee
与超过 1200万 开发者一起发现、参与优秀开源项目,私有仓库也完全免费 :)
免费加入
克隆/下载
贡献代码
同步代码
取消
提示: 由于 Git 不支持空文件夾,创建文件夹后会生成空的 .keep 文件
Loading...
README
Apache-2.0

HodorScheduler

介绍

Hodor是一个专注于任务调度以及任务编排一站式分布式任务调度系统。

通过提供任务编程API和接口协议, 方便支持用户使用不同的编程语言实现任务执行以及自定义任务类型的扩展,以满足各种业务场景下的任务执行。

支持多种接入方式,支持Java SDK包方式、Java Agent独立进程方式接入,以及兼容XXLJob框架、Spring Task框架任务无缝接入。

多语言执行器实现,计划支持Go或者Rust语言执行器(正在设计与开发中)的接入,满足在资源有限的执行器节点执行,比如物联网场景下面的定时任务执行。

支持丰富的任务类型,除了支持普通的Java、脚本任务的执行,也支持了大数据任务Hadoop、Spark、Flink、Kettle等主流任务的执行,满足用户的不同场景。

主要特性

  1. 支持Cron、FixedRate、FixedDelay、OnceTime方式配置定时任务,可以方便地实现各种定时任务需求。
  2. 支持DAG方式进行任务编排,使得任务之间的关系更加清晰,同时提高了系统的可扩展性和灵活性。
  3. 支持Java任务和大数据任务调度,兼容XXLJob、SpringTask调度任务,满足不同场景下的任务调度需求。
  4. 支持可扩展执行器,用户可以根据自己的需求自定义实现任务类型,从而更好地适应不同的业务场景。
  5. 支持任务静态分片、动态分片、广播、WorkFlow等多种任务执行模式。
  6. 支持任务自动创建与手动创建以及批量创建等方式,任务的暂停、恢复、Kill等动作,以及实时日志查看等。
  7. 基于CopySet算法实现任务副本分配,提高了任务的可靠性和容错性,从而保障了任务的高可用性。

详细文档请查阅:https://www.yuque.com/tomgs/hodor

架构设计

hodor架构设计图

快速开始

项目结构:

hodor-scheduler
├─docs                                          # 文档存放位置
├─hodor-actuator                                # hodor任务执行器
│  ├─hodor-actuator-api                         # 执行器扩展接口
│  ├─hodor-actuator-java                        # jar包方式任务执行器
│  ├─hodor-actuator-agent                       # 独立进程方式任务执行器
│  ├─hodor-actuator-xxljob                      # 支持xxl-job执行器
│  └─hodor-actuator-springtask                  # 支持spring task执行器
├─hodor-admin                                   # hodor管理控制台(TODO)
├─hodor-client                                  # 提供与hodor server的交互sdk
├─hodor-common                                  # hodor公共依赖
├─hodor-core                                    # hodor核心业务
├─hodor-examples                                # hodor测试用例
├─hodor-cache                                   # hodor扩展
│  ├─hodor-cache-local                          # 基于本地内存的缓存
│  ├─hodor-cache-embedded                       # 基于embedded的分布式缓存
│  └─hodor-cache-redis                          # 基于redis的分布式缓存
├─hodor-model                                   # hodor公共数据模型
├─hodor-storage                                 # hodor storage模块,任务资源管理
├─hodor-register                                # hodor注册中心
│  ├─hodor-register-api
│  ├─hodor-register-embedded                    # 基于Raft实现注册中心
│  └─hodor-register-zookeeper                   # 基于zk实现注册中心
├─hodor-remoting                                # hodor通信框架
│  ├─hodor-remoting-api
│  └─hodor-remoting-netty
├─hodor-scheduler                               # hodor调度器实现
│  ├─hodor-scheduler-api
│  └─hodor-scheduler-quartz
└─hodor-server                                  # hodor调度服务

本地源码运行方式

  1. 在mysql下执行docs目录下面的hodor_ddl.sql。
  2. 在项目根目录下通过mvn package -DskipTests=true命令编译整个项目,因为涉及到一些代码的自动生成。
  3. 进入到hodor-server,启动HodorServer调度器。
  4. 进入到examples下面的hodor-actuator-java-example,启动java任务执行器程序
  5. java执行器启动完成后会自动将任务注册到调度器里面来

任务使用说明

详细参考 执行器操作指南

java任务接入demo

详细demo代码参考 hodor-examples/hodor-actuator-java-example 模块

普通java任务

@Job(group = "testGroup", jobName = "test1", cron = "0/30 * * * * ?")
public String test1(JobExecutionContext context) {
    log.info("Job [testGroup#test1] execute, context: {}", context);
    JobLogger logger = context.getJobLogger();
    logger.info("start executor job test1");
    logger.info("job argument: {}", context.getJobParameter());
    logger.info("executing......");
    logger.info("executed");
    return "a=123";
}

动态分片任务

// 任务动态切分
@Job(command = "splitStage")
public List<ShardData> split(JobExecutionContext context) {
    ShardData shardData = ShardData.builder().id(0).params("hello").build();
    ShardData shardData1 = ShardData.builder().id(1).params("world").build();
    return Lists.newArrayList(shardData, shardData1);
}

// 任务并行执行
@Job(command = "parallelJob")
public String parallelJob(JobExecutionContext context) {
    log.info("Job [parallelJob] execute, context: {}", context);
    JobLogger logger = context.getJobLogger();
    final ShardData parentJobData = context.getParentJobData(ShardData.class);
    logger.info("start executor job parallelJob, parentJobData: {}", parentJobData);
    log.info("start executor job parallelJob, parentJobData: {}", parentJobData);
    return "a=123";
}

// 任务汇总执行
@Job(command = "reduceJob2")
public void reduceJob(JobExecutionContext context) {
    log.info("Job [reduceJob] execute, context: {}", context);
    JobLogger logger = context.getJobLogger();
    logger.info("job argument: {}", context.getJobParameter());
    logger.info("executed");
    logger.info("Job [reduceJob] execute, context: {}", context);
    logger.info("Job [reduceJob] execute, results {}", context.getParentJobExecuteResults()
    );
    logger.info("Job [reduceJob] execute, statues {}", context.getParentJobExecuteStatuses());
}

xxljob任务接入demo

详细demo代码参考 hodor-examples/hodor-actuator-xxljob-example 模块

/**
 * 1、简单任务示例(Bean模式)
 */
@XxlJob("demoJobHandler")
public void demoJobHandler() throws Exception {
    XxlJobHelper.log("XXL-JOB, Hello World.");

    for (int i = 0; i < 5; i++) {
        XxlJobHelper.log("beat at:" + i);
        TimeUnit.SECONDS.sleep(2);
    }
    // default success
}

/**
 * 2、分片广播任务
 */
@XxlJob("shardingJobHandler")
public void shardingJobHandler() throws Exception {
    // 分片参数
    int shardIndex = XxlJobHelper.getShardIndex();
    int shardTotal = XxlJobHelper.getShardTotal();

    XxlJobHelper.log("分片参数:当前分片序号 = {}, 总分片数 = {}", shardIndex, shardTotal);

    // 业务逻辑
    for (int i = 0; i < shardTotal; i++) {
        if (i == shardIndex) {
            XxlJobHelper.log("第 {} 片, 命中分片开始处理", i);
        } else {
            XxlJobHelper.log("第 {} 片, 忽略", i);
        }
    }
}

spring task任务接入demo

详细demo代码参考 hodor-examples/hodor-actuator-springtask-example 模块

@Scheduled(fixedRate = 3000)
public void task1() throws InterruptedException {
    TimeUnit.SECONDS.sleep(2);
    System.out.println("执行 fixedRate 任务的时间:" + new Date(System.currentTimeMillis()));
}

@Scheduled(fixedDelay = 4000)
public void task2() throws InterruptedException {
    TimeUnit.SECONDS.sleep(2);
    System.out.println("执行 fixedDelay 任务的时间:" + new Date(System.currentTimeMillis()));
}

@Scheduled(cron = "0/5 * * * * ?")
public void task3() {
    System.out.println("执行 cron 任务的时间:" + new Date(System.currentTimeMillis()));
}

agent 方式任务接入demo

详细参考 执行器操作指南 中的agent方式的使用。

hodor打包部署说明

详细参考 安装部署操作指南

交流学习

目前,Hodor仍处于建设的过程中,团队成员只能抽出空闲时间投入其中,主要精力还是放在日常工作中。 如果有兴趣的同学想要参与Hodor的实现或者一起学习,欢迎通过微信联系我们,我们非常欢迎新的成员加入!

Hodor的未来发展方向包括以下几个方面:

  1. 前端界面和控制台开发,为用户提供更加友好、直观的任务管理界面。
  2. 执行器开发工作,包括执行编程框架和大数据任务执行器的优化,以提高系统的性能和稳定性。
  3. 通信协议改造,计划将目前自定义的协议改造为grpc的方式,以提高系统的可扩展性和兼容性。
  4. 优化CopySet算法的实现,以支持机架感知,更好地分配任务副本。
  5. 开发多语言执行器,主要放在go和rust语言,以满足用户不同的需求和偏好。

Wechat:(备注:Hodor)

image-20220529141555032

Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "{}" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright {yyyy} {name of copyright owner} Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

简介

Hodor是一款专注于任务编排和高可用性的分布式任务调度系统。通过提供任务编程API和接口协议,使用户可以使用不同的编程语言自定义任务类型,以支持各种业务场景下的任务执行。 目前已经实现了Java任务执行器和大数据任务执行器。 展开 收起
Apache-2.0
取消

发行版

暂无发行版

贡献者

全部

近期动态

加载更多
不能加载更多了
Java
1
https://gitee.com/dromara/hodor.git
git@gitee.com:dromara/hodor.git
dromara
hodor
hodor
master

搜索帮助