1 Star 0 Fork 232

幽枫微梦 / earth-frost

forked from justlive1 / earth-frost 
加入 Gitee
与超过 1200万 开发者一起发现、参与优秀开源项目,私有仓库也完全免费 :)
免费加入
克隆/下载
贡献代码
同步代码
取消
提示: 由于 Git 不支持空文件夾,创建文件夹后会生成空的 .keep 文件
Loading...
README
Apache-2.0

earth-frost

earth-frost是一个轻量级分布式任务调度框架。

介绍

  • 调度模块和执行模块分离
  • 使用redis作为数据库
  • 基于订阅模式实现服务注册和发现
  • 演示地址: http://www.justlive.vip (请勿恶意攻击)

环境

  • Angular: v1.x
  • JDK: 1.8+
  • Maven: 3+
  • Redis: 2.8+
  • Spring boot: 2.x
  • Thymeleaf: 3

功能

1.0.0

  • 简单易上手,支持web界面进行任务的CRUD
  • 支持动态修改任务的开始,停止
  • 调度中心支持集群部署,将任务的调度进行封装,支持分配各种不同的任务
  • 执行器支持分布式,支持集群部署,可进行相应分组,在调度中心界面查看
  • 支持伸缩扩展,调度中心和执行器都是基于redis订阅模式进行服务注册发现和任务调度,服务上下线发现及时
  • 支持失败重试
  • 任务监控和报警
  • 动态编译任务,支持web界面编辑任务源码,创建任务
  • 支持父子任务
  • 运行报表

1.1.0

  • 支持钉钉预警通知
  • 调度记录增加执行时间
  • 任务超时报警机制
  • 支持分片任务
  • 支持SimpleTrigger任务
  • 支持非spring项目开发执行器
  • misfire

1.2.0

  • 修复分片父子任务执行多次问题
  • 增加配置最大日志数
  • 增加Spring项目执行案例

开发

frost-api
	对外实体和接口
frost-core
	定义调度、执行、注册发现等核心功能的接口和实现
frost-center
	调度中心服务,包含安全认证和UI展示,依赖core实现调度逻辑
frost-executor
	执行器服务,依赖core实现任务执行逻辑,包含开发脚本任务的部分实例以及配置样例	
frost-client
	客户端api,第三方项目可依赖client实现与调度中心交互

快速入门

启动Redis数据库

请下载并安装2.8+版本的Redis,单机或集群模式皆可。

调度中心和执行器均支持集群模式部署,集群模式下各节点需配置相同的Redis服务

导入源码

解压源码,按maven格式将源码导入IDE,源码结构如上述 [开发] 章节。

配置部署调度中心

1.配置内容说明

# 登录账号
spring.boot.auth.enabled=true
spring.security.user.name=frost
spring.security.user.password=frost

# 报警通知发送邮件
spring.mail.host=smtp.mail.com
spring.mail.username=
spring.mail.password=

# 邮件通知
frost.notifier.mail.enabled=true
frost.notifier.mail.from=${spring.mail.username}
frost.notifier.mail.to=
frost.notifier.mail.subject=#{job.name} (#{job.id}) throws an exception
frost.notifier.mail.text=#{job.name} (#{job.id}) \n #{event.message}

# 钉钉通知
frost.notifier.dingtalk.enabled=false
frost.notifier.dingtalk.subject=There is something wrong with #{job.name} (#{job.id})
frost.notifier.dingtalk.text=#{job.name} (#{job.id}) \n #{event.message}
frost.notifier.dingtalk.accessToken=
frost.notifier.dingtalk.linkUrl=localhost:20000/center

# 系统配置
frost.system.corePoolSize=50
frost.system.maximumPoolSize=50
frost.system.keepAliveTime=300
frost.system.queueCapacity=200
# center处理worker数
frost.system.workers=10
# job并行处理个数
frost.system.parallel=2

# redis配置
# 0:单机模式, 1:集群模式,2:云托管模式,3:哨兵模式,4:主从模式
frost.redisson.mode=0

# 公共配置
frost.redisson.password=
frost.redisson.slaveConnectionPoolSize=50
frost.redisson.masterConnectionPoolSize=50
frost.redisson.timeout=3000

# 单机模式
frost.redisson.address=redis://localhost:6379

# 集群模式
frost.redisson.nodeAddresses=redis://localhost:6379,redis://localhost:6380,redis://localhost:6381
frost.redisson.scanInterval=2000

# 云托管模式
frost.redisson.nodeAddresses=redis://localhost:6379,redis://localhost:6380,redis://localhost:6381
frost.redisson.scanInterval=2000
frost.redisson.dnsMonitoringInterval=5000

# 哨兵模式
frost.redisson.sentinelAddresses=redis://localhost:6379,redis://localhost:6380,redis://localhost:6381
frost.redisson.scanInterval=2000
frost.redisson.masterName=mymaster

# 主从模式
frost.redisson.masterAddress=redis://127.0.0.1:6379
frost.redisson.slaveAddresses=redis://localhost:6380,redis://localhost:6381
frost.redisson.dnsMonitoringInterval=5000

2.部署项目

正确进行上述配置,可将项目编译打包 mvn package(jar或war,相关打包操作参照Spring boot),调度中心访问地址:http://localhost:20000/center,访问可进入登录界面 登录页面 登录成功后进入报表页面 报表页面

3.集群模式

支持集群模式部署,提供调度系统可用性。

集群模式需要注意:保持 登录账号保持一致,保持使用redis配置相同。建议使用Nginx为调度中心做负载均衡

配置部署执行器

提供的frost-executor项目可直接使用,也可集成到现有业务项目中使用

1.依赖jar

在执行器项目中依赖如下jar包

<dependency>
	<groupId>vip.justlive</groupId>
	<artifactId>frost-core</artifactId>
</dependency>

2.配置说明

# 系统配置
frost.system.corePoolSize=50
frost.system.maximumPoolSize=50
frost.system.keepAliveTime=300
frost.system.queueCapacity=200
# job并行处理个数
frost.system.parallel=2

# 执行器名称
frost.executor.name=${spring.application.name}
# 执行器Key
frost.executor.key=executor-demo
frost.executor.ip=
frost.executor.port=${server.port}
# 是否支持执行脚本任务
frost.executor.scriptJobEnabled=true
# 错过执行的阈值(毫秒)
frost.executor.misfireThreshold=5000
# 每个job最大日志保留数
frost.executor.maxLogSize=-1

# redis配置
# 0:单机模式, 1:集群模式,2:云托管模式,3:哨兵模式,4:主从模式
frost.redisson.mode=0

# 公共配置
frost.redisson.password=
frost.redisson.slaveConnectionPoolSize=50
frost.redisson.masterConnectionPoolSize=50
frost.redisson.timeout=3000

# 单机模式
frost.redisson.address=redis://localhost:6379

# 集群模式
frost.redisson.nodeAddresses=redis://localhost:6379,redis://localhost:6380,redis://localhost:6381
frost.redisson.scanInterval=2000

# 云托管模式
frost.redisson.nodeAddresses=redis://localhost:6379,redis://localhost:6380,redis://localhost:6381
frost.redisson.scanInterval=2000
frost.redisson.dnsMonitoringInterval=5000

# 哨兵模式
frost.redisson.sentinelAddresses=redis://localhost:6379,redis://localhost:6380,redis://localhost:6381
frost.redisson.scanInterval=2000
frost.redisson.masterName=mymaster

# 主从模式
frost.redisson.masterAddress=redis://127.0.0.1:6379
frost.redisson.slaveAddresses=redis://localhost:6380,redis://localhost:6381
frost.redisson.dnsMonitoringInterval=5000

3.部署执行器

执行 Container.initExecutor(...) 进行初始化执行器,项目打包按照集成项目打包方式即可,部署后打开调度中心页面查看执行器页面。具体代码可参照frost-executor,包含Spring和非Spring项目启动方式 执行器列表 列表中出现部署的执行器则说明部署成功

开发一个简单的任务

该案例使用脚本任务模式,比使用实例模式只需要部署调度中心和执行器即可,脚本任务可在调度中心在线维护

1.新增任务

登录调度中心,在任务管理界面点击新增任务,任务类型选择脚本模式,输入相应参数,点击保存。 任务列表 添加脚本任务

2.脚本模式开发

在任务管理列表的操作栏,点击刚新建任务的脚本按钮,进入脚本脚本编辑页面。系统已经初始化了示例脚本任务,可按需进行修改。 修改脚本

3.触发执行

点击任务右侧“触发一次”按钮可触发一次任务(通常是配置cron定时触发)

4.查看结果

进入调度记录页面可查看调度和执行结果 调度记录

任务配置属性说明

任务类型:
	脚本模式:传统的任务模式,需要在执行器开发对应的Bean,此模式的任务只能调度分配到拥有该类的执行器上
	脚本模式:任务以Java源码的形式保存在数据库中,调度中心可将任务分配到任何开启支持脚本模式的执行器上,可使用@Resource/@Autowired注入执行器里中的其他服务
任务名称:任务的名称,描述任务以便于管理
执行器:指定运行在哪个执行器分组
执行器逻辑:指定任务运行的bean
任务模式:
	cron任务:支持cron表达式的定时任务
	简单任务:选择日期时间定时执行(当时间早于当前时间则立即执行)
	延时任务:启动任务后,固定延时执行任务
cron:触发任务的表达式(cron任务模式)
执行时间:选择执行的时间,精确到分(为什么不支持到秒?1.没有必要,2.h5的datetime-local仅支持到分,懒得用日期插件了)(简单任务模式)
延时:启动后延时时间,执行间隔时间,单位秒(延时任务模式)
子任务:可在任务列表中选择已创建的任务,当本任务执行成功时,将会触发子任务的一次主动调度
参数:运行任务的参数,字符串格式,执行逻辑可通过JobContext::getParam获取参数
失败处理:
	失败通知(默认):调度或执行失败后,触发通知,默认提供了邮件,可继承AbstractEventNotifier进行扩展
	失败重试:调度或执行失败后,会主动再次调度或执行
通知邮件:调度中心配置了全局的默认推送邮件地址,每个任务可自定义添加通知邮件列表
超时预警:可填写任务的预期处理时间,超过时间还未执行完任务则进行预警
分片运行:勾选后任务分片运行,可填写分片总数,注意:分片任务需要任务逻辑支持分片方式,否则会重复执行
自动运行:勾选后创建任务即是运行状态

开发实例模式任务

任务需要在frost-executor项目中编码并部署

1.在执行器项目中开发执行逻辑

- 继承 BaseJob 抽象类,默认实现了init、destory、exception方法
- 实现 BaseJob 的 execute 方法,任务处理的主要逻辑
- 异常处理(可选)重写 BaseJob 接口 exception 方法,自定义异常处理,返回true时执行全局异常处理逻辑,返回false不执行全局异常处理逻辑
- 实例化job(使用new或者ioc容器,推荐使用@vip.justlive.oxygen.core.ioc.Bean实例化)
- 注册到执行器,添加注解 @Job(value = "执行逻辑id", desc = "执行逻辑描述")

2.在调度中心,新建任务

参考上述任务配置属性说明,任务类型选择实例模式,执行器选择执行器配置中frost.job.executor.key的值,执行逻辑选择@Job注解定义的值 添加任务 添加cron任务 添加延时任务

开发脚本模式任务

1.在调度中心,新建脚本任务

参考实例模式的新建任务,任务模式选择脚本模式

注意:需要有执行器frost.job.executor.scriptJobEnabled=true开启支持脚本任务执行 添加脚本任务

2.在调度中心,开发任务代码

点击指定任务的脚本按钮,前往在线编辑任务界面,支持对任务代码进行开发(可在IDE中编辑完毕,复制粘贴到编辑中),支持使用静态工具类获取执行器中其他Bean

注意:分布式执行器要获取执行器中的Bean要确认是否所有执行器都存在该Bean,可配合指定执行器的分组来限定执行脚本任务的执行器 修改脚本

开发分片任务

1.在执行器项目中开发执行逻辑

分片任务通过将一个任务同时分发给所有执行器(或指定分片数量),协同进行任务处理,当进行大数据量操作时可显著提高处理速度。

@Bean
@Job(value = "shardingJob", desc = "分片job例子")
public class ShardingJob extends BaseJob {

  @Override
  public void execute(JobContext ctx) {
	// 获取分片信息
    JobSharding sharding = ctx.getSharding();
	// 分片序号
    int index = sharding.getIndex();
    // 分片总数
    int total = sharding.getTotal();
    
    ... 业务逻辑
  }
}

2.在调度中心,新建任务

参考上述实例任务配置任务,勾选分片运行,可使用默认执行器个数用于支持动态扩容或指定分片总数 添加任务

任务管理

1.执行器列表

点击执行器管理导航进入执行器列表,展示在线的执行器 执行器列表

列表字段说明

  • 执行器分组: 通过配置文件中的frost.job.executor.key进行分组,注意:分组相同且@Job的value相同的IJob逻辑应该完全相同
  • 名称:执行器应用的名称,便于区分和管理
  • 信息:展开分组时展示该执行器下注册的任务列表,合并分组时展示分组下在线执行器数量

2.任务管理

2.1 新建任务

参考开发实例模式或脚本模式任务章节

2.2 执行一次任务

点击操作栏“执行一次”按钮可触发执行一次任务 执行任务

2.3 暂停任务

点击操作栏“暂停”按钮可暂停正常任务,后续调度将不再执行 暂停任务

2.4 恢复任务

点击操作栏“恢复”按钮可恢复暂停任务 恢复任务

2.5 编辑任务

点击操作栏“编辑”按钮可编辑任务属性,参照任务配置属性进行修改

2.6 在线编辑脚本

点击操作栏“脚本”按钮跳转到在线编辑脚本页面,可进行脚本任务源码修改 脚本任务 修改脚本

2.7 跳转任务日志

点击操作栏“日志”按钮跳转到调度记录页面,只展示该任务的记录 跳转任务日志

2.8 删除任务

点击操作栏“删除”按钮删除任务 删除任务

3.使用client

3.1 依赖frost-client

<dependency>
    <groupId>vip.justlive</groupId>
    <artifactId>frost-client</artifactId>
</dependency>

3.2 实例化client

ClientProperties clientProps = new ClientProperties();
clientProps.setBaseUrl("http://localhost:20000/center");
clientProps.setUsername("frost");
clientProps.setPassword("frost");
JobApiFacade jobApiFacade = new JobApiFacadeImpl(clientProps);

3.3 调用JobApiFacade进行操作

联系信息

E-mail: qq11419041@163.com

Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: You must give any other recipients of the Work or Derivative Works a copy of this License; and You must cause any modified files to carry prominent notices stating that You changed the files; and You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "{}" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright 2018 justlive1 Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

简介

分布式任务调度,只依赖redis的实现版本 展开 收起
Java
Apache-2.0
取消

发行版

暂无发行版

贡献者

全部

近期动态

加载更多
不能加载更多了
Java
1
https://gitee.com/qfmx/earth-frost.git
git@gitee.com:qfmx/earth-frost.git
qfmx
earth-frost
earth-frost
master

搜索帮助

14c37bed 8189591 565d56ea 8189591