# garlicts-crawler

**Repository Path**: garlicts/garlicts-crawler

## Basic Information

- **Project Name**: garlicts-crawler
- **Description**: 分布式爬虫框架
- **Primary Language**: Java
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2018-10-08
- **Last Updated**: 2020-12-19

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# garlicts-crawler

#### 项目介绍
分布式爬虫框架，重构中

#### 软件架构
软件架构说明


#### 安装教程

新建工程garlicts-crawler-executor-demo

1. xxxx

```
package com.garlicts.framework.crawler.executor.demo;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;

import com.garlicts.framework.crawler.common.IJobHandler;
import com.garlicts.framework.crawler.common.dto.Result;
import com.garlicts.framework.crawler.executor.ExecutorTask;
import com.garlicts.framework.crawler.executor.crawler.CrawlerWork;

@ExecutorTask(name = "firstCrawlerTask")
public class FirstCrawlerTask implements IJobHandler {
	
	private static final Logger LOGGER = LoggerFactory.getLogger(FirstCrawlerTask.class);

	@Autowired
	CrawlerWork crawlerWork;
	
	public Result execute(String param) throws Exception {
		LOGGER.info("execute firstTask");
		
		return SUCCESS;
	}

}

```

2. xxxx

```
package com.garlicts.framework.crawler.executor.demo;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.ComponentScan;

@SpringBootApplication
@ComponentScan(basePackages = {"com.garlicts"})
@EnableAutoConfiguration
public class GarlictsCrawlerExecutorDemoApplication {

	public static void main(String[] args) {
		SpringApplication.run(GarlictsCrawlerExecutorDemoApplication.class, args);
	}
}

```

3. application.properties

```
executorName=firstExecutor
serviceAddress=127.0.0.1:8000
zkAddress=127.0.0.1:2181
```

新建工程garlicts-crawler-scheduler-demo

1.xxx

```
package com.garlicts.framework.crawler.executor.demo;

import org.quartz.JobDetail;
import org.quartz.Trigger;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import com.garlicts.framework.crawler.scheduler.SchedulerUtil;

@Configuration
public class QuartzConfig {
	
	@Autowired
	SchedulerUtil schedulerUtil;
	
	@Bean
	public JobDetail createJobDetail() {
		
		String jobName = "firstCrawlerTask"; 
		String jobContent = "jsonStr";
		String executorName = "firstExecutor";
		return schedulerUtil.createJobDetail(jobName, jobContent, executorName);
		
	}
	
	@Bean
	public Trigger createJobTrigger() {
		
		String jobName = "firstCrawlerTask"; 
		String jobContent = "jsonStr";
		String executorName = "firstExecutor";		
		String cron = "*/5 * * * * ?";
		return schedulerUtil.createJobTrigger(jobName, jobContent, executorName, cron);
		
	}
	
}

```

2.xxx

```
package com.garlicts.framework.crawler.executor.demo;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.ComponentScan;

@SpringBootApplication
@ComponentScan(basePackages = {"com.garlicts"})
@EnableAutoConfiguration
public class GarlictsCrawlerSchedulerDemoApplication {

	public static void main(String[] args) {
		SpringApplication.run(GarlictsCrawlerSchedulerDemoApplication.class, args);
	}
}

```

3.xxxx

```
executorName=firstExecutor
serviceAddress=127.0.0.1:8000
zkAddress=127.0.0.1:2181
```


#### 使用说明

1. xxxx
2. xxxx
3. xxxx

#### 参与贡献

1. Fork 本项目
2. 新建 Feat_xxx 分支
3. 提交代码
4. 新建 Pull Request


#### 码云特技

1. 使用 Readme\_XXX.md 来支持不同的语言，例如 Readme\_en.md, Readme\_zh.md
2. 码云官方博客 [blog.gitee.com](https://blog.gitee.com)
3. 你可以 [https://gitee.com/explore](https://gitee.com/explore) 这个地址来了解码云上的优秀开源项目
4. [GVP](https://gitee.com/gvp) 全称是码云最有价值开源项目，是码云综合评定出的优秀开源项目
5. 码云官方提供的使用手册 [https://gitee.com/help](https://gitee.com/help)
6. 码云封面人物是一档用来展示码云会员风采的栏目 [https://gitee.com/gitee-stars/](https://gitee.com/gitee-stars/)