# simple-spider

**Repository Path**: gogant/simple-spider

## Basic Information

- **Project Name**: simple-spider
- **Description**: 简单网络爬虫
- **Primary Language**: Java
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2014-11-24
- **Last Updated**: 2020-12-18

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

##网络爬虫

####使用例子

	// 创建爬虫，线程池大小为：5
    HttpSpider spider = new HttpSpider(5);
    // 设置URL选择规则
    spider.enableURLPicker()
            .when("com/$", "/home/\\d+$").find("/home/\\d+$", "/mm/\\d+$") // home
            .when("/mm/\\d+$").find("/mm/\\d+/\\d+$", "com/\\d+/\\d+/\\d+.jpg")//mm-page
            .when("/mm/\\d+/\\d+$").find("/mm/\\d+/\\d+$", "com/\\d+/\\d+/\\d+.jpg");// jpgs
    // 对爬取图片进行处理
    spider.addURLHandler(new HttpURLHandler() {
        File base = new File("e:/images");
        // 过滤图片的URL
        public boolean onFilter(Context context) {
            return context.like("%.jpg");
        }
        // 保存图片
        public void onResponse(HttpResponse resp) {
            System.out.println("fetch image: " + resp.getURL());
            resp.saveToPath(base);
        }
        // 打印异常信息
        public void onException(Context context, Throwable e) {
            System.err.println(e.getMessage());
        }
    });
    // 爬取网站下所有的图片
    spider.fetch("http://www.mmjpg.com/", 100, 1, TimeUnit.DAYS);