Metrics
0
Watch 26 Star 62 Fork 11

henrylee2cn / surferGoApache-2.0

Sign up for free
Explore and code with more than 2 million developers,Free private repositories !:)
Sign up
surfer [high level concurrency downloader] 是一款Go语言编写的高并发爬虫下载器,拥有surf与phantom两种下载内核,支持固定UserAgent自动保存cookie与随机大量UserAgent禁用cookie两种模式,高度模拟浏览器行为,可实现模拟登录等功能。 spread retract

Clone or download
Cancel
Notice: Creating folder will generate an empty file .keep, because not support in Git
Loading...
README.md

Surfer GoDoc GitHub release

Package surfer is a high level concurrency http client. It has surf and phantom download engines, highly simulated browser behavior, the function of analog login and so on.

简体中文

Features

  • Both surf and phantomjs engines are supported
  • Support random User-Agent
  • Support cache cookie
  • Support http/https

Usage

package main

import (
    "github.com/henrylee2cn/surfer"
    "io/ioutil"
    "log"
)

func main() {
    // Use surf engine
    resp, err := surfer.Download(&surfer.Request{
        Url: "http://github.com/henrylee2cn/surfer",
    })
    if err != nil {
        log.Fatal(err)
    }
    b, err := ioutil.ReadAll(resp.Body)
    log.Println(string(b), err)

    // Use phantomjs engine
    resp, err = surfer.Download(&surfer.Request{
        Url:          "http://github.com/henrylee2cn",
        DownloaderID: 1,
    })
    if err != nil {
        log.Fatal(err)
    }
    b, err = ioutil.ReadAll(resp.Body)
    log.Println(string(b), err)

    resp.Body.Close()
    surfer.DestroyJsFiles()
}

Full example

License

Surfer is under Apache v2 License. See the LICENSE file for the full license text.

Comments ( 1 )

You need to Sign in for post a comment

Help Search