# spider

**Repository Path**: yyh915/spider

## Basic Information

- **Project Name**: spider
- **Description**: sysnc spider github
- **Primary Language**: Rust
- **License**: MIT
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2024-03-11
- **Last Updated**: 2024-03-11

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# Spider

[![Build Status](https://github.com/madeindjs/spider/actions/workflows/rust.yml/badge.svg)](https://github.com/madeindjs/spider/actions)
[![Crates.io](https://img.shields.io/crates/v/spider.svg)](https://crates.io/crates/spider)
[![Documentation](https://docs.rs/spider/badge.svg)](https://docs.rs/spider)
[![Rust](https://img.shields.io/badge/rust-1.56.1%2B-blue.svg?maxAge=3600)](https://github.com/madeindjs/spider)

The fastest web crawler and indexer. Foundational building blocks for data curation workloads.

- Concurrent
- Streaming
- Decentralization
- Headless Chrome Rendering
- HTTP Proxies
- Cron Jobs
- Subscriptions
- Smart Mode
- Blacklisting and Budgeting Depth
- [Changelog](CHANGELOG.md)

## Getting Started

The simplest way to get started is to use the [Spider Cloud](https://spiderwebai.xyz) hosted service, use the coupon code `SPIDEY24` to get $5 initial credits at checkout. View the [spider](./spider/README.md) or [spider_cli](./spider_cli/README.md) directory for local installations. You can also use spider with Node.js using [spider-nodejs](https://github.com/spider-rs/spider-nodejs) and Python using [spider-py](https://github.com/spider-rs/spider-py).

## Benchmarks

See [BENCHMARKS](./benches/BENCHMARKS.md).

## Examples

See [EXAMPLES](./examples/).

## License

This project is licensed under the [MIT license].

[MIT license]: https://github.com/spider-rs/spider/blob/master/LICENSE

## Contributing

See [CONTRIBUTING](CONTRIBUTING.md).