# mcp-crawl4ai

**Repository Path**: marklang/mcp-crawl4ai

## Basic Information

- **Project Name**: mcp-crawl4ai
- **Description**: No description available
- **Primary Language**: Unknown
- **License**: MIT
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2026-05-11
- **Last Updated**: 2026-05-11

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

<div align="center">

# MCP-Crawl4AI

<img src=".github/assets/img/logo.png" alt="MCP-Crawl4AI Logo" width="200"/>

*A Model Context Protocol server for web crawling powered by Crawl4AI*

<!-- BADGES:START -->
<!-- generated by add-badges 2026-03-02 -->
[![CI](https://img.shields.io/github/actions/workflow/status/wyattowalsh/mcp-crawl4ai/ci.yml?style=for-the-badge&logo=githubactions&logoColor=white&label=CI)](https://github.com/wyattowalsh/mcp-crawl4ai/actions/workflows/ci.yml)
[![Codecov](https://img.shields.io/codecov/c/github/wyattowalsh/mcp-crawl4ai?style=for-the-badge&logo=codecov&logoColor=white)](https://codecov.io/gh/wyattowalsh/mcp-crawl4ai)
[![PyPI](https://img.shields.io/pypi/v/mcp-crawl4ai?style=for-the-badge&logo=pypi&logoColor=white)](https://pypi.org/project/mcp-crawl4ai/)

[![Ruff](https://img.shields.io/badge/linter-Ruff-D7FF64?style=for-the-badge&logo=ruff&logoColor=black)](https://docs.astral.sh/ruff/)
[![Pre-commit](https://img.shields.io/badge/pre--commit-FAB040?style=for-the-badge&logo=precommit&logoColor=black)](https://pre-commit.com/)
[![Dependabot](https://img.shields.io/badge/Dependabot-025E8C?style=for-the-badge&logo=dependabot&logoColor=white)](https://github.com/wyattowalsh/mcp-crawl4ai/security/dependabot)

[![Python 3.13+](https://img.shields.io/badge/python-3.13+-3776AB?style=for-the-badge&logo=python&logoColor=white)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/github/license/wyattowalsh/mcp-crawl4ai?style=for-the-badge&logo=github&logoColor=white)](https://github.com/wyattowalsh/mcp-crawl4ai/blob/main/LICENSE)

[![MCP Compatible](https://img.shields.io/badge/MCP-compatible-green?style=for-the-badge)](https://modelcontextprotocol.ai)
[![FastMCP v3](https://img.shields.io/badge/FastMCP-v3-4B8BBE?style=for-the-badge&logo=python&logoColor=white)](https://github.com/jlowin/fastmcp)
[![Crawl4AI](https://img.shields.io/badge/Crawl4AI-powered-FF6B35?style=for-the-badge)](https://github.com/unclecode/crawl4ai)
[![Docker](https://img.shields.io/badge/Docker-2496ED?style=for-the-badge&logo=docker&logoColor=white)](https://www.docker.com/)
[![Documentation](https://img.shields.io/badge/docs-online-brightgreen?style=for-the-badge)](https://mcp-crawl4ai.w4w.dev)

[![GitHub stars](https://img.shields.io/github/stars/wyattowalsh/mcp-crawl4ai?style=for-the-badge&logo=github&logoColor=white)](https://github.com/wyattowalsh/mcp-crawl4ai/stargazers)
[![GitHub last commit](https://img.shields.io/github/last-commit/wyattowalsh/mcp-crawl4ai?style=for-the-badge&logo=github&logoColor=white)](https://github.com/wyattowalsh/mcp-crawl4ai/commits)
<!-- BADGES:END -->

</div>

---

## Overview

**MCP-Crawl4AI** is a [Model Context Protocol](https://modelcontextprotocol.io) server that gives AI systems access to the live web. Built on [FastMCP v3](https://github.com/jlowin/fastmcp) and [Crawl4AI](https://github.com/unclecode/crawl4ai), it exposes **4 tools**, **2 resources**, and **3 prompts** through the standardized MCP interface, backed by a lifespan-managed headless Chromium browser.

**Only 2 runtime dependencies** &mdash; `fastmcp` and `crawl4ai`.

> [!TIP]
> **[Full documentation site &rarr;](https://mcp-crawl4ai.w4w.dev)**

### Key Features

- **Full MCP compliance** via FastMCP v3 with tool annotations (`readOnlyHint`, `destructiveHint`, etc.)
- **4 focused tools** centered on canonical scrape/crawl plus session lifecycle/artifacts
- **3 prompts** for common LLM workflows (summarize, extract schema, compare pages)
- **2 resources** exposing server configuration and version info
- **Headless Chromium** managed as a lifespan singleton (start once, reuse everywhere)
- **Multiple transports** &mdash; stdio (default) and Streamable HTTP
- **LLM-optimized output** &mdash; markdown, cleaned HTML, raw HTML, or plain text
- **Canonical option groups** for extraction, runtime, diagnostics, sessions, rendering, and traversal
- **List and deep traversal** in one `crawl` contract
- **Session-aware workflows** with explicit session close and artifact retrieval tools
- **Auto browser setup** &mdash; detects missing Playwright browsers and installs automatically

---

## Installation

<details>
<summary><strong>pip</strong></summary>

```bash
pip install mcp-crawl4ai
mcp-crawl4ai --setup       # one-time: installs Playwright browsers
```

</details>

<details>
<summary><strong>uv (recommended)</strong></summary>

```bash
uv add mcp-crawl4ai
mcp-crawl4ai --setup       # one-time: installs Playwright browsers
```

</details>

<details>
<summary><strong>Docker</strong></summary>

```bash
docker build -t mcp-crawl4ai .
docker run -p 8000:8000 mcp-crawl4ai
```

The Docker image includes Playwright browsers &mdash; no separate setup needed.

</details>

<details>
<summary><strong>Development</strong></summary>

```bash
git clone https://github.com/wyattowalsh/mcp-crawl4ai.git
cd mcp-crawl4ai
uv sync --group dev
mcp-crawl4ai --setup
```

</details>

> [!NOTE]
> The server auto-detects missing Playwright browsers on first startup and attempts to install them automatically. You can also run `mcp-crawl4ai --setup` or `crawl4ai-setup` manually at any time.

---

## Quick Start

<details open>
<summary><strong>stdio (default &mdash; for Claude Desktop, Cursor, etc.)</strong></summary>

```bash
mcp-crawl4ai
```

</details>

<details>
<summary><strong>HTTP transport</strong></summary>

```bash
mcp-crawl4ai --transport http --port 8000
```

> [!NOTE]
> HTTP binds to `127.0.0.1` by default (private/local only); for external exposure, set `--host` explicitly and use a reverse proxy for TLS/auth.

</details>

<details>
<summary><strong>Claude Desktop configuration</strong></summary>

Add to your Claude Desktop MCP settings (`claude_desktop_config.json`):

```json
{
  "mcpServers": {
    "crawl4ai": {
      "command": "mcp-crawl4ai",
      "args": ["--transport", "stdio"]
    }
  }
}
```

</details>

<details>
<summary><strong>Claude Code configuration</strong></summary>

```bash
claude mcp add crawl4ai -- mcp-crawl4ai --transport stdio
```

</details>

<details>
<summary><strong>MCP Inspector</strong></summary>

```bash
npx @modelcontextprotocol/inspector uv --directory . run mcp-crawl4ai
```

</details>

---

## Tools

The canonical surface now exposes **4 tools**:

### `scrape`

Scrape one URL or a bounded list of URLs (up to 20) with a single canonical envelope response.

- Input: `targets` (`str | list[str]`) and optional grouped `options`
- Supports extraction (`schema`, `extraction_mode`), runtime controls, diagnostics, session settings, render settings, and artifact capture
- Returns canonical JSON envelope with `schema_version`, `tool`, `ok`, `data/items`, `meta`, `warnings`, `error`

### `crawl`

Crawl with canonical traversal controls.

- `options.traversal.mode="list"` for bounded list traversal
- `options.traversal.mode="deep"` for recursive BFS/DFS traversal from a single seed
- Shares scrape option groups plus traversal options in the same envelope shape

### `close_session`

Close a stateful session created via `options.session.session_id`.

### `get_artifact`

Retrieve artifact metadata/content captured during `scrape` or `crawl` when `options.conversion.capture_artifacts` is enabled.

### Choosing core vs advanced usage

- **Core path (recommended):** use `scrape`/`crawl` with minimal options (`runtime`, `conversion.output_format`, `traversal.mode="list"`). This keeps behavior predictable and low-risk for most agent workflows.
- **Advanced path (explicit opt-in):** use deep traversal, custom dispatcher controls, JS transforms, extraction schemas, and artifact capture only when required by task outcomes.
- **Safety budgets and gates:** inspect `config://server` for `settings.defaults`, `settings.limits`, `settings.policies`, and `settings.capabilities` to understand active constraints and feature gates before enabling advanced options.

---

## Resources

| URI | MIME Type | Description |
|-----|-----------|-------------|
| `config://server` | `application/json` | Current server configuration: name, version, tool list, browser config |
| `crawl4ai://version` | `application/json` | Server and dependency version information (server, crawl4ai, fastmcp) |

## Prompts

| Prompt | Parameters | Description |
|--------|------------|-------------|
| `summarize_page` | `url`, `focus` (default: `"key points"`) | Crawl a page and summarize its content with the specified focus |
| `build_extraction_schema` | `url`, `data_type` | Inspect a page and build a CSS extraction schema for `scrape` |
| `compare_pages` | `url1`, `url2` | Crawl two pages and produce a structured comparison |

---

## Architecture

```mermaid
graph TD
    A[MCP Client] -->|stdio / HTTP| B[FastMCP v3]
    B --> C[Tool Router]
    C --> D[scrape]
    C --> E[crawl]
    C --> F[close_session]
    C --> G[get_artifact]
    D & E & F & G --> N[AsyncWebCrawler Singleton]
    N --> O[Headless Chromium]
    B --> P[Resources]
    B --> Q[Prompts]

    style B fill:#4B8BBE,color:#fff
    style N fill:#FF6B35,color:#fff
    style O fill:#2496ED,color:#fff
```

The server uses a single-module architecture:

- **FastMCP v3** handles MCP protocol negotiation, transport, tool/resource/prompt registration, and message routing
- **Lifespan-managed `AsyncWebCrawler`** starts a headless Chromium browser once at server startup and shares it across all tool invocations, then shuts it down cleanly on exit
- **4 tool functions** decorated with `@mcp.tool()` define the canonical surface
- **2 resource functions** decorated with `@mcp.resource()` return JSON
- **3 prompt functions** decorated with `@mcp.prompt` return structured `Message` lists

> [!IMPORTANT]
> There are no intermediate manager classes or custom HTTP clients. The server delegates all crawling to crawl4ai's `AsyncWebCrawler` and all protocol handling to FastMCP. Only **2 runtime dependencies**.

---

## Configuration

<details>
<summary><strong>CLI flags</strong></summary>

| Flag | Default | Description |
|------|---------|-------------|
| `--transport` | `stdio` | Transport protocol: `stdio` or `http` |
| `--host` | `127.0.0.1` | Host to bind (HTTP transport only) |
| `--port` | `8000` | Port to bind (HTTP transport only) |
| `--setup` | &mdash; | Install Playwright browsers and exit |

</details>

<details>
<summary><strong>Environment variables</strong></summary>

No environment variables are required. The server uses sensible defaults for all configuration. Crawl4AI's own environment variables (e.g., `CRAWL4AI_VERBOSE`) are respected if set.

</details>

---

## Testing

```bash
# Run all tests
uv run pytest

# Coverage gate (>=90%)
uv run pytest --cov=mcp_crawl4ai --cov-report=term-missing

# Smoke tests only
uv run pytest -m smoke

# Unit tests only
uv run pytest -m unit

# Integration workflow tests
uv run pytest -m integration

# End-to-end workflow tests
uv run pytest -m e2e

# Manual live test (requires browser)
uv run python tests/manual/test_live.py
```

> [!NOTE]
> All automated tests run in-memory using `fastmcp.Client(mcp)` &mdash; no browser or network required. The test suite mocks `AsyncWebCrawler` for fast, deterministic execution.

---

## Contributing

See the [Contributing Guide](.github/CONTRIBUTING.md) for details on setting up your development environment, coding standards, and the pull request process.

## License

This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.

---

<div align="center">

**MCP-Crawl4AI** &mdash; *Connecting AI to the Live Web*

</div>