# crawler-tool

**Repository Path**: hippoDocker/crawler-tool

## Basic Information

- **Project Name**: crawler-tool
- **Description**: 爬虫链路分析工具
- **Primary Language**: Python
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 1
- **Created**: 2026-05-24
- **Last Updated**: 2026-07-13

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# BrowserFlow - Browser Traffic Analyzer

A traffic analysis tool for enterprise intranet systems, designed to trace system redirect chains, extract authentication parameters, analyze encryption logic, and assist in writing web scrapers.

## Features

- **Request Interception**: Capture all HTTP/HTTPS requests via Playwright Chromium engine, bypassing system proxy
- **Smart Filtering**: 7 preset filter modes (Capture All / Exclude Static / API Only / Auth Only, etc.), with custom wildcard support
- **Authentication Analysis**: Auto-extract Bearer Token, Cookie, Header, CAS/OAuth2/SAML authentication parameters
- **Encryption Detection**: Auto-detect encrypted fields in requests/responses (AES/RSA/MD5/Base64/SM2/SM4, etc.), analyze entropy and encoding patterns
- **Chain Tracing**: Visualize redirect chains across multiple systems (e.g., HR -> OA -> Procurement)
- **Request Details**: Double-click a request to view 7 dimensions of information (Basic Info / Request Headers / Request Body / Response Headers / Response Body / Auth Analysis / JS Source)
- **Code Generation**: One-click copy as cURL command or Python requests code
- **Parameter Diff**: Compare multiple requests to the same URL, auto-detect changing parameters (pagination/timestamps/signatures)
- **Pause Recording**: Pause/resume capture anytime, new requests won't disturb analysis
- **AI Analysis**: Selectively check requests for AI-powered analysis of authentication parameters and scraper recommendations
- **Export**: JSON / HAR 1.2 format export

## Requirements

- Python 3.10+ (3.11+ recommended)
- Windows 10/11

## Installation

### Option 1: Download EXE (Recommended)

Download `BrowserFlow.exe` from the Releases page and double-click to run. On first launch, it will automatically download the Chromium browser (~180MB) with progress display.

### Option 2: One-Click Scripts (Source Code)

After downloading the project, run the corresponding script to automatically install and launch:

| Script | Platform | Function |
|--------|----------|----------|
| `setup.py` | All platforms | Double-click to run: Install deps -> Install Chromium -> Launch app |
| `setup.sh` | macOS / Linux | Terminal: Same as above |
| `build.py` | All platforms | Package as EXE (`python build.py`) |
| `build.sh` | macOS / Linux | Terminal: Same as above |

**Windows users**: Double-click `setup.py` (or `python setup.py`) to launch, `python build.py` to package EXE

**macOS / Linux users**:

```bash
# First run (install + launch)
python3 setup.py
# or
chmod +x setup.sh && ./setup.sh

# Package
python3 build.py
# or
chmod +x build.sh && ./build.sh
```

### Option 3: Manual Installation

```bash
# 1. Install dependencies
pip install -r requirements.txt

# 2. Install Chromium browser
playwright install chromium

# 3. Launch the app
python main.py
```

## Packet Capture Tutorial

### Step 1: Launch Browser

1. Double-click `BrowserFlow.exe` or run `python main.py`
2. Click the **"Launch Browser"** button at the bottom left
3. Chromium will open as a separate window (first time requires download, please wait)
4. Status bar shows "Browser launched" when connected

### Step 2: Set Filter Rules

Select a filter preset from the toolbar dropdown:

| Filter Mode | Description | Use Case |
|-------------|-------------|----------|
| Capture All | Capture all requests including JS/CSS/images | Analyzing frontend encryption logic |
| Exclude Static | Exclude .css .png .jpg etc., keep JS | **Recommended for daily use** |
| API Only | Only capture URLs containing "api" | Backend API focus |
| HTML Only | Only capture .html / .htm requests | Analyzing page redirects |
| Auth Only | Only capture requests with auth/login/sso/token keywords | Login flow analysis |
| Custom | Enter wildcard expression, e.g. `*login*` | Special needs |

### Step 3: Visit Target Website

1. Enter the target URL in Chromium's address bar
2. Operate the website normally (login, click menus, paginate, etc.)
3. The request list on the right will display captured requests in real-time

### Step 4: Analyze Requests

**Request List:**
- Each request shows: `Method | Host | Path`
- Colors: Green = GET, Orange = POST, Purple = OPTIONS/CORS

**Request Details:**
- Double-click any request to open the detail window with 7 tabs:
  - **Basic Info**: Method, URL, duration, status code
  - **Request Headers**: Full request header information
  - **Request Body**: POST data (auto-formatted JSON)
  - **Response Headers**: Response header information
  - **Response Body**: Response content (auto-formatted JSON)
  - **Auth Analysis**: Auto-extracted Token, Cookie, SSO info, encryption field detection
  - **JS Source**: Displayed for .js requests, highlights encryption keywords (encrypt/AES/RSA etc.)

**Quick Actions:**
- Click **"Copy as cURL"** at the bottom of the detail window
- Click **"Copy as Python"** to get Python requests code

### Step 5: Authentication Analysis

Switch to the **"Auth"** tab on the right:
- Auto-identify requests containing authentication information
- Display Token type (Bearer/Cookie/URL param) and value
- Double-click to view full auth details (Token, Cookie, Auth Headers, SSO info)

### Step 6: Chain Tracing

Switch to the **"Chain"** tab:
- View redirect relationships between multiple systems
- Detect 301/302/303/307/308 redirect chains
- WebSocket message recording

### Step 7: Parameter Diff (Analyze Encryption Parameter Changes)

1. Click the **"Compare Requests"** button in the toolbar
2. System automatically finds request groups with duplicate URLs
3. Select a group to compare
4. View which parameters are changing (pagination, timestamps, signatures, etc.)
5. Change patterns auto-detected: incrementing numbers, timestamps, etc.

### Step 8: AI-Powered Analysis

1. Click **"AI Config"** to fill in API Key, Base URL, Model
2. Switch to the **"AI Analysis"** tab
3. Options:
   - **Select & Analyze**: Check specific requests to send to AI
   - **Analyze All**: Send all captured requests to AI
   - **Auth Only**: Quickly filter requests with authentication info
4. AI will analyze and output:
   - Redirect chain identification
   - Authentication parameter extraction
   - SSO flow analysis
   - Encryption/decryption logic analysis
   - Scraper code recommendations (Python requests)

### Step 9: Export Logs

Click the **"Export Logs"** button in the toolbar:
- Export as JSON format containing all captured request data
- Can be used for further analysis or imported into other tools

## Common Packet Capture Scenarios

### Scenario 1: Analyzing Enterprise SSO Login

```
1. Filter: "Exclude Static"
2. Visit SSO login page
3. Enter credentials and log in
4. Observe auth/login/sso/token related requests
5. View extracted Token in the "Auth" tab
6. Use AI analysis for complete SSO flow and scraper recommendations
```

### Scenario 2: Analyzing Frontend Encryption Parameters

```
1. Filter: "Capture All" (need to capture JS files)
2. Operate the target website to trigger encrypted requests
3. Double-click encrypted request, check "Auth Analysis" tab for encryption detection
4. Double-click related .js files, view highlighted encryption code in "JS Source" tab
5. Use "Compare Requests" to analyze parameter change patterns
6. Use AI analysis for encryption logic and scraper code
```

### Scenario 3: Tracing Multi-System Redirect Chains

```
1. Filter: "HTML Only" or "Auth Only"
2. Start from the entry system (e.g., HR system)
3. Navigate through other systems (OA, Procurement, etc.)
4. View redirect relationships in the "Chain" tab
5. Use AI analysis for complete redirect chain and parameter passing
```

## Toolbar Reference

| Button | Function |
|--------|----------|
| Recording / Resume | Pause/resume request capture (paused requests are restored on resume) |
| Clear | Clear all captured requests |
| Filter Dropdown | Select preset filter mode |
| Apply | Apply custom filter expression |
| Compare Requests | Compare multiple requests to the same URL, analyze parameter changes |
| Export Logs | Export as JSON format |
| AI Config | Configure AI interface (API Key / Base URL / Model) |

## Browser Controls

| Button | Function |
|--------|----------|
| Back / Forward | Navigate back/forward |
| Refresh | Reload page |
| Home | Go to blank page |
| Address bar + Go | Manual URL navigation |

## AI Analysis Configuration

Click the **"AI Config"** button in the toolbar:

| Field | Description | Example |
|-------|-------------|---------|
| API Key | OpenAI-compatible API key | `sk-xxxx...` |
| Base URL | API endpoint (without /v1) | `https://api.openai.com/v1` |
| Model | Model name to use | `gpt-4` / `deepseek-chat` |

Configuration is auto-saved to `~/.browserflow/ai_config.json` and loaded on next launch.

Supports all OpenAI-compatible interfaces:
- OpenAI (GPT-4 / GPT-3.5-turbo)
- Deepseek (deepseek-chat / deepseek-coder)
- Qwen (qwen-turbo / qwen-plus)
- Ernie Bot (ernie-bot)
- Any other OpenAI API compatible service

Click **"Fetch Model List"** to auto-fetch available models.

## Log Files

Application logs are saved at: `~/.browserflow/logs/browserflow.log`

- Auto-rotation: max 5MB per file, 3 backups kept
- Error logs separately saved: `~/.browserflow/logs/browserflow_error.log`
- Click the log path in the status bar to copy it

## Tech Stack

- Python 3.10+
- Playwright - Browser automation and request interception
- tkinter - GUI framework (built into Python)
- requests - AI API calls

## Project Structure

```
crawler-tool/
├── main.py                # Entry point (auto-selects PyQt6/tkinter)
├── main_tkinter.py        # tkinter main UI (multi-threaded architecture)
├── setup.py               # One-click install & launch (double-click to run)
├── setup.sh               # One-click install & launch (macOS/Linux)
├── build.py               # PyInstaller packaging script
├── build.sh               # One-click package (macOS/Linux)
├── requirements.txt       # Dependencies
├── core/                  # Core modules
│   ├── browser.py         # Playwright browser management (Chromium install detection)
│   ├── browser_worker.py  # Browser worker thread (independent lifecycle management)
│   ├── interceptor.py     # Request interception & filtering (route-based, handles chunked encoding)
│   ├── analyzer.py        # Auth analysis + encryption detection + parameter diff
│   ├── link_tracker.py    # Chain tracing
│   ├── ai_client.py       # AI client (OpenAI-compatible, config persistence)
│   ├── logger.py          # Unified logging (console + file rotation)
│   └── message_queue.py   # Thread-safe message queue
├── utils/
│   └── export.py          # Log export (JSON / HAR 1.2 / cURL / Python)
├── tools/
│   └── crawler_analyzer.py # Crawler analyzer tool
├── docs/
│   └── tutorial.html      # Tutorial
└── data/
    └── logs/              # Export log directory
```

## License

MIT License