# spider

**Repository Path**: sykconfig/spider

## Basic Information

- **Project Name**: spider
- **Description**: No description available
- **Primary Language**: Java
- **License**: Apache-2.0
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2026-05-08
- **Last Updated**: 2026-05-08

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# Spider Task Management System

A full-stack spider/scraper task management system built with Vue 3 + Spring Boot 3.

## Tech Stack

### Frontend
- Vue 3 (Composition API + `<script setup>`)
- Vite 5
- Naive UI (Component library)
- Pinia (State management)
- Vue Router 4 (History mode)
- Axios (HTTP client)
- ECharts 5 (Data visualization)
- @vicons/ionicons5 (Icons)
- SCSS (Styles)

### Backend
- Spring Boot 3 (JDK 17+)
- MyBatis-Plus (ORM)
- MySQL 8 (Database)
- Sa-Token (Authentication + RBAC)
- Knife4j (API documentation)
- Redis (Caching + Session storage)
- OkHttp3 (HTTP client for scraping)
- EasyExcel (Data export)
- Hutool (Utilities)
- Lombok

## Project Structure

```
D:\58\
├── spider-ui/           # Frontend (Vue 3 + Vite)
├── spider-server/       # Backend (Spring Boot 3)
├── README.md
├── spader.json          # BOSS直聘 scraper example (reference)
└── spader.py            # BOSS直聘 scraper example (reference)
```

## Quick Start

### 1. Database Setup

```sql
-- Execute the init SQL script
mysql -u root -p < spider-server/src/main/resources/db/init.sql
```

### 2. Backend (spider-server)

```bash
cd spider-server
mvn clean install -DskipTests
mvn spring-boot:run
```

Or run `SpiderApplication.java` directly in your IDE.

API docs: http://localhost:8080/doc.html
API base: http://localhost:8080/api

### 3. Frontend (spider-ui)

```bash
cd spider-ui
npm install
npm run dev
```

Frontend: http://localhost:5173

### Default Login
- Username: `admin`
- Password: `admin123`

## Features

### Spider Task Management
- Create/Edit/Delete spider tasks with configurable URL, headers, cookies, and parameters
- Support GET/POST/PUT/DELETE HTTP methods
- Parameter JSON editor with formatting
- Cookie and header management

### Scheduled Execution
- Cron-based scheduling using Spring ThreadPoolTaskScheduler
- Preset cron expressions (every 5min, 30min, hourly, daily, weekly)
- Manual trigger for immediate execution
- Async execution via dedicated thread pool

### Monitoring & Logging
- Real-time task execution status
- Detailed execution logs with full request/response capture
- Response body truncation at 500KB
- Error tracking and reporting

### Data Management
- Store scraped JSON results
- Search and filter scraped data
- Export to Excel (EasyExcel)

### Dashboard
- ECharts-powered dashboard
- Task statistics (total, enabled, running)
- Success rate gauge
- Execution trends (line chart)
- Task comparison (bar chart)
- Success/failure distribution (pie chart)

### System Management
- User CRUD with role assignment
- Role CRUD with menu permissions
- Dynamic menu tree management
- RBAC with Sa-Token

## Configuration

### Backend (application.yml)

Key configuration in `spider-server/src/main/resources/application.yml`:
- MySQL connection (default: root/saSA123@localhost:3306/spider_db)
- Redis connection (default: localhost:6379)
- Sa-Token token configuration
- Server port (default: 8080)

### Frontend (vite.config.js)

- Dev server: 0.0.0.0:5173 (LAN accessible)
- API proxy: /api -> http://localhost:8080

## API Documentation

### Backend API Docs
- Interactive: http://localhost:8080/doc.html (Knife4j/Swagger)
- Markdown: `spider-server/docs/api-docs.md`

### Frontend Docs
- `spider-ui/docs/frontend-docs.md`

## Default Menu Structure

- Dashboard
- System Management
  - User Management
  - Role Management
  - Menu Management
- Spider Management
  - Task Management
  - Execution Logs
  - Data Management