# mtl-rag

**Repository Path**: mutongli/mtl-rag

## Basic Information

- **Project Name**: mtl-rag
- **Description**: No description available
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2026-10-15
- **Last Updated**: 2026-10-29

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

<div align="center">

# mtl-rag

**端到端 RAG 系统：多格式文档解析 · 混合检索 · 流式生成**

[中文](#中文) · [English](#english)

</div>

---

## 中文

### 简介

`mtl-rag` 是一个可开箱即用的端到端检索增强生成（RAG）系统：把本地文档（PDF / DOCX / Markdown / TXT / HTML / CSV / XLSX）解析、分块、向量化并写入 Qdrant，再以「混合检索 + 重排」召回上下文，交由在线 LLM 流式作答，回答末尾自动附带与入模片段严格对应的参考资料。

系统同时提供三种对外形态：命令行（CLI）、HTTP API（FastAPI）与 Web UI（Streamlit），三者共用同一套核心管道。

### 特性

- **多格式解析**：PDF（含表格）/ DOCX / Markdown / TXT / HTML / CSV / XLSX；目录可递归批量导入，单文件失败不影响整体。
- **两类知识库**：
  - 非结构化库：支持五种分块策略（`fixed` / `recursive` / `semantic` / `parent_child` / `sentence_window`）。
  - 结构化库：字段级 `embed` / `show` / `filter` 控制、自定义 Jinja2 序列化模板、主键确定性去重；`filter` 字段自动建 payload 索引。
- **文档内嵌图片**：默认本地 OCR（RapidOCR）文字化入库，可切换在线多模态 VLM 或关闭，原图落盘供 UI 展示。
- **混合检索**：dense 向量 + BM25 sparse 双路召回 + RRF 跨变体融合；支持同父去重、阈值过滤与空结果兜底、可选 MMR 多样性。
- **检索增强（均可配置开关）**：LLM 查询改写、Multi-Query 多变体并行检索。
- **可插拔重排**：`noop`（关闭）/ `bge-local`（本地 fastembed ONNX，默认）/ `jina-api`（在线兼容 Jina 协议）。
- **流式生成**：LLM 逐段流式输出；空检索时直接返回固定拒答话术，不调用 LLM、不附参考资料。
- **幂等重入库**：uuid5 确定性 point id + 入库前按 `source` 删除旧点，重复导入同一文件不会产生重复数据。
- **三种对外形态**：CLI / HTTP API（含 SSE 流式端点 + Swagger 文档）/ Streamlit Web UI。

### 架构

```
文档 ──► 解析(document) ──► 分块(chunking) ──► Embedding + 入库(ingestion/embedding/vectorstore)
                                                          │
                                                          ▼
                     Qdrant ◄── 检索(retrieval：dense/hybrid + RRF + rerank + 改写)
                                                          │
                                                          ▼
                                        RAG 编排(rag) ──► LLM 流式生成
                                                          │
                                                          ▼
                                        CLI / HTTP API / Web UI
```

### 目录结构

```
mtl_rag/
├── core/          # 配置（pydantic-settings）与日志
├── document/      # 各格式解析器 + 内嵌图片处理（OCR / VLM）
├── chunking/      # 五种可插拔分块策略
├── embedding/     # OpenAI 兼容 Embedding + fastembed sparse 向量
├── vectorstore/   # Qdrant 双模式连接（server / local）与 collection 管理
├── ingestion/     # 非结构化 / 结构化入库管道 + 上传编排（API 与 Web 共用）
├── retrieval/     # 检索管道、RRF 融合、查询改写、可插拔 reranker
├── rag/           # RAG 编排引擎 + Prompt 模板（token 预算管理）
├── api/           # FastAPI 应用（kb / ingest / ask / health）
├── web/           # Streamlit Web UI（问答 / 知识库 / 导入 三页）
├── cli/           # click 命令入口
├── llm.py         # 同步 SSE 流式 LLM 客户端
├── registry.py    # 知识库注册表（SQLite WAL）
└── exceptions.py
prompts/           # default_rag.j2 提示模板
tests/             # 单元测试（本地嵌入式 Qdrant + Stub，不触网）
docs/              # 需求台账 + 任务文档
```

### 环境要求

- Python ≥ 3.11（开发验证于 3.12）
- 可选：Qdrant 服务（未部署时自动回退到本地嵌入式模式）
- 联网可用的 Embedding / LLM API（OpenAI 兼容协议）

### 安装

```bash
python -m venv .venv
# Windows
.venv\Scripts\activate
# macOS / Linux
source .venv/bin/activate

# 国内建议用清华镜像
pip install -e ".[dev]" -i https://pypi.tuna.tsinghua.edu.cn/simple
```

> **Windows 平台约束**：`pyproject.toml` 已将 `onnxruntime` 按平台锁定。Win32 限 `<1.20`——本机 `msvcp140.dll` 若为旧版（如 14.24），`onnxruntime>=1.20` 会在导入时崩溃；Linux/macOS 的 manylinux wheel 自带原生依赖，不设此限制（要求 glibc ≥ 2.28）。

### 配置

复制模板并填入真实值：

```bash
cp .env.example .env
```

所有字段均可被同名环境变量覆盖，字段名大小写不敏感。关键项：

| 分组 | 配置项 | 说明 |
|------|--------|------|
| Qdrant | `QDRANT_URL` / `QDRANT_API_KEY` | 服务模式地址与密钥 |
| Qdrant | `QDRANT_LOCAL_PATH` | 本地嵌入式模式存储路径（未部署服务时使用） |
| Embedding | `EMBEDDING_PROVIDER` / `EMBEDDING_API_KEY` / `EMBEDDING_MODEL` / `EMBEDDING_DIM` | 必填；`DIM` 必须与模型输出维度一致（如 `text-embedding-3-small=1536`） |
| LLM | `LLM_PROVIDER` / `LLM_API_KEY` / `LLM_MODEL` | 必填；`LLM_BASE_URL` 用于兼容 endpoint（Ollama 伪装 / 国产云），直连官方留空 |
| LLM | `LLM_MAX_CONTEXT_TOKENS` / `LLM_OUTPUT_RESERVE_TOKENS` / `LLM_TEMPERATURE` | token 预算与生成温度 |
| Rerank | `RERANK_ENABLED` / `RERANK_PROVIDER` | 默认开、`bge-local`；可切 `jina-api` |
| 分块 | `CHUNK_STRATEGY` / `CHUNK_SIZE_TOKENS` / `CHUNK_OVERLAP_TOKENS` | 策略与切分粒度 |
| 图片 | `IMAGE_PROCESSING` / `IMAGE_STORE_PATH` / `IMAGE_MAX_PER_DOC` | `ocr`（默认）/ `vlm` / `off` |
| 检索 | `RETRIEVAL_TOP_K` / `RETRIEVAL_TOP_N_AFTER_RERANK` / `SIMILARITY_THRESHOLD` / `USE_MMR` | 召回与入模片段数 |
| 检索增强 | `RETRIEVAL_HYBRID` / `RRF_K` / `QUERY_REWRITE_ENABLED` / `MULTI_QUERY_ENABLED` | 均为开关 |
| 应用 | `APP_HOST` / `APP_PORT` | `mtl-rag serve` 默认监听地址 |

### 使用

#### CLI

```bash
# 知识库管理
mtl-rag kb list                                # 列出全部知识库及 chunk 数
mtl-rag kb create docs --type unstructured     # 创建非结构化库
mtl-rag kb create items --type structured --schema schema.json  # 创建结构化库
mtl-rag kb info docs                           # 查看类型 / schema / chunk 数
mtl-rag kb delete docs --yes                   # 删除（含 Qdrant collection）

# 文档导入（文件或目录，目录默认递归）
mtl-rag ingest ./data --kb docs
mtl-rag ingest ./data --kb docs --no-recursive

# 流式问答
mtl-rag ask "这份报告的核心结论是什么？" --kb docs
mtl-rag ask "..." --kb docs --full             # 一次性输出完整回答
mtl-rag ask --kb docs --interactive            # 交互模式，循环提问
mtl-rag ask "..." --kb docs --top-k 20 --top-n 5

# 启动 HTTP API
mtl-rag serve --host 0.0.0.0 --port 8000
```

`mtl-rag kb create --schema` 支持两种书写形态：`{"primary": ..., "serialize_template": ..., "fields": {...}}`，或扁平结构（主键写在字段配置的 `primary: true`）。

#### HTTP API

启动后访问 `http://localhost:8000/docs` 查看 Swagger。

| 方法 | 路径 | 说明 |
|------|------|------|
| GET | `/health` | 健康检查（不带版本前缀） |
| GET | `/api/v1/knowledge-bases` | 列出知识库 |
| POST | `/api/v1/knowledge-bases` | 创建知识库（重名 → 409） |
| GET | `/api/v1/knowledge-bases/{kb_name}` | 知识库详情（不存在 → 404） |
| DELETE | `/api/v1/knowledge-bases/{kb_name}` | 删除知识库（204） |
| POST | `/api/v1/ingest` | 导入文档（multipart：`kb_name` + `files`） |
| POST | `/api/v1/ask` | 非流式问答，返回回答 + `sources` |
| POST | `/api/v1/ask/stream` | SSE 流式问答 |

SSE 线格式：逐段 `data: {"content": "..."}`，末尾 `data: {"done": true}`；中途出错则以 `data: {"error": "..."}` 事件承载。

```bash
# 创建知识库
curl -X POST http://localhost:8000/api/v1/knowledge-bases \
  -H "Content-Type: application/json" \
  -d '{"name": "docs", "type": "unstructured"}'

# 导入文档
curl -X POST http://localhost:8000/api/v1/ingest \
  -F "kb_name=docs" -F "files=@./data/report.pdf"

# 流式问答
curl -N -X POST http://localhost:8000/api/v1/ask/stream \
  -H "Content-Type: application/json" \
  -d '{"query": "核心结论是什么？", "knowledge_base": "docs"}'
```

#### Web UI

```bash
streamlit run mtl_rag/web/app.py
```

三个页面：**问答**（流式回答 + 参考资料展开）、**知识库**（创建 / 删除）、**导入**（多文件上传 + 结果表）。

### 测试

```bash
# Windows
.\.venv\Scripts\python.exe -m pytest tests -q
# macOS / Linux
python -m pytest tests -q
```

测试使用本地嵌入式 Qdrant 与确定性 Stub Embedding，并 stub 掉 LLM 调用，全程不触网；HTTP API 用 `TestClient`、Web 用 Streamlit `AppTest` 驱动。

### 跨平台部署注意

- **Docker / slim 镜像**：`rapidocr` 依赖的 `opencv-python` 需要系统库，需先安装 `libgl1` 与 `libglib2.0-0`。
- **模型缓存**：`fastembed` 首次运行需从 HuggingFace 下载 ONNX 模型，国内部署请配置 HF 镜像或提前预热缓存。
- **VC++ 运行库**：Windows 下 `onnxruntime` 版本受 `msvcp140.dll` 版本制约（见上文「Windows 平台约束」）。

### 文档索引

| 文档 | 说明 |
|------|------|
| [docs/requirements/需求.md](docs/requirements/需求.md) | 需求台账 |
| [docs/tasks/rag-system/00-总任务计划.md](docs/tasks/rag-system/00-总任务计划.md) | 任务总计划与依赖关系 |
| [docs/tasks/rag-system/](docs/tasks/rag-system/) | 各子任务方案与验收记录（task-01 … task-08） |
| [.env.example](.env.example) | 全量配置项模板 |

---

## English

### Overview

`mtl-rag` is an out-of-the-box end-to-end Retrieval-Augmented Generation system. It parses local documents (PDF / DOCX / Markdown / TXT / HTML / CSV / XLSX), chunks and embeds them into Qdrant, retrieves context via hybrid search with reranking, and streams the answer from an online LLM — appending a reference list that strictly maps to the chunks fed into the prompt.

Three interfaces share the same core pipeline: a CLI, an HTTP API (FastAPI) and a Web UI (Streamlit).

### Features

- **Multi-format parsing**: PDF (incl. tables) / DOCX / Markdown / TXT / HTML / CSV / XLSX; recursive directory import where a single failure does not abort the batch.
- **Two knowledge-base types**:
  - Unstructured: five pluggable chunking strategies (`fixed` / `recursive` / `semantic` / `parent_child` / `sentence_window`).
  - Structured: field-level `embed` / `show` / `filter` flags, custom Jinja2 serialization templates, deterministic primary-key deduplication; `filter` fields get payload indexes.
- **Embedded images**: local OCR (RapidOCR) by default, switchable to an online VLM or disabled; original images are persisted for UI display.
- **Hybrid retrieval**: dense vectors + BM25 sparse recall with RRF fusion; same-parent dedup, score thresholding with empty-result fallback, optional MMR diversity.
- **Retrieval enhancements (toggleable)**: LLM query rewriting and multi-query parallel retrieval.
- **Pluggable reranking**: `noop` / `bge-local` (local fastembed ONNX, default) / `jina-api`.
- **Streaming generation**: token-by-token output; on empty retrieval a fixed refusal is returned directly, without calling the LLM.
- **Idempotent re-ingestion**: deterministic uuid5 point ids plus delete-by-`source` before insert.
- **Three interfaces**: CLI / HTTP API (SSE streaming + Swagger docs) / Streamlit Web UI.

### Architecture

```
docs ──► parse(document) ──► chunk(chunking) ──► embed + ingest(ingestion/embedding/vectorstore)
                                                        │
                                                        ▼
                   Qdrant ◄── retrieve(retrieval: dense/hybrid + RRF + rerank + rewriting)
                                                        │
                                                        ▼
                                      RAG orchestration(rag) ──► streamed LLM answer
                                                        │
                                                        ▼
                                      CLI / HTTP API / Web UI
```

### Project layout

```
mtl_rag/
├── core/          # settings (pydantic-settings) and logging
├── document/      # per-format parsers + embedded image handling (OCR / VLM)
├── chunking/      # five pluggable chunking strategies
├── embedding/     # OpenAI-compatible embeddings + fastembed sparse vectors
├── vectorstore/   # Qdrant dual-mode connection (server / local) and collection management
├── ingestion/     # unstructured / structured ingestion pipelines + shared upload orchestration
├── retrieval/     # retrieval pipeline, RRF fusion, query rewriting, pluggable rerankers
├── rag/           # RAG engine + prompt templates (token-budget aware)
├── api/           # FastAPI app (kb / ingest / ask / health)
├── web/           # Streamlit UI (chat / knowledge base / import)
├── cli/           # click command entrypoint
├── llm.py         # synchronous SSE streaming LLM client
├── registry.py    # knowledge-base registry (SQLite WAL)
└── exceptions.py
prompts/           # default_rag.j2 prompt template
tests/             # unit tests (local embedded Qdrant + stubs, offline)
docs/              # requirement ledger + task documents
```

### Requirements

- Python ≥ 3.11 (verified on 3.12)
- Optional: a Qdrant server (falls back to embedded local mode when unavailable)
- Network access to Embedding / LLM APIs (OpenAI-compatible)

### Installation

```bash
python -m venv .venv
# Windows
.venv\Scripts\activate
# macOS / Linux
source .venv/bin/activate

pip install -e ".[dev]"
```

> **Windows constraint**: `pyproject.toml` pins `onnxruntime` per platform. On Win32 it is capped at `<1.20` — with an older `msvcp140.dll` (e.g. 14.24), `onnxruntime>=1.20` crashes on import. Linux/macOS manylinux wheels bundle their native dependencies and are not restricted this way (glibc ≥ 2.28 required).

### Configuration

```bash
cp .env.example .env
```

Every field can be overridden by an environment variable of the same (case-insensitive) name. Key options:

| Group | Key | Description |
|-------|-----|-------------|
| Qdrant | `QDRANT_URL` / `QDRANT_API_KEY` | Server-mode endpoint and key |
| Qdrant | `QDRANT_LOCAL_PATH` | Embedded local storage path (when no server is deployed) |
| Embedding | `EMBEDDING_PROVIDER` / `EMBEDDING_API_KEY` / `EMBEDDING_MODEL` / `EMBEDDING_DIM` | Required; `DIM` must match the model output (e.g. `text-embedding-3-small=1536`) |
| LLM | `LLM_PROVIDER` / `LLM_API_KEY` / `LLM_MODEL` | Required; `LLM_BASE_URL` for compatible endpoints, leave empty for official APIs |
| LLM | `LLM_MAX_CONTEXT_TOKENS` / `LLM_OUTPUT_RESERVE_TOKENS` / `LLM_TEMPERATURE` | Token budget and generation temperature |
| Rerank | `RERANK_ENABLED` / `RERANK_PROVIDER` | On by default, `bge-local`; can switch to `jina-api` |
| Chunking | `CHUNK_STRATEGY` / `CHUNK_SIZE_TOKENS` / `CHUNK_OVERLAP_TOKENS` | Strategy and granularity |
| Images | `IMAGE_PROCESSING` / `IMAGE_STORE_PATH` / `IMAGE_MAX_PER_DOC` | `ocr` (default) / `vlm` / `off` |
| Retrieval | `RETRIEVAL_TOP_K` / `RETRIEVAL_TOP_N_AFTER_RERANK` / `SIMILARITY_THRESHOLD` / `USE_MMR` | Recall and prompt-fragment counts |
| Enhancements | `RETRIEVAL_HYBRID` / `RRF_K` / `QUERY_REWRITE_ENABLED` / `MULTI_QUERY_ENABLED` | All toggles |
| App | `APP_HOST` / `APP_PORT` | `mtl-rag serve` bind address |

### Usage

#### CLI

```bash
# Knowledge bases
mtl-rag kb list
mtl-rag kb create docs --type unstructured
mtl-rag kb create items --type structured --schema schema.json
mtl-rag kb info docs
mtl-rag kb delete docs --yes

# Ingest (file or directory; directories recurse by default)
mtl-rag ingest ./data --kb docs
mtl-rag ingest ./data --kb docs --no-recursive

# Streaming Q&A
mtl-rag ask "What are the key takeaways?" --kb docs
mtl-rag ask "..." --kb docs --full
mtl-rag ask --kb docs --interactive
mtl-rag ask "..." --kb docs --top-k 20 --top-n 5

# HTTP API
mtl-rag serve --host 0.0.0.0 --port 8000
```

`--schema` accepts either `{"primary": ..., "serialize_template": ..., "fields": {...}}` or a flat object with `primary: true` on the key field.

#### HTTP API

Swagger is available at `http://localhost:8000/docs`.

| Method | Path | Description |
|--------|------|-------------|
| GET | `/health` | Health check (no version prefix) |
| GET | `/api/v1/knowledge-bases` | List knowledge bases |
| POST | `/api/v1/knowledge-bases` | Create (duplicate name → 409) |
| GET | `/api/v1/knowledge-bases/{kb_name}` | Details (missing → 404) |
| DELETE | `/api/v1/knowledge-bases/{kb_name}` | Delete (204) |
| POST | `/api/v1/ingest` | Ingest (multipart: `kb_name` + `files`) |
| POST | `/api/v1/ask` | Non-streaming answer with `sources` |
| POST | `/api/v1/ask/stream` | SSE streaming answer |

SSE wire format: `data: {"content": "..."}` per segment, terminated by `data: {"done": true}`; mid-stream errors are delivered as `data: {"error": "..."}`.

#### Web UI

```bash
streamlit run mtl_rag/web/app.py
```

Three pages: **Chat** (streamed answers with expandable references), **Knowledge base** (create / delete), **Import** (multi-file upload with a result table).

### Tests

```bash
python -m pytest tests -q
```

Tests run fully offline using an embedded local Qdrant, deterministic stub embeddings and a stubbed LLM; the HTTP API is driven via `TestClient` and the Web UI via Streamlit `AppTest`.

### Cross-platform deployment

- **Docker / slim images**: `opencv-python` (a `rapidocr` dependency) needs system libraries — install `libgl1` and `libglib2.0-0`.
- **Model cache**: `fastembed` downloads ONNX models from HuggingFace on first run; configure an HF mirror or pre-warm the cache.
- **VC++ runtime**: on Windows the `onnxruntime` version is constrained by `msvcp140.dll` (see the "Windows constraint" note above).

### Documentation

| Document | Description |
|----------|-------------|
| [docs/requirements/需求.md](docs/requirements/需求.md) | Requirement ledger |
| [docs/tasks/rag-system/00-总任务计划.md](docs/tasks/rag-system/00-总任务计划.md) | Task plan and dependencies |
| [docs/tasks/rag-system/](docs/tasks/rag-system/) | Per-task specs and acceptance records (task-01 … task-08) |
| [.env.example](.env.example) | Full configuration template |

---

## License

未指定 / Not specified.