diff --git a/kylinbot-auto-test/README.md b/README.md similarity index 38% rename from kylinbot-auto-test/README.md rename to README.md index 4aaa9905af683f91176470e342e734070bf421a7..84d226edd0dbf2beeb0fdf123963616b2f686244 100644 --- a/kylinbot-auto-test/README.md +++ b/README.md @@ -1,18 +1,20 @@ # KylinBot 自动化测试脚本 -面向 **麒麟 Linux**的 WebSocket 对话批量测试工具：按用例驱动模型执行任务，用期望产物文件判断是否通过，汇总结果并生成可离线打开的 HTML 报告；可选调用通义大模型写文字总结。 +面向 **麒麟 Linux** 的 WebSocket 对话批量测试工具：按用例驱动模型执行任务，用期望产物文件判断是否通过，汇总结果并生成可离线打开的 HTML 报告；可选调用通义大模型写文字总结。也支持在 Windows 上做本地调试。 ## 目录结构 ``` -自动化测试脚本/ +kylinbot-auto-test/ ├── batch_run_ws_chat.py # 批量入口（推荐从这里跑） ├── ws_chat_benchmark_lib.py # 单次 WebSocket 压测 / 对话执行 +├── get_ws_token.py # 网关 Token 自动获取 / 缓存 / 探活 ├── collect_batch_summary.py # 扫描 run_summary.json → batch_summary.json ├── check_file.py # 期望产物路径展开与存在性检查 ├── test_cases.json # 测试用例 ├── requirement.txt # Python 依赖 ├── utils/ +│ ├── environment.py # 采集运行环境（系统 / 包版本 / Gateway 状态） │ └── summary.py # 大模型总结 batch_summary.json └── web/ # HTML 报告模板（跑完后会复制到 batch 目录） ├── index.html @@ -25,12 +27,12 @@ - Python 3.12+ - 可访问的 KylinBot Gateway（默认 `ws://127.0.0.1:42617/ws/chat`） -- 网关 Token +- 网关 Token（须以 `zc` 开头；未手动提供时可自动配对获取） ### 安装依赖 ```bash -cd /path/to/自动化测试脚本 +cd /path/to/kylinbot-auto-test python3 -m pip install -r requirement.txt ``` @@ -41,22 +43,26 @@ python3 -m pip install -r requirement.txt ### 环境变量 -| 变量 | 用途 | -|------|------| -| `KYLINBOT_WS_TOKEN` 或 `KYLINBOT_GATEWAY_TOKEN` | 网关 Token（也可用 `--token`） | -| `DASHSCOPE_API_KEY` 或 `OPENAI_API_KEY` | 大模型总结 API Key（可选） | -| `DASHSCOPE_BASE_URL` | 默认 `https://dashscope.aliyuncs.com/compatible-mode/v1` | -| `DASHSCOPE_MODEL` | 默认 `qwen3.7-max`，可选 | +| 变量 | 用途 | +| --------------------------------------------------- | ----------------------------------------------------------------------- | +| `KYLINBOT_WS_TOKEN` 或 `KYLINBOT_GATEWAY_TOKEN` | 网关 Token（也可用`--token`） | +| `KYLINBOT_WS_TOKEN_FILE` | 本地明文 Token 缓存路径（默认项目根目录`.ws_token`） | +| `KYLINBOT_GATEWAY_URL` | 自动取 Token 时的 Gateway HTTP 根地址（默认`http://127.0.0.1:42617`） | +| `DASHSCOPE_API_KEY` 或 `OPENAI_API_KEY` | 大模型总结 API Key（可选） | +| `DASHSCOPE_BASE_URL` | 默认`https://dashscope.aliyuncs.com/compatible-mode/v1` | +| `DASHSCOPE_MODEL` | 默认`qwen3.7-max` | 在麒麟上写入 `~/.bashrc`： ```bash -export KYLINBOT_WS_TOKEN="你的网关token" +export KYLINBOT_WS_TOKEN="zc_你的网关token" export DASHSCOPE_API_KEY="sk-xxx" ``` 然后执行 `source ~/.bashrc`，用 `echo "$DASHSCOPE_API_KEY"` 确认已生效。 +> `.ws_token` 已在 `.gitignore` 中忽略，请勿把真实 Token / API Key 提交进仓库。 + ## 测试用例 `test_cases.json` 每条用例示例： @@ -75,32 +81,50 @@ export DASHSCOPE_API_KEY="sk-xxx" ] ``` -| 字段 | 说明 | -|------|------| -| `id` / `name` | 用例标识与显示名 | -| `enabled` | 是否默认执行 | -| `count` | 该用例重复次数 | -| `mode` | `agentic` / `direct` | -| `message` | 发给模型的测试命令 | -| `output` | 期望产物路径；跑完后检查是否存在，作为 **通过判定** | +### 字段说明 + +| 字段 | 必填 | 说明 | +|------|------|------| +| `id` | 建议 | 用例唯一标识；`--only` / `--skip` 按此过滤 | +| `name` | 可选 | 展示名称，缺省时用 `id` | +| `message` | 是 | 发给 Agent 的完整 Prompt | +| `output` | 是 | 模型对话产出文件，用于判断模型是否正确执行 | +| `enabled` | 否 | 默认 `true`；为 `false` 时除非 `--only` 指定否则跳过 | +| `count` | 否 | 该用例重复跑几次，默认 `1` | +| `mode` | 否 | `agentic` 或 `direct`，默认 `agentic`（见下方说明） | +| `timeout` | 否 | 单次超时秒数；不写则用命令行 `--timeout`（默认 `180`） | +| `model` | 否 | 覆盖模型 | +| `approval_decision` | 否 | `approve` / `deny` / `always` | +| `note` | 否 | 备注，运行时打印 | +| `extra_args` | 否 | 额外透传给 bench 脚本的参数列表 | + +另支持（可选）：`token_saving`、`subordinate_agent`。 + +### `mode` 说明 -可选字段：`model`、`timeout`、`token_saving`、`approval_decision`、`subordinate_agent`、`note`、`extra_args`。 +- **`agentic`**（默认）：Agent 模式，可走工具调用 / 多步执行，适合文件读写、文档处理等任务类用例。 +- **`direct`**：直连对话模式，不走 Agent 工具链，适合纯问答类场景。 -路径支持 `~/桌面` 与 `~/Desktop`，脚本会映射到系统真实桌面目录（`xdg-user-dir DESKTOP` / `桌面` / `Desktop`）。 +命令行 `--mode` 会覆盖用例里的 `mode`。 + +路径支持 `~/桌面` 与 `~/Desktop`，脚本会映射到系统真实桌面目录（`xdg-user-dir DESKTOP` / `桌面` / `Desktop`）。跑完后检查 `output` 是否存在，作为通过判定。 ## 快速开始 ```bash -cd /path/to/自动化测试脚本 +cd /path/to/kylinbot-auto-test # 干跑：只打印命令 -python3 batch_run_ws_chat.py --token "$KYLINBOT_WS_TOKEN" --dry-run +python3 batch_run_ws_chat.py --dry-run + +# 正式跑（未传 --token 时自动读缓存 / 探活 / 必要时重新配对） +python3 batch_run_ws_chat.py -v -# 正式跑（推荐加 -v 看流式输出） +# 手动指定 Token python3 batch_run_ws_chat.py --token "$KYLINBOT_WS_TOKEN" -v # 指定用例 / 次数 / 输出根目录 -python3 batch_run_ws_chat.py --token "$KYLINBOT_WS_TOKEN" \ +python3 batch_run_ws_chat.py \ --cases test_cases.json \ --only 文档校对 \ --count 1 \ @@ -110,17 +134,20 @@ python3 batch_run_ws_chat.py --token "$KYLINBOT_WS_TOKEN" \ ### 常用参数 -| 参数 | 说明 | -|------|------| -| `--token` | 网关 Token | -| `--url` | WebSocket 地址，默认 `ws://127.0.0.1:42617/ws/chat` | -| `--cases` | 用例 JSON，默认 `test_cases.json` | -| `--only` / `--skip` | 按 `id` 白名单 / 黑名单 | -| `--count` | 覆盖所有用例的采样次数 | -| `--save-root` | 相对路径 → `<桌面>//batch_时间戳`；也可传绝对路径 | -| `--fail-fast` | 一失败即停 | -| `--dry-run` | 只打印命令 | -| `-v` | 打印 thinking / 回复 / 工具调用 | +| 参数 | 说明 | +| ----------------------- | -------------------------------------------------------------- | +| `--token` | 网关 Token（须以`zc` 开头） | +| `--token-file` | 本地 Token 缓存路径 | +| `--refresh-token` | 忽略本地缓存，强制重新配对 | +| `--gateway-url` | 自动取 Token 时的 HTTP 根地址 | +| `--url` | WebSocket 地址，默认`ws://127.0.0.1:42617/ws/chat` | +| `--cases` | 用例 JSON，默认`test_cases.json` | +| `--only` / `--skip` | 按`id` 白名单 / 黑名单 | +| `--count` | 覆盖所有用例的采样次数 | +| `--save-root` | 相对路径 →`<桌面>//batch_时间戳`；也可传绝对路径 | +| `--fail-fast` | 一失败即停 | +| `--dry-run` | 只打印命令 | +| `-v` | 打印 thinking / 回复 / 工具调用 | ## 执行流程 @@ -129,6 +156,8 @@ test_cases.json │ ▼ batch_run_ws_chat.py + │ 未提供 token → get_ws_token 缓存/配对 + │ 采集 environment.json │ 对每条用例调用 ▼ ws_chat_benchmark_lib.py ──WebSocket──► KylinBot Gateway @@ -136,9 +165,10 @@ ws_chat_benchmark_lib.py ──WebSocket──► KylinBot Gateway │ 判定：ws_ok（收到 done）+ ok（output 文件存在） ▼ batch_时间戳/ + ├── environment.json ├── 001_/run_*/run_summary.json ├── batch_index.json / .md - ├── batch_summary.json ← 汇总（含各 run 关键字段） + ├── batch_summary.json ← 汇总（含 environment + 各 run 关键字段） ├── batch_llm_summary.md ← 可选大模型总结 └── web/index.html ← 离线报告（内嵌数据，直接打开） ``` @@ -149,6 +179,22 @@ batch_时间戳/ - `ok`：期望 `output` 文件存在（未配置 `output` 时不因文件检查失败） - 页面「结果」列优先看 sample 的 `ok` +## Token 自动获取 + +未传 `--token` / 环境变量时，`batch_run_ws_chat.py` 会调用 `get_ws_token.py` 逻辑： + +1. 本地无有效 `zc` 明文 → `POST /admin/paircode/new` + `POST /pair` → 写入 `.ws_token` +2. 本地有明文 → 先使用 +3. `GET /api/status` 探活；不可用则重新 pair 并覆盖缓存 + +也可单独使用： + +```bash +python3 get_ws_token.py +python3 get_ws_token.py --refresh +eval "$(python3 get_ws_token.py --export)" +``` + ## 结果与报告 默认输出目录示例： @@ -159,12 +205,13 @@ batch_时间戳/ 重要文件： -| 文件 | 内容 | -|------|------| -| `batch_summary.json` | 各 run 的 `timestamp` / `url` / `message` / `output` / `samples`，以及可选 `summary` | -| `web/index.html` | 可视化报告：通过率、耗时柱状图、用例表、大模型总结 | -| `web/data.js` | 内嵌 JSON，支持 `file://` 直接打开 | -| `*/run_*/sample_*.md` | 单次对话详情（thinking、工具、回复） | +| 文件 | 内容 | +| ----------------------- | ------------------------------------------------------------------------------------------------------------------ | +| `environment.json` | 运行环境（OS、Python、`kylin-bot` 包版本、Gateway status 等） | +| `batch_summary.json` | 各 run 的`timestamp` / `url` / `message` / `output` / `samples`，以及 `environment` 与可选 `summary` | +| `web/index.html` | 可视化报告：运行环境、通过率、耗时柱状图、用例表、大模型总结 | +| `web/data.js` | 内嵌 JSON，支持`file://` 直接打开 | +| `*/run_*/sample_*.md` | 单次对话详情（thinking、工具、回复） | 在麒麟上双击或用浏览器打开： @@ -209,14 +256,14 @@ python3 check_file.py ## 判定与指标说明（简要） -| 字段 | 含义 | -|------|------| -| `send_message_total_ms` | 发 `message` → 收到 `done` | -| `first_thinking_ms` | 发 `message` → 首次 `thinking` 帧 | -| `first_visible_text_ms` | 发 `message` → 首次 `chunk` | -| `queue_to_first_frame_ms` | 发 `message` → 任意首帧 | -| `ws_ok` | 会话是否正常结束 | -| `ok` / `output_ok` | 期望产物是否存在 | +| 字段 | 含义 | +| --------------------------- | ------------------------------------- | +| `send_message_total_ms` | 发`message` → 收到 `done` | +| `first_thinking_ms` | 发`message` → 首次 `thinking` 帧 | +| `first_visible_text_ms` | 发`message` → 首次 `chunk` | +| `queue_to_first_frame_ms` | 发`message` → 任意首帧 | +| `ws_ok` | 会话是否正常结束 | +| `ok` / `output_ok` | 期望产物是否存在 | HTML 报告中耗时以 **秒** 显示。 @@ -225,4 +272,4 @@ HTML 报告中耗时以 **秒** 显示。 1. 跑前若期望产物已存在，可能被误判为成功；脚本会打印警告。 2. 大模型总结依赖外网访问 DashScope；无 Key 时会跳过总结，不影响测试退出码逻辑（汇总/网页仍会尽量生成）。 3. IDE 里运行可能读不到 `~/.bashrc` 里的环境变量，请在终端执行或在 IDE 中配置环境变量。 -4. 请勿把真实 Token / API Key 提交进仓库。 +4. Token 须以 `zc` 开头；请勿把真实 Token / API Key 提交进仓库。 diff --git a/kylinbot-auto-test/.gitignore b/kylinbot-auto-test/.gitignore new file mode 100644 index 0000000000000000000000000000000000000000..d20a88e5865404a23ce52a9dfbe667e3ce0524b8 --- /dev/null +++ b/kylinbot-auto-test/.gitignore @@ -0,0 +1,30 @@ +.ws_token + +# Python +__pycache__/ +*.py[cod] +*$py.class +*.so +.Python +*.egg-info/ +.eggs/ +dist/ +build/ +*.egg + +# Virtual environments +.venv/ +venv/ +env/ + +# Testing / coverage / type check caches +.pytest_cache/ +.mypy_cache/ +.ruff_cache/ +.coverage +htmlcov/ +.tox/ +.cache/ + +# Jupyter +.ipynb_checkpoints/ diff --git a/kylinbot-auto-test/batch_run_ws_chat.py b/kylinbot-auto-test/batch_run_ws_chat.py index f3d6c78507d2bfe95099e378c687ab28683eda15..fb8a1d2aeed0fe2db8c4a6032a1a32293af3abc5 100644 --- a/kylinbot-auto-test/batch_run_ws_chat.py +++ b/kylinbot-auto-test/batch_run_ws_chat.py @@ -4,10 +4,11 @@ Target: Kylin / Linux (also works on Windows for local debug). Example: - python3 batch_run_ws_chat.py --token - python3 batch_run_ws_chat.py --token --cases test_cases.json --count 1 -v - python3 batch_run_ws_chat.py --token --save-root ~/桌面/ws-chat-batch - python3 batch_run_ws_chat.py --token --only doc-summary,ppt-generate + python3 batch_run_ws_chat.py + python3 batch_run_ws_chat.py --cases test_cases.json --count 1 -v + python3 batch_run_ws_chat.py --save-root ~/桌面/ws-chat-batch + python3 batch_run_ws_chat.py --only doc-summary,ppt-generate + python3 batch_run_ws_chat.py --token # 也可手动传入；须以 zc 开头 """ from __future__ import annotations @@ -35,6 +36,18 @@ if str(SCRIPT_DIR) not in sys.path: from check_file import expand_output_path, output_exists, resolve_desktop_dir # noqa: E402 from collect_batch_summary import write_batch_summary # noqa: E402 +from get_ws_token import ( # noqa: E402 + DEFAULT_BASE_URL, + DEFAULT_TOKEN_FILE, + http_base_from_ws_url, + require_zc_token, + resolve_gateway_token, +) +from utils.environment import ( # noqa: E402 + collect_environment, + format_environment_lines, + write_environment_file, +) def load_cases(path: Path) -> list[dict[str, Any]]: @@ -93,7 +106,30 @@ def parse_args(argv: list[str] | None = None) -> argparse.Namespace: parser.add_argument( "--token", default=os.environ.get("KYLINBOT_WS_TOKEN") or os.environ.get("KYLINBOT_GATEWAY_TOKEN"), - help="Gateway token. Env: KYLINBOT_WS_TOKEN / KYLINBOT_GATEWAY_TOKEN", + help=( + "Gateway token（须以 zc 开头）。" + "未提供时自动读本地缓存 / 探活 / 必要时重新配对。" + "Env: KYLINBOT_WS_TOKEN / KYLINBOT_GATEWAY_TOKEN" + ), + ) + parser.add_argument( + "--token-file", + default=os.environ.get("KYLINBOT_WS_TOKEN_FILE", str(DEFAULT_TOKEN_FILE)), + help=f"本地明文 token 缓存路径（默认 {DEFAULT_TOKEN_FILE.name}）", + ) + parser.add_argument( + "--refresh-token", + action="store_true", + help="忽略本地 token 缓存，强制重新配对并覆盖", + ) + parser.add_argument( + "--gateway-url", + default=os.environ.get("KYLINBOT_GATEWAY_URL"), + help=( + "自动取 token 时使用的 Gateway HTTP 根地址。" + "默认由 --url 推导，或 http://127.0.0.1:42617。" + "Env: KYLINBOT_GATEWAY_URL" + ), ) parser.add_argument( "--url", @@ -262,10 +298,36 @@ def task(argv: list[str] | None = None) -> tuple[int, Path | None]: print(f"bench script not found: {bench_path}", file=sys.stderr) return 2, None if not args.token and not args.dry_run: + gateway_url = args.gateway_url + if not gateway_url: + try: + gateway_url = http_base_from_ws_url(args.url) + except RuntimeError: + gateway_url = DEFAULT_BASE_URL print( - "缺少 --token（或环境变量 KYLINBOT_WS_TOKEN / KYLINBOT_GATEWAY_TOKEN）", + f"未提供 token，使用本地缓存/自动配对（gateway={gateway_url}）…", file=sys.stderr, ) + try: + args.token = resolve_gateway_token( + gateway_url, + token_file=args.token_file, + force_refresh=args.refresh_token, + quiet=False, + ) + except RuntimeError as exc: + print(f"错误: {exc}", file=sys.stderr) + print("获取配对码/token 失败，已退出，未执行测试。", file=sys.stderr) + return 2, None + + if args.token: + try: + args.token = require_zc_token(args.token) + except RuntimeError as exc: + print(f"错误: {exc}", file=sys.stderr) + return 2, None + elif not args.dry_run: + print("缺少有效 token", file=sys.stderr) return 2, None cases = load_cases(cases_path) @@ -284,6 +346,17 @@ def task(argv: list[str] | None = None) -> tuple[int, Path | None]: batch_dir = batch_root / f"batch_{stamp}" if not args.dry_run: batch_dir.mkdir(parents=True, exist_ok=True) + environment = collect_environment( + ws_url=args.url, + token=args.token, + model_override=args.model, + ) + env_path = write_environment_file(batch_dir, environment) + print("environment") + for line in format_environment_lines(environment): + print(f" {line}") + print(f" wrote {env_path}") + print() print(f"cases file : {cases_path}") print(f"bench : {bench_path}") diff --git a/kylinbot-auto-test/collect_batch_summary.py b/kylinbot-auto-test/collect_batch_summary.py index 6d162c3110f69c74795a557987861a5fc294e189..0e51cbf9349bfce41f80cddad8284108806625b1 100644 --- a/kylinbot-auto-test/collect_batch_summary.py +++ b/kylinbot-auto-test/collect_batch_summary.py @@ -10,12 +10,21 @@ from datetime import datetime from pathlib import Path from typing import Any +SCRIPT_DIR = Path(__file__).resolve().parent +if str(SCRIPT_DIR) not in sys.path: + sys.path.insert(0, str(SCRIPT_DIR)) + +from utils.environment import ( # noqa: E402 + enrich_environment_with_runs, + load_environment_file, +) + # Fields kept from each run_summary.json -KEEP_FIELDS = ("timestamp", "url", "message", "samples", "output") +KEEP_FIELDS = ("timestamp", "url", "message", "samples", "output", "model_override") def extract_summary_fields(summary: dict[str, Any]) -> dict[str, Any]: - """Keep only timestamp / url / message / samples from a run summary.""" + """Keep selected fields from a run summary.""" return {key: summary.get(key) for key in KEEP_FIELDS} @@ -47,10 +56,16 @@ def collect_run_summaries(batch_dir: Path) -> dict[str, Any]: entry["path"] = rel runs.append(entry) + environment = enrich_environment_with_runs( + load_environment_file(batch_dir), + runs, + ) + return { "batch_dir": str(batch_dir), "collected_at": datetime.now().isoformat(timespec="seconds"), "count": len(runs), + "environment": environment, "runs": runs, } diff --git a/kylinbot-auto-test/get_ws_token.py b/kylinbot-auto-test/get_ws_token.py new file mode 100644 index 0000000000000000000000000000000000000000..47698dec9ac0beea70483553fff8582b56b5aabf --- /dev/null +++ b/kylinbot-auto-test/get_ws_token.py @@ -0,0 +1,355 @@ +#!/usr/bin/env python3 +"""自动获取 / 缓存 KylinBot Gateway WebSocket Token（KYLINBOT_WS_TOKEN）。 + +流程： + 1. 本地无 zc_ 明文 → POST /admin/paircode/new + POST /pair → 写入本地缓存 + 2. 本地有明文 → 先使用 + 3. GET /api/status 探活；不可用则重新 pair，并覆盖本地缓存 + +用法示例： + python3 get_ws_token.py + python3 get_ws_token.py --refresh + eval "$(python3 get_ws_token.py --export)" + python3 batch_run_ws_chat.py # 未传 --token 时自动走本逻辑 +""" + +from __future__ import annotations + +import argparse +import json +import os +import stat +import sys +import urllib.error +import urllib.request +from pathlib import Path +from typing import Optional +from urllib.parse import urlparse + + +DEFAULT_BASE_URL = "http://127.0.0.1:42617" +TOKEN_PREFIX = "zc" +SCRIPT_DIR = Path(__file__).resolve().parent +DEFAULT_TOKEN_FILE = SCRIPT_DIR / ".ws_token" + + +def eprint(*args: object) -> None: + print(*args, file=sys.stderr) + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser( + description="获取并缓存 KYLINBOT_WS_TOKEN（本地探活，失效则重新配对）", + ) + parser.add_argument( + "--base-url", + default=os.environ.get("KYLINBOT_GATEWAY_URL", DEFAULT_BASE_URL), + help=f"Gateway HTTP 根地址（默认 {DEFAULT_BASE_URL}，可用 KYLINBOT_GATEWAY_URL）", + ) + parser.add_argument( + "--token-file", + default=os.environ.get("KYLINBOT_WS_TOKEN_FILE", str(DEFAULT_TOKEN_FILE)), + help=f"本地明文 token 缓存路径（默认 {DEFAULT_TOKEN_FILE.name}）", + ) + parser.add_argument( + "--timeout", + type=float, + default=30.0, + help="HTTP 超时秒数（默认 30）", + ) + parser.add_argument( + "--refresh", + action="store_true", + help="忽略本地缓存，强制重新配对并覆盖", + ) + parser.add_argument( + "--export", + action="store_true", + help="输出可 eval 的 export KYLINBOT_WS_TOKEN=... 语句", + ) + parser.add_argument( + "-q", + "--quiet", + action="store_true", + help="减少 stderr 日志", + ) + return parser.parse_args() + + +def normalize_base_url(url: str) -> str: + return url.rstrip("/") + + +def http_base_from_ws_url(ws_url: str) -> str: + """ws://host:port/ws/chat -> http://host:port""" + parsed = urlparse(ws_url) + scheme = "https" if parsed.scheme in ("wss", "https") else "http" + if not parsed.hostname: + raise RuntimeError(f"无法从 WebSocket URL 解析主机: {ws_url}") + netloc = parsed.hostname + if parsed.port: + netloc = f"{netloc}:{parsed.port}" + return f"{scheme}://{netloc}" + + +def http_json( + method: str, + url: str, + *, + headers: Optional[dict[str, str]] = None, + timeout: float = 30.0, +) -> dict: + status, data = http_json_with_status( + method, url, headers=headers, timeout=timeout + ) + if status >= 400: + raise RuntimeError( + f"HTTP {status} {method} {url}\n" + f"{json.dumps(data, ensure_ascii=False) if data else ''}".strip() + ) + return data + + +def http_json_with_status( + method: str, + url: str, + *, + headers: Optional[dict[str, str]] = None, + timeout: float = 30.0, +) -> tuple[int, dict]: + req = urllib.request.Request(url, method=method, headers=headers or {}) + try: + with urllib.request.urlopen(req, timeout=timeout) as resp: + body = resp.read().decode("utf-8", errors="replace") + status = getattr(resp, "status", 200) + except urllib.error.HTTPError as exc: + err_body = exc.read().decode("utf-8", errors="replace") + try: + data = json.loads(err_body) if err_body.strip() else {} + except json.JSONDecodeError: + data = {"error": err_body.strip()} if err_body.strip() else {} + if not isinstance(data, dict): + data = {"error": err_body.strip()} + return exc.code, data + except urllib.error.URLError as exc: + raise RuntimeError(f"无法连接 {url}: {exc.reason}") from exc + except TimeoutError as exc: + raise RuntimeError(f"请求超时 {url} (timeout={timeout}s)") from exc + + try: + data = json.loads(body) if body.strip() else {} + except json.JSONDecodeError as exc: + raise RuntimeError( + f"响应不是 JSON (HTTP {status} {method} {url}):\n{body.strip()}" + ) from exc + + if not isinstance(data, dict): + raise RuntimeError(f"期望 JSON object，实际: {type(data).__name__}") + return status, data + + +def fetch_pair_code(base_url: str, timeout: float) -> str: + """POST /admin/paircode/new；失败立即报错退出，不继续换 token。""" + url = f"{base_url}/admin/paircode/new" + try: + data = http_json("POST", url, timeout=timeout) + except RuntimeError as exc: + raise RuntimeError( + "获取配对码失败。请确认 KylinBot Gateway 已启动，" + f"且本机可访问 {url}\n" + f"原因: {exc}" + ) from exc + + code = data.get("pairing_code") + if code is None or str(code).strip() == "": + message = data.get("message") or data.get("error") or json.dumps( + data, ensure_ascii=False + ) + raise RuntimeError( + "获取配对码失败：接口未返回 pairing_code。" + f"请确认 gateway 已开启 pairing。详情: {message}" + ) + + code = str(code).strip() + if not code.isdigit(): + raise RuntimeError(f"获取配对码失败：pairing_code 格式异常: {code!r}") + return code + + +def exchange_token(base_url: str, pair_code: str, timeout: float) -> str: + url = f"{base_url}/pair" + data = http_json( + "POST", + url, + headers={"X-Pairing-Code": pair_code}, + timeout=timeout, + ) + token = data.get("token") + if not token: + message = data.get("message") or data.get("error") or json.dumps( + data, ensure_ascii=False + ) + raise RuntimeError(f"/pair 未返回 token: {message}") + return str(token).strip() + + +def require_zc_token(token: str) -> str: + token = (token or "").strip() + if not token: + raise RuntimeError("token 为空") + if not token.startswith(TOKEN_PREFIX): + raise RuntimeError( + f"token 必须以 {TOKEN_PREFIX!r} 开头，当前前缀: {token[:8]!r}" + ) + return token + + +def load_cached_token(token_file: Path) -> Optional[str]: + if not token_file.is_file(): + return None + try: + raw = token_file.read_text(encoding="utf-8").strip() + except OSError as exc: + raise RuntimeError(f"读取本地 token 失败: {token_file}: {exc}") from exc + if not raw: + return None + # 只取第一行，避免误粘贴多行 + first = raw.splitlines()[0].strip() + try: + return require_zc_token(first) + except RuntimeError: + return None + + +def save_cached_token(token_file: Path, token: str) -> None: + token = require_zc_token(token) + token_file.parent.mkdir(parents=True, exist_ok=True) + token_file.write_text(token + "\n", encoding="utf-8") + try: + os.chmod(token_file, stat.S_IRUSR | stat.S_IWUSR) + except OSError: + # Windows 等平台可能不支持 chmod，忽略 + pass + + +def token_is_alive( + base_url: str, + token: str, + *, + timeout: float = 30.0, + quiet: bool = False, +) -> bool: + """GET /api/status 探活。连不上 gateway 直接抛错；401/403 等视为不可用。""" + url = f"{normalize_base_url(base_url)}/api/status" + if not quiet: + eprint(f"[get_ws_token] 探活 GET {url}") + try: + status, data = http_json_with_status( + "GET", + url, + headers={"Authorization": f"Bearer {token}"}, + timeout=timeout, + ) + except RuntimeError as exc: + raise RuntimeError( + f"探活失败（无法访问 gateway）。请确认 Gateway 已启动。\n原因: {exc}" + ) from exc + + if status == 200: + if not quiet: + eprint("[get_ws_token] 本地 token 可用") + return True + + if not quiet: + detail = data.get("error") or data.get("message") or json.dumps( + data, ensure_ascii=False + ) + eprint(f"[get_ws_token] 本地 token 不可用 (HTTP {status}): {detail}") + return False + + +def obtain_gateway_token( + base_url: str = DEFAULT_BASE_URL, + *, + timeout: float = 30.0, + quiet: bool = False, +) -> str: + """申请配对码并换取 token，校验后返回。配对码获取失败则直接抛错。""" + base_url = normalize_base_url(base_url) + if not quiet: + eprint(f"[get_ws_token] POST {base_url}/admin/paircode/new") + + pair_code = fetch_pair_code(base_url, timeout) + + if not quiet: + eprint(f"[get_ws_token] 配对码: {pair_code}") + eprint(f"[get_ws_token] POST {base_url}/pair") + + try: + token = exchange_token(base_url, pair_code, timeout) + except RuntimeError as exc: + raise RuntimeError(f"用配对码换取 token 失败: {exc}") from exc + + token = require_zc_token(token) + if not quiet: + eprint("[get_ws_token] 成功获取 token") + return token + + +def resolve_gateway_token( + base_url: str = DEFAULT_BASE_URL, + *, + token_file: Path | str = DEFAULT_TOKEN_FILE, + timeout: float = 30.0, + force_refresh: bool = False, + quiet: bool = False, +) -> str: + """读取本地缓存 → 探活 → 必要时重新配对并覆盖缓存。""" + base_url = normalize_base_url(base_url) + path = Path(token_file).expanduser() + + if not force_refresh: + cached = load_cached_token(path) + if cached: + if not quiet: + eprint(f"[get_ws_token] 使用本地缓存: {path}") + if token_is_alive(base_url, cached, timeout=timeout, quiet=quiet): + return cached + if not quiet: + eprint("[get_ws_token] 缓存失效，重新配对并覆盖本地文件") + elif not quiet: + eprint(f"[get_ws_token] 本地无有效缓存: {path}") + elif not quiet: + eprint("[get_ws_token] --refresh：强制重新配对") + + token = obtain_gateway_token(base_url, timeout=timeout, quiet=quiet) + save_cached_token(path, token) + if not quiet: + eprint(f"[get_ws_token] 已写入本地缓存: {path}") + return token + + +def main() -> int: + args = parse_args() + try: + token = resolve_gateway_token( + args.base_url, + token_file=args.token_file, + timeout=args.timeout, + force_refresh=args.refresh, + quiet=args.quiet, + ) + except RuntimeError as exc: + eprint(f"错误: {exc}") + return 1 + + if args.export: + safe = token.replace("'", "'\"'\"'") + print(f"export KYLINBOT_WS_TOKEN='{safe}'") + else: + print(token) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/kylinbot-auto-test/utils/environment.py b/kylinbot-auto-test/utils/environment.py new file mode 100644 index 0000000000000000000000000000000000000000..f2e9a64c12dd14699b70dc016eea59a84431ce7d --- /dev/null +++ b/kylinbot-auto-test/utils/environment.py @@ -0,0 +1,186 @@ +#!/usr/bin/env python3 +"""采集批次运行环境信息（写入 batch_summary.environment / Web 展示）。""" + +from __future__ import annotations + +import json +import platform +import re +import subprocess +import sys +from datetime import datetime +from pathlib import Path +from typing import Any, Optional +from urllib.error import HTTPError, URLError +from urllib.parse import urlparse +from urllib.request import Request, urlopen + + +ENV_FILENAME = "environment.json" + + +def _ws_to_http_origin(ws_url: str) -> str: + parsed = urlparse(ws_url) + scheme = "https" if parsed.scheme in ("wss", "https") else "http" + if not parsed.hostname: + raise ValueError(f"invalid WebSocket URL: {ws_url}") + netloc = parsed.hostname + if parsed.port: + netloc = f"{netloc}:{parsed.port}" + return f"{scheme}://{netloc}" + + +def get_kylin_bot_dpkg_version() -> Optional[str]: + """通过 `dpkg -s kylin-bot` 解析 Version 字段。""" + try: + completed = subprocess.run( + ["dpkg", "-s", "kylin-bot"], + check=False, + capture_output=True, + text=True, + timeout=10, + ) + except (OSError, subprocess.TimeoutExpired): + return None + + text = (completed.stdout or "") + "\n" + (completed.stderr or "") + for line in text.splitlines(): + if line.lower().startswith("version:"): + return line.split(":", 1)[1].strip() or None + # 兼容 grep Version 风格的单行输出 + match = re.search(r"(?im)^Version:\s*(.+)$", text) + if match: + return match.group(1).strip() or None + return None + + +def get_python_version() -> str: + return ( + f"{sys.version_info.major}.{sys.version_info.minor}.{sys.version_info.micro} " + f"({platform.python_implementation()})" + ) + + +def fetch_gateway_status( + ws_url: str, + token: Optional[str], + *, + timeout: float = 15.0, +) -> dict[str, Any]: + origin = _ws_to_http_origin(ws_url) + url = f"{origin}/api/status" + headers = {"Accept": "application/json"} + if token: + headers["Authorization"] = f"Bearer {token.strip()}" + req = Request(url, headers=headers, method="GET") + try: + with urlopen(req, timeout=timeout) as resp: + body = resp.read().decode("utf-8", errors="replace") + data = json.loads(body) + return data if isinstance(data, dict) else {"raw": data} + except HTTPError as exc: + detail = exc.read().decode("utf-8", errors="replace")[:300] + return {"error": f"HTTP {exc.code}", "detail": detail} + except (URLError, TimeoutError, json.JSONDecodeError, OSError, ValueError) as exc: + return {"error": str(exc)} + + +def collect_environment( + *, + ws_url: str, + token: Optional[str] = None, + model_override: Optional[str] = None, + gateway_status: Optional[dict[str, Any]] = None, +) -> dict[str, Any]: + """采集环境基础信息。""" + status = gateway_status if gateway_status is not None else fetch_gateway_status( + ws_url, token + ) + status_ok = isinstance(status, dict) and "error" not in status + + env: dict[str, Any] = { + "collected_at": datetime.now().isoformat(timespec="seconds"), + "kylinbot_version": get_kylin_bot_dpkg_version(), + "kylinbot_version_source": "dpkg -s kylin-bot | grep Version", + "python_version": get_python_version(), + "provider": status.get("provider") if status_ok else None, + "model": status.get("model") if status_ok else None, + "temperature": status.get("temperature") if status_ok else None, + "model_override": (model_override or None), + "gateway_status_error": None if status_ok else status.get("error") or status, + "ws_url": ws_url, + } + return env + + +def derive_actual_models(runs: list[dict[str, Any]]) -> dict[str, Any]: + """从 runs/samples 汇总实际用到的 provider / model。""" + providers: set[str] = set() + models: set[str] = set() + for run in runs: + samples = run.get("samples") if isinstance(run, dict) else None + if not isinstance(samples, list): + continue + for sample in samples: + if not isinstance(sample, dict): + continue + if sample.get("provider"): + providers.add(str(sample["provider"])) + if sample.get("model"): + models.add(str(sample["model"])) + return { + "actual_providers": sorted(providers), + "actual_models": sorted(models), + } + + +def enrich_environment_with_runs( + environment: dict[str, Any] | None, + runs: list[dict[str, Any]], +) -> dict[str, Any]: + env = dict(environment or {}) + env.update(derive_actual_models(runs)) + return env + + +def write_environment_file(batch_dir: Path, environment: dict[str, Any]) -> Path: + path = batch_dir / ENV_FILENAME + path.write_text( + json.dumps(environment, ensure_ascii=False, indent=2, default=str), + encoding="utf-8", + ) + return path + + +def load_environment_file(batch_dir: Path) -> Optional[dict[str, Any]]: + path = batch_dir / ENV_FILENAME + if not path.is_file(): + return None + try: + data = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + return None + return data if isinstance(data, dict) else None + + +def format_environment_lines(environment: dict[str, Any]) -> list[str]: + """终端打印用的多行摘要。""" + override = environment.get("model_override") + actual_models = environment.get("actual_models") or [] + actual_providers = environment.get("actual_providers") or [] + lines = [ + f"kylinbot 版本 : {environment.get('kylinbot_version') or 'unknown'}", + f"Python 版本 : {environment.get('python_version') or 'unknown'}", + f"默认 provider : {environment.get('provider') or 'unknown'}", + f"默认 model : {environment.get('model') or 'unknown'}", + f"temperature : {environment.get('temperature') if environment.get('temperature') is not None else 'unknown'}", + ] + if override: + lines.append(f"model override: {override}") + if actual_models: + lines.append(f"实际 models : {', '.join(actual_models)}") + if actual_providers: + lines.append(f"实际 providers: {', '.join(actual_providers)}") + if environment.get("gateway_status_error"): + lines.append(f"gateway status: error ({environment['gateway_status_error']})") + return lines diff --git a/kylinbot-auto-test/web/app.js b/kylinbot-auto-test/web/app.js index 6d6d74b5988d97e47eaa9b82d022c1e9e61f5959..493feca5c4ea9f1930abaff7e2c853d87c5ad715 100644 --- a/kylinbot-auto-test/web/app.js +++ b/kylinbot-auto-test/web/app.js @@ -7,6 +7,7 @@ collectedAt: $("collectedAt"), batchDir: $("batchDir"), runCount: $("runCount"), + envSummary: $("envSummary"), statPass: $("statPass"), statFail: $("statFail"), statSamples: $("statSamples"), @@ -78,6 +79,44 @@ return rows; } + function displayValue(value) { + if (value == null || value === "") return "未知"; + if (Array.isArray(value)) { + return value.length ? value.join("、") : "未知"; + } + return String(value); + } + + function pickUsedModel(env) { + if (env.model_override) return env.model_override; + if (Array.isArray(env.actual_models) && env.actual_models.length) { + return env.actual_models.join("、"); + } + return env.model; + } + + function renderEnvironment(data) { + const el = els.envSummary; + if (!el) return; + const env = data?.environment && typeof data.environment === "object" + ? data.environment + : null; + if (!env) { + el.textContent = "暂无运行环境信息。"; + return; + } + const version = displayValue(env.kylinbot_version); + const python = displayValue(env.python_version); + const model = displayValue(pickUsedModel(env)); + const temperature = + env.temperature == null || env.temperature === "" + ? "未知" + : String(env.temperature); + el.textContent = + `本次测试环境：kylinbot 版本 ${version}，Python 版本 ${python}，` + + `使用模型 ${model}，temperature ${temperature}。`; + } + function renderMeta(data) { els.collectedAt.textContent = fmtTime(data.collected_at); els.batchDir.textContent = data.batch_dir || "—"; @@ -201,6 +240,7 @@ function render(data, source) { const rows = flattenRows(data); renderMeta(data); + renderEnvironment(data); renderStats(rows); renderChart(rows); renderTable(rows); diff --git a/kylinbot-auto-test/web/data/batch_summary.json b/kylinbot-auto-test/web/data/batch_summary.json index 877c3b11bc5d430a3ec60279cdadcd6c4c4eda9b..4ef7fe5f2544a5168b14de3d0cda9a12b275cc3f 100644 --- a/kylinbot-auto-test/web/data/batch_summary.json +++ b/kylinbot-auto-test/web/data/batch_summary.json @@ -2,6 +2,22 @@ "batch_dir": "/home/greatwall/桌面/ws-chat-batch/batch_20260821_150645", "collected_at": "2026-08-21T15:15:29", "count": 2, + "environment": { + "collected_at": "2026-08-21T15:06:45", + "kylinbot_version": "1.0.0-1", + "kylinbot_version_source": "dpkg -s kylin-bot | grep Version", + "python_version": "3.12.3 (CPython)", + "provider": "custom:https://dashscope.aliyuncs.com/compatible-mode/v1", + "model": "qwen3.7-plus", + "temperature": 0.7, + "model_override": null, + "gateway_status_error": null, + "ws_url": "ws://127.0.0.1:42617/ws/chat", + "actual_providers": [ + "custom:https://dashscope.aliyuncs.com/compatible-mode/v1" + ], + "actual_models": ["qwen3.7-plus"] + }, "runs": [ { "timestamp": "2026-08-21T15:12:20+08:00", diff --git a/kylinbot-auto-test/web/index.html b/kylinbot-auto-test/web/index.html index 385aa80092c038efcd7e5eb18a5ba0ec7a537cce..1f7c1aec8c03fa9e391fcbcdea0cefa9025afdda 100644 --- a/kylinbot-auto-test/web/index.html +++ b/kylinbot-auto-test/web/index.html @@ -29,6 +29,13 @@
+
+
+

运行环境

+
+

—

+
+
采集时间 diff --git a/kylinbot-auto-test/web/styles.css b/kylinbot-auto-test/web/styles.css index f4577f1ac6d315f6cde7c23a35804b769ffee243..0af2aad6989fdd0eeda8789057c40a483df49721 100644 --- a/kylinbot-auto-test/web/styles.css +++ b/kylinbot-auto-test/web/styles.css @@ -179,6 +179,25 @@ h1 { overflow: hidden; } +.env-panel { + margin-bottom: 1rem; + padding: 0; + overflow: hidden; + animation: rise 0.5s ease both; +} + +.env-panel .panel-head { + padding: 1.1rem 1.2rem 0.2rem; +} + +.env-summary { + margin: 0; + padding: 0.35rem 1.2rem 1.15rem; + color: var(--ink); + font-size: 0.98rem; + line-height: 1.7; +} + .stats { display: grid; grid-template-columns: repeat(4, minmax(0, 1fr));