# okriscv-harness

**Repository Path**: openkylin/okriscv-harness

## Basic Information

- **Project Name**: okriscv-harness
- **Description**: No description available
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2026-10-22
- **Last Updated**: 2026-10-22

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# okriscv-agent harness

## Purpose

This repository is an automated pipeline that finds performance problems in open-source
software on real RISC-V hardware, fixes them, proves each fix on the hardware, and drafts
the upstream pull request. A human approves every PR before it is submitted.

It works with any upstream repository. Nothing in the code or configuration is specific to
a project. Everything a project needs (how to build it, test it, and benchmark it) is
learned by an agent from the repository itself.

It is an independent implementation inspired by the Yuansheng team's harness described in
`../riscv-agent-patch-summary.md`. That harness was never published; this one was written
from scratch.

## Workflow

The harness learns each project once (**onboard**). Then every hot spot it finds becomes a
**record**, which moves through five stages. Each stage writes into the record's folder, so
a record holds the complete history of one issue.

```
 0. onboard   repository URL ─▶ agent reads the repo, builds / tests / benchmarks it ON THE BOARD ─▶ project profile
 1. collect   board: build, run the profile's workloads under perf (call stacks), group hot code into issues ─▶ records
 2. Trace     agent: why is this issue slow?  (perf data + disassembly + call stacks + source + ISA knowledge)
 3. Craft     agent: patch + validation plan (tests, target benchmark, every changed variant); built on the board; independent review
    ↳ upstream-check: already fixed or already proposed upstream? (before Craft, and again before the PR)
 4. CI        here: the project's own lint/format checks; board: baseline vs patched trees, tests on every core type, paired A/B benchmark ─▶ verdict
              (build / test failures caused by the patch go back to Craft for repair)
 5. PR        agent drafts it from CI numbers; a human approves; then publish (GitHub PR or format-patch)
```

0. **onboard** (`okriscv/onboard/`): an agent gets a repository URL and a board, and nothing
   else. It reads the repository, finds the build system, and builds natively on the board
   with debug symbols and frame pointers (enabling the project's own RISC-V options). It
   finds a smoke test, a full test suite, and 3–8 representative workloads (the project's
   own benchmarks or perf tests), running each one on the board, and lists the directories
   that hold test and benchmark driver code (`driver_paths`). It writes a **project
   profile** (`workspace/projects/<name>/profile.json`). The validator accepts the profile
   only if every command in it actually succeeded on the board, so the profile cannot
   contain guesses.
1. **collect** (`okriscv/collect.py`) builds the baseline from the profile, then runs each
   workload on each core type for a bounded time window. It uses `perf stat` for counters
   and `perf record -g` for call stacks, injecting JIT code automatically when the project
   emits jitdump data. Samples are grouped into issues by call stack (`okriscv/stacks.py`): each
   sample belongs to the innermost library function of the project; time in the project's own
   test and benchmark drivers (the profile's `driver_paths`, matched against each function's
   source file) is shown as context and never becomes an issue; each generated (JIT) kernel is
   its own issue; and a helper whose samples mostly come from one caller of
   comparable weight is folded into that caller. Each issue becomes a record holding the
   counters, the annotated disassembly of its hottest function, its call-stack grouping,
   the workload's own output, and the machine's facts.
2. **Trace** (`okriscv/trace/`) is an AI agent that sees evidence only through tools that log
   every lookup. It works through seven stages (evidence inventory → baseline → analysis
   boundary → pattern scan against the knowledge guide → root cause → validation forecast
   → self-check) and writes a *blueprint*: findings, gaps, and proposed actions. A
   deterministic validator rejects claims not backed by logged evidence, and an independent
   agent reviews the diagnosis against the raw data.
3. **Craft** (`okriscv/craft/`) is an AI agent working in an isolated checkout at the measured
   revision. It submits a plan before editing and edits only planned files, then
   build-checks the patch with an incremental build on the board. It submits the patch
   together with a **validation plan**: the existing tests that cover the change, a
   focused benchmark for the diagnosed case (the target), and a test and benchmark for
   **every variant** of the changed code (each instantiation or size, type, dispatch branch),
   adding perf cases to the project when none exist. An independent reviewer checks correctness,
   ISA guards, fidelity to the blueprint, five hallucination checks, and that no changed
   variant is left unmeasured.
4. **CI** (`okriscv/ci/`, `okriscv/trees.py`, `okriscv/lint.py`): first the project's own PR
   checks listed in its profile (e.g. oneDNN's clang-format 18), on this machine; a formatter
   rewrites the patch, other violations go back to Craft. Then the baseline tree is built once per revision. The
   candidate is a copy of it plus the patch, rebuilt incrementally. A symlink selects which
   tree a command sees, which keeps absolute paths baked into builds valid for any build
   system. CI runs the profile's smoke tests and Craft's tests on every core type, re-running
   failures on the baseline to tell regressions from pre-existing failures. It then runs
   Craft's benchmark (target), its per-variant coverage benchmarks, and short profile
   workloads (controls); a slowdown in any of them rejects the patch. Paired rounds in
   alternating order, a pinned core, the clock recorded, and a bootstrap 95% confidence
   interval. Patch-caused failures go back to Craft. Connection drops, timeouts and setup
   errors do not.
**Upstream checks** (`okriscv/upstream.py`, `agents/upstream-check.md`) run twice: after Trace
(before money is spent on Craft) and after CI (before the PR). Both look at upstream commits
since the measured revision that touch the relevant files, and at PRs and issues found
through the GitHub search API. An agent judges each item as fixes-it / duplicate / related /
unrelated. `fixed_upstream`, `duplicate` and `incomplete` (a search failed) stop the issue;
related work is linked in the PR. Before the PR, if upstream changed the patched files since
CI ran, the patch is rebased onto current upstream and CI runs again.

5. **PR** (`okriscv/pr.py`) has an AI agent draft the title, commit message and description from
   CI-measured numbers. `pr approve` records a human reviewer. Only then does `pr publish`
   open a **draft** PR from your fork, or produce a `format-patch` for projects that take
   patches by email.

`okriscv-agent log RECORD` writes `RECORD/LOG.md`: how the issue was found, why it is
slow, how it was fixed, and how the fix was verified.

## Usage

Requirements on this machine: Python 3.11+, git, the Claude Code CLI (`claude`), and `gh` for
GitHub projects. On the board: SSH key login, a native compiler toolchain and `perf`.

Describe your board first (board files other than templates are kept out of git):

```sh
cp boards/k3.example.toml boards/k3.toml   # then set host, port and user
```

```sh
bin/okriscv-agent onboard https://github.com/opencv/opencv.git --board boards/k3.toml
bin/okriscv-agent run     --project opencv --board boards/k3.toml     # collect → … → PR drafts
# models: agents/models.toml per agent; per run --model M (all agents) or --model-for craft=M
bin/okriscv-agent status
bin/okriscv-agent log     workspace/records/opencv/<workload>@<core>/<issue>

# individual stages (record commands infer the project from the record)
bin/okriscv-agent collect --project opencv --board boards/k3.toml [--workloads a,b] [--cores x100]
bin/okriscv-agent trace   RECORD...
bin/okriscv-agent craft   RECORD... --board boards/k3.toml
bin/okriscv-agent ci      RECORD... --board boards/k3.toml [--reuse-tests]
bin/okriscv-agent pr prepare RECORD
bin/okriscv-agent pr approve RECORD --reviewer "<name>"
bin/okriscv-agent pr publish RECORD --yes
```

Requirements:
- Python 3.11+ and the `claude` CLI (agents run headless through it).
- Key-based SSH to the board, with `perf`, a compiler and `git` installed there.
- `gh` (only for `pr publish` to GitHub).

## Repository layout

| path | what it is |
|---|---|
| `bin/okriscv-agent` | The only entry point (runs `okriscv/cli.py`). |
| `okriscv/` | Harness code, in Python with no third-party dependencies. |
| `okriscv/cli.py` | Command line: one subcommand per stage, plus `run`, `status` and `log`. |
| `okriscv/onboard/` | Stage 0. `session.py` holds the repository-reading tools, the board shell, and the log of every board command. `run.py` is the agent loop. `mcp_server.py` exposes the tools. |
| `okriscv/profile.py` | The project profile format and its validator (every command must have succeeded on the board). |
| `okriscv/collect.py` | Stage 1: windowed profiling per workload and core type, plus record writing. |
| `okriscv/stacks.py` | Groups perf call stacks into issues (runs on the board). |
| `okriscv/trees.py` | Baseline and candidate source/build trees on the board, switched by a symlink. Used by collect, Craft's build check and CI. |
| `okriscv/trace/` | Stage 2. `session.py` holds the evidence tools and their hashed registry. `rvv.py` decodes raw RVV instructions. `validate.py` and `assemble.py` hold the rules and the blueprint builder. `semantic.py` is the independent judge. `run.py` is the agent loop. `mcp_server.py` exposes the tools. |
| `okriscv/craft/` | Stage 3. `session.py` handles the worktree, edit tools, board build check, and candidate plus validation plan. `run.py` holds the craft/review/revise loop and the repair entry point. `mcp_server.py` exposes the tools. |
| `okriscv/ci/` | Stage 4. `pipeline.py` handles build, tests, benchmarks and report. `stats.py` does the paired statistics. `executor.py` runs commands locally or over SSH. `run.py` holds the CI loop with repair hand-off. |
| `okriscv/upstream.py` | Upstream checks: staleness and rebase before the PR, and fixed/duplicate/related detection against upstream commits, PRs and issues. |
| `okriscv/pr.py` | Stage 5: draft, approval gate, publish (refused without a passing pre-PR upstream check). |
| `okriscv/report.py` | Builds the per-issue `LOG.md`. |
| `okriscv/agent.py`, `okriscv/mcp.py` | Run an agent headless via `claude -p`, and a minimal MCP server for agent tools. |
| `okriscv/cores.py`, `okriscv/util.py` | Placement on heterogeneous cores, the board lock, hashing and JSON helpers. |
| `agents/` | System prompts for the seven agents: Onboard, Trace, its semantic judge, Craft, its reviewer, the upstream checker, and the PR writer; `models.toml` picks each agent's model. |
| `knowledge/guide-performance-optimization/` | Project-independent RISC-V optimization knowledge that Trace consults: routing rules (`triggers/`), confidence rules and core profiles (`references/`), and 32 optimization patterns (`patterns/`). |
| `boards/` | One file per machine: SSH address and core types (e.g. K3: X100 on CPUs 0–7, A100 on CPUs 8–15). `k3.example.toml` is the template for a SpacemiT K3; your own copies are not tracked by git. `local.toml` runs everything on this machine, for testing the harness itself. |
| `docs/stages.md` | Each stage explained step by step, with examples. |
| `docs/followups.md` | Agreed changes to the harness, with the evidence behind each, and their status. |
| `tests/` | Unit tests (validator rules, CI pipeline on a toy project, statistics) and the RVV decoder round-trip against GNU objdump. |
| `workspace/` | **Generated.** Profiles, records and PR status are versioned; clones, snapshots and archives are not. See below. |

### `workspace/` (generated)

Versioned: `projects/`, `records/`, `prs/`. Ignored: `src/`, `snapshots/` (re-creatable), `archive/`,
`local-board/`, lock files. Boards appear in them only by name (`[board] name` in the board file),
never by address.

| path | contents |
|---|---|
| `workspace/projects/<name>/` | `profile.json` written by onboard, plus `onboarding/` (session, every board command run, agent transcript). |
| `workspace/src/<name>` | Blobless clones of upstream repositories. |
| `workspace/snapshots/<name>-<rev>` | Read-only checkouts of the exact revision measured (Trace reads source here). |
| `workspace/records/` | **The results.** One folder per issue. |
| `workspace/prs/` | Status of the PRs submitted upstream (by hand): `INDEX.md` and one file per PR. |

### A record: `workspace/records/<project>/<workload>@<core>/<issue>/`

| sub-folder | written by | contents |
|---|---|---|
| `evidence/` | collect | `perf-stat.txt`, `annotate.txt`, `issue.txt` (call-stack grouping: functions, library calls, callers), `workload-output.txt`, `metadata.json` (machine, revision, command) |
| `trace/` | Trace | `diagnosis.md`, `diaglog.md` (the seven stages), `blueprint.json`, `claim-to-evidence.json`, validation results, checksums |
| `craft/` | Craft | `patch-plan.json`, `patch.diff`, `patch-candidate.json` (with the validation plan), `reviews/` |
| `ci/` | CI | `ci-report.json`, `performance-comparison.md`, `logs/` |
| `upstream-check-*.json` | upstream check | Result of the pre-craft and pre-PR checks: upstream commits, PR/issue candidates, verdict per item |
| `pr/` | PR | `pr.md` (`# title`, then the description), `commit-message.txt`, `status.json` |
| `LOG.md` | log | The whole story of the issue in one file. |

## Status

- Before the onboarding redesign, the pipeline ran end to end on a SpacemiT K3 against
  oneDNN, using a hand-written description of oneDNN. It produced three patches; two passed
  CI with measured gains (128× and +6–7% on their target benchmarks).
- Since the redesign, nothing is project-specific. OpenCV (on the K3) and lz4 (locally) are
  being onboarded from their URLs alone, as the first test of the generic pipeline.

## Limitations

- Call-stack grouping needs frame pointers. Onboard asks for them; without them, grouping
  falls back to individual functions.
- JIT-generated code can be annotated only when the project emits jitdump data. Otherwise
  JIT kernels appear by name but cannot be disassembled.
- Onboarding, collect and CI each build the project on the board; large projects take
  hours the first time per revision.

## Tests

```sh
python3 -m unittest discover -s tests   # validator rules, CI on a toy project, statistics
python3 tests/rvv_roundtrip.py          # RVV decoder vs GNU objdump
```