# gpt05b **Repository Path**: az13js/gpt05b ## Basic Information - **Project Name**: gpt05b - **Description**: No description available - **Primary Language**: Unknown - **License**: MIT - **Default Branch**: main - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 0 - **Forks**: 0 - **Created**: 2026-10-31 - **Last Updated**: 2026-11-09 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README # gpt05b — a from-scratch C++17 inference engine for a 0.5B GPT with extreme low-bit quantization `gpt05b` implements a **494M-parameter Qwen2.5-0.5B architecture** — 24 decoder layers, 896-wide hidden state, grouped-query attention (14 query / 2 KV heads), SwiGLU MLP, RoPE, RMSNorm and tied embeddings — entirely in C++17, with no BLAS, no PyTorch and no third-party libraries. On top of it sits a quantization stack (4.1, 2.35 and 1.35 bits per weight in the superblock formats) and a set of AVX-512 / AVX-512-VNNI kernels tuned for single-stream latency and batched throughput. > **Environment note.** The benchmark tables are measured with a > **deterministic synthetic checkpoint** of the exact target architecture > (same shapes, same parameter count, same compute graph), because the full > f32/safetensors files are not part of the repository. The quantized > **Qwen2.5-0.5B-Instruct** checkpoints and their `tokenizer.json` are included > under `models/` (Git LFS), and are validated end to end: identical greedy > token ids and matching perplexity against HF Transformers, plus quantization > and throughput numbers — see [`docs/REAL_WEIGHTS.md`](docs/REAL_WEIGHTS.md). > Unless stated otherwise, numbers below were measured on the AVX-512 Xeon host; > the AVX2 table in “CPUs without AVX-512” comes from a separate i7-7700 host. ## Highlights | | | |---|---| | Model | Qwen2.5-0.5B architecture, **494,032,768 parameters** | | Weights | 32-value blocks and 64/128-value superblocks: F32, F16, Q8, Q4, Q2, ternary, binary | | Best decode | **65–76 tok/s** single stream, 4 vCPU (4.10 bpw superblock to 1.35 bpw binary) | | Extreme low-bit | **52–54 tok/s** at 2.35 bpw, **68–76 tok/s** at 1.35 bpw | | Batched decode | **193 tok/s** at batch 16 (binary), 182 tok/s at batch 48 (Q4_8) | | Prefix cache | 480-token prompt prefill **2476 ms → 22.6 ms** on repeat | | Prefill | **~230–237 tok/s** (short prompts); ~95–99 at 4k, ~61 at 8k, ~37 at 16k (128-token default tile) | | Quality tiers | Q8: 4.5e-4 · Q4_8+Q8: 0.023–0.040 · Q4: 0.060–0.068 · ternary: 0.227 · binary: 0.292 | | Accuracy | FP32 path matches scalar reference to **4e-6** relative L2; prefill vs decode ≤2.9 % on the AVX-512/VNNI build (3–4 % on AVX2, where decode uses the int8 emulation) | | Memory | 241 MiB (Q4) / 501 MiB (Q8) / 79 MiB (binary), KV cache optional | | Dependencies | C++17 compiler; AVX-512/VNNI for the fast kernels, with an AVX2 or scalar portable fallback | The engine is deterministic, mmap-loads checkpoints, and has an adaptive thread pool: workers spin for ~1–3 ms between the ~150 small barriers of a decoded token and block on a condition variable when idle, so the HTTP server uses **0 % CPU between requests** (the previous pure-spin pool burned 3 cores). ![Throughput by format](docs/perf.svg) ![Quantization fidelity](docs/quality.svg) ![Batch scaling](docs/batch.svg) ![Quantization Pareto](docs/pareto.svg) ## Build and quick start The build is CMake-based (GCC or Clang on Linux; C++17 plus `-march=native` by default): ```bash cmake -S . -B build -DCMAKE_BUILD_TYPE=Release cmake --build build -j ctest --test-dir build --output-on-failure # unit + model/tokenizer/server tests ``` Build options: `-DGPT05B_BUILD_TESTS=OFF`, `-DGPT05B_BUILD_TRAINER=OFF`, and `-DGPT05B_ARCH=avx2|avx512|scalar` (default `native`). The resulting binary is `build/gpt05b`, the same path the test and benchmark scripts expect. # Generate the synthetic 0.5B checkpoint (deterministic: seed + spec -> bytes) mkdir -p models ./build/gpt05b gen --out models/q4.bin --quant q4_0 --qgroup 128 # ...or the full inventory required by run_all/validate_all/final_check # (20 checkpoints, about 7 GB free in models/; see GEN_ALL_MIN_GB) ./bench/gen_all.sh ./build/gpt05b info --model models/q4.bin ./build/gpt05b bench --model models/q4.bin --threads 4 --ctx 1024 \ --prefill 64 --decode 64 ./build/gpt05b ref-check --model models/q4.bin --n 2 --threads 4 ./build/gpt05b bench --model models/q4.bin --kv f16 --ctx 4096 ... ./build/gpt05b bench-batch --model models/q4.bin --batch 8 --threads 4 --ctx 1024 ./build/gpt05b batch-check --model models/q4.bin --batch 8 --threads 4 ./build/gpt05b serve --model models/q4.bin --port 8080 --threads 4 ./bench/run_pareto.sh # fidelity/latency sweep over all checkpoints ./bench/validate_all.sh # reference + batch checks for every format ./bench/final_check.sh # build + tests + validation (demo eval when trained model exists) ``` ### CPUs without AVX-512 The kernel set is selected at compile time from `-march=native`. See [`docs/PORTABILITY.md`](docs/PORTABILITY.md) for the tested build matrix, fallback performance and the no-runtime-dispatch caveat. On an AVX-512 host the engine runs the 512-bit VNNI kernels used for the numbers in this README. On AVX2 the same sources compile an AVX2 path: * decode (`T=1`) uses an **int8 kernel emulated with `vpmaddwd`**: activations are quantized exactly like the VNNI path (per 32/64/128 weights, with the same scale and offset-correction terms), and weight codes are multiplied as 16-bit integers. `matmul()` selects this path for Q4_0, Q8_0, Q4_8 and Q4_8G64 when `--qgroup`/the checkpoint layout is 32, 64 or 128. * single-token fallbacks use a 4-rows-per-pass AVX2 GEMV that unpacks each weight block in registers instead of writing an fp32 row buffer. * `dot_f32`, `dequantize_row` (Q4/Q8/Q2/TQ/B1 blocks) and the Q4/Q2/B1 superblock dequantizers have AVX2 implementations. Machines without AVX2 at all still build and run a scalar path. Results are numerically equivalent for the fp32 path and use the same int8 activation approximation as the VNNI kernels for decode; `ctest`, `ref-check` and `batch-check` pass on AVX-512, AVX2 and scalar builds. `bench` reports the active kernel set (`avx512f+vnni`, `avx2` or `scalar`). An AVX-512 CPU **without VNNI** (e.g. Skylake-SP) also builds: the 512-bit kernels use `vpdpbusd`, so such a build selects the AVX2 path instead of failing. Binaries are host-specific when built with `-march=native`; build on the target machine or pass an explicit `-DGPT05B_ARCH=...` when the binary will run elsewhere. Measured on an i7-7700 host (4 threads, 0.5B synthetic checkpoint, AVX2-only, GCC 9), decode/prefill in tokens per second: | format | decode | prefill | |---|---|---| | q4_0 (qgroup 128) | 61–66 | 89–99 | | q4_8 | 68–72 | 91–105 | | q8_0 | 29–36 | 61–75 | | q2_8 | 23–29 | 81–89 | | b1_8 | 39–44 | 106–119 | The host is shared, so individual runs vary by up to ~30 %; compare runs from the same session. The AVX-512 numbers in this README were measured on different hardware and remain the fast-path reference. ### A from-scratch trained demo model `tools/train_gpt.cpp` trains a 1 M-parameter byte-level Qwen-style transformer (4×128, GQA 4/2, SwiGLU 512) on the repository's own text, so the engine and quantizers can be exercised on genuinely trained weights: ```bash cmake --build build --target train_gpt ./build/train_gpt --gradcheck ./build/train_gpt --steps 8000 --out models/trained_128_f32.bin ./build/train_gpt --width 256 --layers 4 --steps 4000 \ --out models/trained_256_f32.bin # wider demo model ./build/gpt05b run --model models/trained_128_f32.bin --prompt "static int" \ --max-tokens 80 --temperature 0.7 ./build/gpt05b ppl --model models/trained_128_f32.bin --file some_text.txt ./build/gpt05b ppl --model models/qwen05b_q48.bin --file some_text.txt \ --tokenizer models/tokenizer.json # real-tokenizer perplexity ``` Best validation loss is **1.83**; the f32 model scores **PPL 8.16** on a 1.2 KB held-out C program, q8 **8.03**, q4_0 **8.36**, q4_8 **9.08**, 2-bit **32**, binary **201**. A wider d=256 model (val 2.32, stopped early) is used for a width study: its q4_8 KL drops from 0.410 to **0.120** because each row now has two 128-wide scales instead of one. A larger 8.4 MB mixed corpus was also tried and did **not** help (held-out PPL 15.26 vs 8.16), so it was discarded. See [`docs/TRAINING.md`](docs/TRAINING.md) for the schedule, gradient checks, samples and the full quantization table. ### Converting real weights `gpt05b import` reads HF safetensors directly (F32/F16/BF16, no Python or numpy needed), maps the Qwen2 tensor names, validates every shape and writes an f32 checkpoint. `--hf` may point to a file or to a directory of `*.safetensors` shards. It was validated at full 0.5B scale: a synthetic 1.98 GB safetensors file imports to a payload identical (SHA-256) to the original checkpoint, and a subsequent `requant --quant q4_8` is byte-identical to the directly generated model. Then use the engine's own quantizers to produce any format (including the superblock types, byte-identical to what `gen` writes): ```bash ./build/gpt05b import --hf /path/to/model.safetensors \ --out models/qwen05b_f32.bin ./build/gpt05b requant --in models/qwen05b_f32.bin \ --out models/qwen05b_q48.bin --quant q4_8 --qgroup 128 ``` `tools/convert_hf.py` is the Python alternative (needs numpy) and supports the same graph; `import` is preferred in this environment. The converter is also validated with the real **Qwen2.5-0.5B-Instruct** files: `import` produces a 1.88 GiB f32 checkpoint whose greedy generations match HF Transformers token for token (f32) and whose perplexity is within 1.2 % of the HF value; `tokenizer.json` encoding matches the HF tokenizer exactly on the model README (1321 tokens) and on numbers, code, URLs and contractions. On real weights the group size dominates the bit count: q4_0 with group 128 loses much more perplexity than group 32, so use `--qgroup 32` (or `q8_0`) when accuracy matters and `--qgroup 64|128` when decode speed does. Measurements, commands and caveats are in [`docs/REAL_WEIGHTS.md`](docs/REAL_WEIGHTS.md). Two prebuilt checkpoints (q4_8 + 8-bit attention, and q4_0 g32, ~529 MiB in total; both quantized with the committed importance matrix) are stored under `models/` with Git LFS; see that page for the download commands. `requant` also works on quantized sources (e.g. re-quantize `q8` to `b1_8`). ### Importance-weighted quantization Per-block scales can only do so much with the outlier channels of real LLM weights. `gpt05b calib` runs the f32 checkpoint over calibration text and records, for every quantized matrix, the RMS of the activations that multiply each input channel. `requant --imatrix` then minimises a weighted reconstruction error in which those channels count more, and searches 19 candidate scales per group instead of using plain max-abs. The on-disk format does not change, so weighted checkpoints run on exactly the same kernels and at the same speed: ```bash ./build/gpt05b calib --model models/qwen_real_f32.bin \ --file bench/texts/calib_en.txt --file bench/texts/calib_zh.txt \ --tokenizer models/tokenizer.json --out models/imatrix_qwen2.5-0.5b-instruct.bin ./build/gpt05b requant --in models/qwen_real_f32.bin \ --out models/qwen_real_q4g32.bin --quant q4_0 --qgroup 32 \ --imatrix models/imatrix_qwen2.5-0.5b-instruct.bin ``` On Qwen2.5-0.5B the weighting lowers held-out perplexity from 39.81 to **34.47** (q4_0 group 32) and from 48.22 to **39.86** (q4_8), at identical file size and speed. The calibration text is two documents that share no text with the held-out evaluation excerpts (`bench/texts/`); a prebuilt matrix for this model is committed under `models/`. Details and the full accuracy/speed table: [`docs/REAL_WEIGHTS.md`](docs/REAL_WEIGHTS.md). `gen` options: `--preset qwen2-0.5b|small|one|tiny`, `--quant`, `--quant-emb`, `--quant-attn`, `--quant-mlp`, `--quant-wqkv`, `--quant-wo`, `--quant-wgateup`, `--quant-wdown`, `--qgroup 32|64|128`, `--nctx`, `--seed`, `--threads`. The per-matrix overrides let one keep `q4_8` for most weights while promoting, say, only the down projection (`--quant-wdown q8_0`). For Q4 the recommended fast checkpoint is `--quant q4_8 --qgroup 128` (4.10 bpw, highest speed at the same fidelity); `--qgroup 64` is the best 4.5 bpw option, and for Q8 `--qgroup 128` is faster. Per-tensor types make mixed precision easy. The cheapest good mix keeps the 4-bit superblock MLP and embeddings and promotes only attention to 8 bits (**4.49 bpw, 264 MiB, KL 0.040**, ~67 tok/s); promoting embeddings too gives KL 0.0246 at 5.70 bpw. Both dominate the plain-Q4 mixed variant on size and speed: ```bash ./build/gpt05b gen --quant q4_8 --quant-attn q8_0 --quant-emb q8_0 \ --qgroup 128 --out models/mixed.bin ``` Producing the full matrix of checkpoints takes ~2 minutes and ~6 GiB of disk. ## Architecture The tensor graph follows Qwen2.5-0.5B exactly, so a converted official checkpoint runs without code changes: ``` tok_emb [151936, 896] (tied with lm_head) per layer (24): attn_norm RMSNorm(896) eps 1e-6 wqkv [1152, 896] + bias[1152] q:896 k:128 v:128 RoPE head_dim 64, theta 1e6 attention causal, GQA 14 query / 2 KV heads wo [896, 896] ffn_norm RMSNorm(896) wgateup [9728, 896] gate:4864 up:4864 wdown [896, 4864] output_norm RMSNorm(896) ``` Parameter count: `151936*896 + 24*(2*896 + 1152*896 + 1152 + 896² + 2*4864*896 + 896*4864) + 896 = 494,032,768`. ## Quantization Every matrix is stored as 32-value blocks. A block carries an fp16 scale and its packed codes; the fast tier optionally shares one scale per 128 values (`--qgroup 128`), which lets a 512-bit VNNI instruction cover a whole group with a single float scale. `--qgroup 64` is the middle ground: better fidelity than 128 (two blocks per scale) at similar decode speed, slightly slower prompt processing. `q4_8` is a true 4-bit format: 128-value superblocks with one fp32 row base, one 8-bit sub-scale per superblock and 64 B of nibbles, for **4.10 bits per weight**. It is 9 % smaller than `q4_0` and the fastest Q4 kernel because the superblock scale fits in a single byte and the row layout stays contiguous. `q4_8g64` applies the same packing to 64-value superblocks (4.16 bpw), which recovers the group-64 fidelity at 8 % less size than `q4_0` and the same decode speed: ```bash ./build/gpt05b gen --quant q4_8 --qgroup 128 --out models/q4s.bin ``` | type | bits/weight | layout | 4-bit codes | |---|---|---|---| | `f32` | 32.0 | reference | — | | `f16` | 16.0 | — | — | | `q8_0` | 8.5 | fp16 scale + 32×int8 | 8-bit symmetric, max/127 | | `q4_0` | 4.5 | fp16 scale + 16 B | int4 −8..7, max/7 | | `q4_8` | 4.10 | fp32 row base + u8 scale + 64 B / 128 | int4 −8..7, superblock scale | | `q4_8g64` | 4.16 | fp32 row base + u8 scale + 32 B / 64 | same, finer scales | | `q2_8` | 2.35 | fp32 row base + u8 scale + 32 B / 128 | 4 levels, Gaussian-tuned | | `tq2_8` | 2.35 | fp32 row base + u8 scale + 32 B / 128 | ternary, mean-abs scale | | `b1_8` | 1.35 | fp32 row base + u8 scale + 16 B / 128 | binary sign | | `q2_0` | 2.5 | fp16 scale + 8 B | 4 levels, Gaussian-tuned + 2×LSQ | | `tq2_0` | 2.5 | fp16 scale + 8 B | ternary {−1,0,1}, LSQ scale | | `b1_0` | 1.5 | fp16 scale + 4 B | binary sign, mean-abs scale | The 2-bit quantizer uses a Gaussian-optimal initial scale (`0.98·RMS`) and two least-squares scale refinements, which cuts reconstruction error by 28 % versus naive max-scaling (0.455 → 0.329 relative on N(0,1) blocks). Measured weight-level relative error: Q4 0.097 (block-32), 0.108 (group-64), 0.118 (group-128 and superblock); Q8 0.005–0.007; Q2 0.343; ternary 0.511; binary 0.601. See `docs/RESULTS.md` for the full table. ### Measured fidelity (synthetic checkpoint, 12 teacher-forced tokens) `KL(f32 ‖ quant)` is the per-token KL divergence from the FP32 model's next-token distribution; relative logit RMSE and top-1 agreement are also shown. The synthetic checkpoint has no language structure, so top-1 agreement is a harsh metric — the KL column is the meaningful one. | quant | bpw | KL (nats/token) | logit RMSE (rel) | top-1 | |---|---|---|---|---| | q8_0 block-32 | 8.5 | 0.00031 | 0.042 | 83.3 % | | q8_0 group-64 | 8.5 | 0.00037 | 0.045 | 91.7 % | | q8_0 group-128 | 8.5 | 0.00045 | 0.050 | 75 % | | q4_0 block-32 | 4.5 | 0.050 | 0.53 | 0 % | | q4_0 group-64 | 4.5 | 0.060 | 0.58 | 0 % | | q4_0 group-128 | 4.5 | 0.067 | 0.61 | 0 % | | q4_8 superblock (4.10 bpw) | 4.10 | 0.068 | 0.62 | 8.3 % | | q4_8g64 superblock (4.16 bpw) | 4.16 | 0.060 | 0.58 | 0 % | | q2_8 superblock (2.35 bpw) | 2.35 | 0.250 | 1.18 | 0 % | | tq2_8 superblock (2.35 bpw) | 2.35 | 0.226 | 1.12 | 0 % | | b1_8 superblock (1.35 bpw) | 1.35 | 0.292 | 1.28 | 0 % | | q2_0 block-32 | 2.5 | 0.251 | 1.18 | 0 % | | q2_0 group-128 | 2.5 | 0.251 | 1.18 | 0 % | | tq2_0 block-32 (ternary) | 2.5 | 0.272 | 1.23 | 0 % | | tq2_0 group-128 | 2.5 | **0.227** | 1.12 | 0 % | | b1_0 group-128 (binary) | 1.5 | 0.292 | 1.28 | 0 % | | q4 MLP + q8 attn/emb | 5.96 | **0.024** | 0.37 | 0 % | (Q4 top-1 is 0 % only because this random model's logits are nearly uniform; the KL of 0.05 nats corresponds to a perplexity ratio of 1.05.) ## Performance Unless noted otherwise these are AVX-512/VNNI measurements; the portable fallbacks are covered in [`docs/PORTABILITY.md`](docs/PORTABILITY.md). Measured with `--threads 4` (the container's CPU quota is 4 vCPU even though 64 logical CPUs are visible), 1024-token KV cache, 64-token decode loop. `GiB/s` is checkpoint bytes × tokens/s, i.e. the effective weight-read bandwidth. | checkpoint | size | prefill tok/s | decode ms/tok | decode tok/s | eff. GiB/s | |---|---|---|---|---|---| | f32 | 1885 MiB | 93 | 71.7 | 13.9 | 27.6 | | q8 block-32 | 501 MiB | 128 | 24.6 | 40.6 | 21.3 | | q8 group-128 | 501 MiB | 220–233 | 17.6–20.1 | 50–57 | 26–30 | | q4 block-32 | 266 MiB | 125 | 21.8 | 45.9 | 12.8 | | q4 group-64 | 266 MiB | 175 | 18.6 | 53.7 | 14.8 | | q4 group-128 | 266 MiB | 209 | 18.2 | 55.1 | 15.3 | | q4 superblock (4.10 bpw) | 241 MiB | 229–237 | 14.6–15.3 | **65–69** | 16.6–17.4 | | q4 superblock 64 (4.16 bpw) | 245 MiB | 143–175 | 17.1–18.6 | 54–58 | 13.8–15.3 | | q2 group-128 | 148 MiB | 189 | 21.7 | 46.1 | 7.1 | | tq2 group-128 | 148 MiB | 206 | 19.5 | 51.2 | 7.9 | | b1 group-128 | 89 MiB | 220 | 15.8 | 63.5 | 5.9 | | q2_8 superblock (2.35 bpw) | 138 MiB | 224 | 18.9 | 52.6 | 7.6 | | tq2_8 superblock (2.35 bpw) | 138 MiB | 233 | 18.6 | 53.7 | 7.9 | | b1_8 superblock (1.35 bpw) | 79 MiB | 208–231 | 13.2–14.6 | **68–76** | 5.7–6.3 | | q4_8 + q8 attn (4.49 bpw) | 264 MiB | 230–238 | 15.0–16.7 | 60–67 | 16.6–17.2 | | q4_8 MLP + q8 attn/emb (5.70 bpw) | 336 MiB | 214–220 | 16.6–18.2 | 55–60 | 19.5–21.2 | | q4_8g64 MLP + q8 attn/emb (5.74 bpw) | 338 MiB | 164–169 | 18.3–18.9 | 53–55 | 19.0–19.4 | Run-to-run variance on this shared VM is ±10 %; the Pareto sweep below is one contiguous run, so its absolute numbers differ slightly from this table. Prompt processing is done in 128-token tiles (`Engine::prefill`, overridable with `GPT_PREFILL_TILE`), fed to the batched VNNI kernels; that lifted 64-token prefill from 80 to ~230 tok/s depending on the weight format, and larger tiles also help 512–8192-token prompts. Prefill attention is flash-style blocked (32 queries per KV pass), and the cache can be FP16 (`--kv f16`) for long contexts. Decode at 4 vCPU is compute-bound on the unpack/dequantization path for Q4/Q8 (at 18.2 ms per token Q4 reads 266 MiB, i.e. 15.3 GiB/s). The 2-bit and binary formats stream less data and their gather-based unpackers are cheaper per value, which is why they match or beat Q4 despite the lower precision. The block-32 versions of Q2/TQ still use the scalar dequantizer; only the group-128 versions have VNNI kernels. Thread scaling (Q4 group-128, decode): | threads | tok/s | |---|---| | 1 | 15.7 | | 2 | 28.0 | | 4 | 50.4 | | 8 | 40.8 (CPU quota is 4; spinning threads thrash) | Batch decode (Q4 group-128, 4 threads) amortizes weight reads across sequences: | batch | ms/step | tok/s | ms/token | |---|---|---|---| | 1 | 19.6 | 51.1 | 19.6 | | 2 | 25.8 | 77.6 | 12.9 | | 4 | 33.5 | 119.4 | 8.4 | | 8 | 58.6 | 136.5 | 7.3 | | 16 | 92.9 | 172.2 | 5.8 | | 32 | 184.8 | 173.2 | 5.8 | | 48 | 290.3 | 165.3 | 6.0 | ## Correctness `ref-check` runs three independent computations on the same checkpoint: 1. the scalar, single-threaded, materialized reference (`src/reference.cpp`); 2. the engine with the FP32 dequantizing kernels; 3. the engine with int8 VNNI kernels. ``` fp32 path: rel L2 4.03e-06 max|diff| 1.03e-05 top1 match vnni path: rel L2 3.87e-02 max|diff| 9.95e-02 top1 mismatch* ``` The FP32 path is bit-level equivalent up to summation order. The VNNI path quantizes activations to int8 per 32-value block, so a ~4 % logit deviation is expected; its *distribution* matches the FP32-activation path (KL 0.0491 vs 0.0490 for the Q4 checkpoint). *Top-1 flips only because the synthetic model's logits are nearly tied. `batch-check` additionally verifies that batched decode equals single-stream decode to the same tolerance and that both match the reference. ## Repository layout ``` src/ quant.{h,cpp} block formats, scalar quantize/dequantize kernels.{h,cpp} AVX-512/VNNI kernels, AVX2 + scalar fallback, matmul scheduler threadpool.h adaptive spin/sleep persistent thread pool modelfile.{h,cpp} checkpoint format, mmap loader, synthetic generator safetensors.{h,cpp} HF safetensors reader (F32/F16/BF16, shards) engine.{h,cpp} transformer forward, KV cache, batch decode, sampling reference.{h,cpp} naive FP32 reference implementation tokenizer.{h,cpp} byte-level BPE (HF tokenizer.json) + byte fallback server.{h,cpp} HTTP inference server (no dependencies) main.cpp CLI: gen/import/requant/info/bench/run/ref-check/ppl/... tests/ unit tests, model/reference checks, tokenizer, errors, server tools/ converters, from-scratch trainer, chart generators bench/ run_all / validate_all / final_check / eval_trained scripts docs/DESIGN.md kernel and memory-layout details docs/RESULTS.md measurement methodology and raw tables docs/TRAINING.md from-scratch demo model and real-weight quantization docs/REPORT.md consolidated engineering report docs/PORTABILITY.md AVX2/scalar fallback matrix, caveats and benchmarks docs/REAL_WEIGHTS.md real Qwen2.5-0.5B-Instruct import and HF verification bench/gen_all.sh generate the benchmark checkpoint inventory ``` ## HTTP server A dependency-free single-threaded server exposes the engine over HTTP: ```bash ./build/gpt05b serve --model models/q4.bin --tokenizer models/tokenizer.json \ --port 8080 --threads 4 --ctx 2048 curl -s localhost:8080/health curl -s -X POST localhost:8080/v1/completions \ -d '{"prompt":"Once upon a time","max_tokens":32,"temperature":0.7,"top_p":0.9}' ``` `/v1/completions` accepts `prompt` or `prompt_ids`, `max_tokens`, `temperature`, `top_p`, `top_k`, `repeat_penalty`, `seed` and `stop`, and returns `text`, `tokens`, `prompt_tokens`, `completion_tokens`, `ms_per_token` and `stop_reason`. Add `"stream": true` for Server-Sent Events: `data: {"token":123,"text":"..."}` per token, a final JSON event and `data: [DONE]`. The server keeps the previous request's KV cache and reuses the longest common token prefix (`cached_tokens` in the response), which makes multi-turn chat and repeated system prompts much cheaper. Requests are handled sequentially (one engine instance); bind to `--host 0.0.0.0` to serve other machines. Without `--tokenizer` the byte fallback tokenizer is used. `GET /v1/models` returns a one-entry model list. `POST /v1/chat/completions` accepts the OpenAI-style `messages` array (`system`/`user`/`assistant`), wraps it in the Qwen `<|im_start|>` template, and returns a `chat.completion` object; generation stops on `<|im_end|>` or `max_tokens`. Streaming is not implemented for chat (use `/v1/completions` with `"stream": true`). `POST /v1/batch` takes `{"prompts": ["...", ...], "max_tokens": 16, ...}` and returns one result per prompt. Prompts may have different lengths (they are prefilled through the fast single-sequence path and copied into per-slot KV caches), while the decode steps run through the batched VNNI kernels. The batch is limited by `--max-batch` (default 8); on q4_8 an 8×96-token batch decodes at ~124 tok/s aggregate (vs 76 tok/s single-stream). ## Documentation * [`docs/DESIGN.md`](docs/DESIGN.md) — checkpoint format, kernel design, blocking strategy, threading, validation methodology. * [`docs/RESULTS.md`](docs/RESULTS.md) — hardware, checkpoint inventory, raw benchmark and fidelity tables, reproduction commands. * [`docs/TRAINING.md`](docs/TRAINING.md) — from-scratch demo trainer, schedule, samples and real-weight quantizer numbers. * [`docs/REPORT.md`](docs/REPORT.md) — consolidated engineering report: formats, kernels, measured results, validation, negative results. * [`docs/PORTABILITY.md`](docs/PORTABILITY.md) — AVX2/scalar 回退的构建矩阵、 实测数据与限制。 * [`docs/REAL_WEIGHTS.md`](docs/REAL_WEIGHTS.md) — real Qwen2.5-0.5B-Instruct import, HF agreement, quantization and throughput numbers. * [`docs/SUMMARY.zh.md`](docs/SUMMARY.zh.md) — 中文摘要（实现、结果、复现、限制）。 ## Limitations * The benchmark tables use generated synthetic checkpoints. Real Qwen2.5-0.5B-Instruct weights are validated separately ([`docs/REAL_WEIGHTS.md`](docs/REAL_WEIGHTS.md)); on those, q4 quantization costs more accuracy than on the synthetic weights, and per-128 group scales are noticeably worse than per-32 ones. * AVX-512 (F/BW/DQ/VL + VNNI) is assumed for the measured fast paths. Builds without AVX-512 use the AVX2 kernels described above — close to the AVX-512 decode rates for Q4 formats, slower for Q8 and the 2-bit formats — while the numbers in this document were measured on AVX-512 hardware. * The block-32 variants of Q2/ternary/binary still fall back to the FP32 dequantizing path; their group-128 variants have VNNI kernels for both decode and prefill. * The KV cache can be FP32 or FP16 (`--kv f16`); int8 K/V is planned for very long contexts. * `tools/convert_hf.py` was written against the documented HF Qwen2 layout but could not be executed here (no network / numpy in this sandbox).