# Dinov3ForMBR

**Repository Path**: robertwangwang/Dinov3ForMBR

## Basic Information

- **Project Name**: Dinov3ForMBR
- **Description**: 开源1234567899999
- **Primary Language**: Unknown
- **License**: Apache-2.0
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2026-04-17
- **Last Updated**: 2026-10-12

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# DINOv3-MBR: Multi-Branch / Cross-Backbone Fusion for Vehicle Re-ID at Foundation-Model Scale

> A vehicle re-identification (Re-ID) codebase built around **DINOv3-pretrained backbones**, carrying three research/engineering lines at once:
> 1. **Dual-backbone MBR** (ConvNeXt + ViT-L, 4 branches concatenated to 8192-d) — the original deployment stack;
> 2. **Option A single-ConvNeXt-MBR** (single backbone + 4 heads) — the sealed SOTA-class model (VeRi-Wild 88.19 mAP);
> 3. **A diagnostic & falsification toolkit** (`embedding_explaing/`) — answering "does the *diversity premise* behind multi-branch / cross-backbone fusion still hold at foundation-model scale?"
>
> Target datasets: **VeRi-776** and **VeRi-Wild**. All evaluation uses the **official cross-camera protocol**.

---

## Table of Contents

- [Key Results](#key-results)
- [Architecture](#architecture)
- [Repository Layout](#repository-layout)
- [Environment & External Dependencies](#environment--external-dependencies)
- [Datasets](#datasets)
- [Training](#training)
- [Evaluation (PyTorch)](#evaluation-pytorch)
- [Deployment (ONNX → TensorRT)](#deployment-onnx--tensorrt)
- [Re-ranking](#re-ranking)
- [Analysis & Experiment Scripts (`embedding_explaing/`)](#analysis--experiment-scripts-embedding_explaing)
- [Model Modules (`models/`)](#model-modules-models)
- [Configuration (`config.py`)](#configuration-configpy)
- [Checkpoints](#checkpoints)
- [Documentation](#documentation)

---

## Key Results

| Model | Dataset (official cross-camera) | mAP | +re-ranking | Note |
|---|---|---|---|---|
| **Option A** (single-ConvNeXt-MBR) | VeRi-Wild test_3000 | **88.19** | **92.38** | Visual-only, on par with the strongest metadata-dependent multi-branch baseline |
| **Option A** | VeRi-Wild test_10000 (large gallery) | **77.47** | **83.68** | Larger gallery benefits more from re-ranking (+6.21) |
| Option A (cross-domain, wild→776 zero-shot) | VeRi-776 | 66.02 | 69.27 | R1 87.07 holds → learned generic discrimination |
| Dual-backbone DINOv3-MBR | VeRi-776 in-domain | **82.42** | — | R1 96.31, original deployment model |
| Single ViT (outside fusion topology) | VeRi-Wild test_3000 | 67.03 | — | Representational-gap control (no-cam 72.10) |

**Main findings**: same-backbone multi-head collapses into redundancy after convergence (zero concatenation gain, effective rank only ~512/8192, Jaccard *rises* 0.79→0.84 with training); cross-backbone (CNN×Transformer) fusion never exceeds the single backbone even when given every advantage; dual-backbone fusion costs ≈ **2.8× latency / 3× FLOPs** of the single backbone. See [`embedding_explaing/结论沉淀.md`](embedding_explaing/结论沉淀.md).

---

## Architecture

### Dual-backbone MBR (original, `models/dinov3_mbr.py`)

```
Input (B,3,256,256)
  ├─ ConvNeXt-Base (DINOv3 pretrained) → (B,1024,16,16)
  │     ├─ R50 Stage4 branch (cls)    → 2048 + logits
  │     └─ R50 Stage4 branch (metric) → 2048
  └─ ViT-L/16 (DINOv3, frozen 0-20 + LoRA 21-24) → (B,1024,16,16)
        ├─ BoT branch (cls)    → 2048 + logits
        └─ BoT branch (metric) → 2048
Final: each of 4 → L2 → concat → (B,8192)
```

### Option A single-ConvNeXt-MBR (sealed, `models/dinov3_convnext_mbr.py`)

Drops the ViT; 4 heads (2 ConvNeXt conv heads + 2 BoT attention heads) **all read the same ConvNeXt full-depth trunk**. The diversity axis is conv-vs-attention. Its interface matches `DINOv3MBR` exactly → reuses the entire `train_wild.py` training/eval code.

### Other architecture variants

| Model file | Description |
|---|---|
| `models/dinov3_vit_reid.py` | Single ViT-L + single BoT branch (2048-d), single-backbone control |
| `models/dinov3_mfu.py` | MFU cross-attention mutual fusion (adapted from FusionReID), replacing naive concat |
| `models/dinov3_mfu_fusion.py` | **v7** dual-backbone full FT + learnable fusion head → single vector (basis of the asymmetric frozen-anchor scheme) |
| `models/dinov3_htm.py` | Faithful reimplementation of the full FusionReID HTM (LRU + shared SEU + token-level MFU + stacking) |

---

## Repository Layout

```
Dinov3ForMBR/
├── config.py                    # All hyperparameters (nested dataclasses)
├── train.py                     # VeRi-776 two-phase DDP training
├── train_wild.py                # VeRi-Wild training (incl. Option A switches)
├── train_wild_singlevit.py      # Single-ViT training
├── train_wild_mfu_fusion.py     # v7 dual-backbone fusion / full HTM training
├── test.py / eval.py / eval_pth.py   # PyTorch evaluation
├── export_onnx.py / export_onnx_optionA.py       # ONNX export
├── build_engine.py / build_engine_optionA.py     # TensorRT engine build
├── eval_trt.py / eval_trt_wild.py                 # TRT evaluation
├── compare_backends.py          # PTH/ONNX/TRT accuracy + throughput comparison
├── analyze_similarity_trt.py    # Intra/inter-class cosine analysis on the TRT engine
├── models/                      # Backbones, branches, architectures, losses
├── data/                        # VeRi-776 / VeRi-Wild datasets + PK sampler
├── base_model_output/           # 776 dual-backbone deployment artifacts (pth/onnx/trt + standalone build/export scripts)
├── output_wild/                 # Option A sealed weights
├── veri_wild_baseline/          # Single-ViT / dual-backbone wild baseline weights
├── embedding_explaing/          # Diagnostic & experiment scripts + results + 结论沉淀.md
├── CLAUDE.md                    # Codebase navigation (for AI assistants)
├── planB.md                     # Architecture documentation (mermaid)
├── DINOv3-MBR_架构诊断与重构.md  # Option A/B diagnostic source document
└── 实验数据.md                   # Experiment data summary (single source of truth)
```

---

## Environment & External Dependencies

- **Python 3.10.12**; dependencies live in `.venv/` (no `requirements.txt`). Key packages: PyTorch (CUDA), torchvision, TensorRT 10.16, Pillow, NumPy, scipy.
- **DINOv3 repository**: must be cloned to `~/PycharmProjects/dinov3/` (loaded via `torch.hub.load(..., source='local')`).
  ```bash
  git clone https://github.com/facebookresearch/dinov3.git ~/PycharmProjects/dinov3
  ```
- **DINOv3 pretrained weights** (placed in `--weights_dir`, default `/home/robertwang/dinov3/`):
  - `dinov3_convnext_base_pretrain_lvd1689m-801f2ba9.pth`
  - `dinov3_vitl16_pretrain_lvd1689m-8aa4cbdd.pth`

```bash
source .venv/bin/activate
```

> ⚠️ Paths in `config.py` are hardcoded to `/home/robertwang/`. **Do not edit config directly**; override via CLI: `--data_root` / `--weights_dir` / `--output_dir` / `--repo_dir`.

---

## Datasets

### VeRi-776 (`--data_root /home/robertwang/veri776/`)
```
veri776/
├── image_train/ image_query/ image_test/
├── name_train.txt name_query.txt name_test.txt
└── ... (filename format XXXX_cYYY_TTTTTTTT.jpg; XXXX=vehicle ID, cYYY=camera)
```

### VeRi-Wild (`--data_root /home/robertwang/Documents/VeRI-Wild/`)
```
VeRI-Wild/
├── images/<vid>/<imgid>.jpg
└── train_test_split/
    ├── test_{3000,5000,10000}_id.txt        # gallery lists
    └── test_{3000,5000,10000}_id_query.txt   # query lists (disjoint from gallery)
```

Default PK sampling: 776 = P8×K8 (batch 64); wild = P16×K4 (batch 384, 6 GPUs).

---

## Training

Two-phase strategy (`train.py` / `train_wild.py`): **Phase 1** freezes ConvNeXt+R50, trains only BoT+ViT LoRA (DDP requires `find_unused_parameters=True`); **Phase 2** unfreezes everything, with warmup + staged LR decay.

```bash
# VeRi-776 dual-backbone (single / multi GPU)
python train.py --data_root /home/robertwang/veri776/
torchrun --nproc_per_node=2 train.py --data_root /home/robertwang/veri776/

# VeRi-Wild dual-backbone
torchrun --nproc_per_node=6 train_wild.py --data_root /home/robertwang/Documents/VeRI-Wild/

# ★ Option A (single-ConvNeXt-MBR, sealed recipe)
torchrun --nproc_per_node=6 train_wild.py \
    --option_a --conv_head convnext --convnext_full_depth --mbr_faithful \
    --data_root /home/robertwang/Documents/VeRI-Wild/

# Single ViT
torchrun --nproc_per_node=6 train_wild_singlevit.py --data_root <wild>

# v7 dual-backbone fusion (--use_htm switches to full FusionReID HTM;
#   --convnext_init/--freeze_convnext gives the asymmetric frozen-anchor scheme)
torchrun --nproc_per_node=6 train_wild_mfu_fusion.py --data_root <wild>
```

| Script | Role |
|---|---|
| `train.py` | VeRi-776 two-phase DDP training (Phase1 BoT align → Phase2 full) |
| `train_wild.py` | VeRi-Wild training; `--option_a` enables Option A; contains `apply_model_preset`/`load_init_weights` helpers reused across the repo |
| `train_wild_singlevit.py` | Single ViT + single BoT branch training (CE+Triplet+Center) |
| `train_wild_mfu_fusion.py` | v7 dual-backbone full FT + learnable fusion; `--use_htm` switches to full HTM, `--convnext_init --freeze_convnext` gives the asymmetric frozen anchor |

---

## Evaluation (PyTorch)

| Script | Role |
|---|---|
| `eval_pth.py` | Evaluate mAP/Rank-1~5 from a checkpoint; supports 776 (cross-camera) and wild (disjoint lists) |
| `test.py` | Extract features, compute mAP + CMC@1/5 (based on the train evaluate logic) |
| `eval.py` | Lightweight validation (query/gallery mAP + CMC) |

```bash
python eval_pth.py --weights output_wild/optionA_Phase2_Full_Train_Wild_best.pth \
    --data_root /home/robertwang/Documents/VeRI-Wild/ --dataset veriwild
```

---

## Deployment (ONNX → TensorRT)

Input `(B,3,256,256)`, output L2-normalized ReID vector (dual-backbone 8192 / Option A 8192). ONNX uses the external-data format (`.onnx` graph ~2MB + `.onnx.data` weights, must be in the same directory). TensorRT version is **10.16**.

```bash
# Dual-backbone
python export_onnx.py                 # → output/dinov3_mbr.onnx (+.data)
python build_engine.py                # FP16 (default); --no_fp16 for FP32
python eval_trt.py   --trt <engine>   # 776
python eval_trt_wild.py --trt <engine> --data_root <wild> --test_size 10000

# Option A (pure FP32, new TRT 10.16 API)
python export_onnx_optionA.py --weights output_wild/optionA_..._best.pth
python build_engine_optionA.py --onnx output_wild/optionA_dinov3_mbr_wild.onnx
python eval_trt_wild.py --trt output_wild/optionA_dinov3_mbr_wild_fp32.trt --data_root <wild>
```

| Script | Role |
|---|---|
| `export_onnx.py` / `export_onnx_optionA.py` | Dual-backbone / Option A weights → ONNX |
| `build_engine.py` | ONNX → TRT; FP16 (LayerNorm/Reduce/Pow forced to FP32 against overflow) or FP32 |
| `build_engine_optionA.py` | Option A → TRT (pure FP32, TRT 10.16 API: `build_serialized_network`/`set_memory_pool_limit`/`execute_async_v3`) |
| `eval_trt.py` / `eval_trt_wild.py` | TRT engine evaluation on 776 / wild |
| `compare_backends.py` | PTH vs ONNX vs TRT accuracy + throughput comparison |
| `analyze_similarity_trt.py` | Intra/inter-class cosine distributions on the TRT engine |
| `base_model_output/{export_onnx,build_engine,eval_onnx}.py` | Standalone export/build/verify scripts for the 776 dual-backbone deployment artifacts |

**Measured**: the `.pth → ONNX → TRT (FP32)` chain is numerically lossless (ΔmAP 0.00); ONNX max diff 3.5e-4.

---

## Re-ranking

k-reciprocal (Zhong et al., CVPR 2017), training-free +3~6 mAP, larger galleries benefit more. Provides a **mathematically-equivalent sparse implementation** (the classical version needs ~190GB and OOMs on test_10000; the sparse version runs the 138k-image gallery in 137s, |ΔmAP|=0.000, 18× faster).

| Script | Role |
|---|---|
| `embedding_explaing/rerank.py` | `re_ranking` (classical dense) + `re_ranking_sparse` (memory-efficient) + `euclidean` |
| `embedding_explaing/rr_optionA.py` | Runs baseline + rerank on Option A sealed weights (with memory guard, classic/sparse cross-check) |

```bash
.venv/bin/python embedding_explaing/rr_optionA.py \
    --weights output_wild/optionA_..._best.pth --full_depth --all_cls \
    --data_root /home/robertwang/Documents/VeRI-Wild/ --test_size 10000 --method sparse
```

---

## Analysis & Experiment Scripts (`embedding_explaing/`)

> This is the research core of the repo. Naming conventions: `bN_*` = training-free probe series; `expN_*` = embedding-interpretability experiments; `b16*` = Option A/v7 branch decomposition; `bootstrap_*` = statistical confidence intervals. Results land in `embedding_explaing/results/`.

### Shared utilities

| Script | Role |
|---|---|
| `common.py` | 8192-vector = 4 branch sub-vectors; shared tools for interpretability experiments (feature extraction, splitting) |
| `b7_fusion_reweight_wild.py` | Provides `BRANCHES` / `extract_branches` / `weighted_concat` / `map_cmc_xcam` (**reused by many scripts**); also does branch re-weighting validation |

### Branch decomposition & complementarity (core diagnostics)

| Script | Role |
|---|---|
| `exp1_branch_decomp.py` / `_wild.py` | 8192 cos decomposes exactly into a sum of 4 branch cosines; per-branch mAP + complementarity (776 / wild) |
| `analysis_dims.py` | Dimension-/branch-level geometry: intra-inter discriminability, which branch the top-K contributing dims fall in |
| `b16_optionA_branch_eval.py` | Option A per-branch evaluation, bypassing the contaminated 8192 concat |
| `b16b_branch_diff.py` | **Option A per-branch mAP + top-K Jaccard complementarity + complementary-correctness** (§5.2 core) |
| `b16c_mfu_branch_diff.py` | v7 cross-backbone (ViT vs ConvNeXt) branch analysis + fusion gain (§5.3) |
| `b9_776_branch.py` | 776 per-branch / single-ViT retrieval (decides whether 776 keeps the dual backbone) |
| `b9_vit_only_wild.py` | wild single ViT (BoT two paths) retrieval — lower bound of the "drop ConvNeXt" direction |

### Spatial attribution: occlusion saliency

| Script | Role |
|---|---|
| `exp3_occlusion_heatmap.py` / `_wild.py` | Occlusion heatmap of final_embedding (776 / wild) |
| `exp3_occlusion_wild.py` | Option A (DINOv3ConvNeXtMBR) occlusion heatmap |
| `exp3_occlusion_branches.py` | **Same-backbone conv-head vs attn-head** occlusion agreement (Pearson + top-20% IoU) |
| `exp3_occlusion_mfu.py` | v7 fused-vector occlusion heatmap |
| `exp3_occlusion_mfu_branches.py` | v7 **cross-backbone vit_g vs cnx_g** occlusion agreement (the 0.30 in §5.3) |
| `exp3_occlusion_mfu_allbranches.py` | v7 all-5-branch occlusion agreement matrix |

### Semantic attribute probes

| Script | Role |
|---|---|
| `exp4_attribute_probe.py` / `_wild.py` | Frozen vector → linear probe predicting color/type/brand (what semantics are encoded) |

### Training-free improvement probes (conclusion: local gains require real training)

| Script | Role |
|---|---|
| `b1_gem_posthoc.py` | GeM pooling in place of GAP (sweep p) |
| `b2_fusion_search.py` | Branch fusion-weight search (equal vs non-equal oracle) |
| `b3_pcb_posthoc.py` | PCB stripe/grid pooling |
| `b4_resolution_probe.py` / `resolution_probe.py` | Increase inference resolution (256→320/384) |
| `b5_crosscam_776.py` / `b6_crosscam_wild.py` | Strict cross-camera vs unfiltered protocol comparison (**reveals 3–4 mAP protocol inflation**) |
| `b8_common_visible_wild.py` | Common-visible part gating + DINOv3 pseudo-parts (K-means prototypes) |
| `b10_raw_vit_wild.py` | Bare DINOv3 ViT-L retrieval baseline (no training) |
| `b14_backbone_raw_wild.py` | Bare-backbone comparison ConvNeXt vs ResNet50 vs ViT (training-free) |
| `b15_cls_fusion_wild.py` | ViT CLS-token fusion probe (our backbone skips the CLS token) |
| `exp5_patch_token.py` | ViT patch tokens as a 5th path |
| `pca_compress_probe.py` | 8192 → PCA{2048,1024,512,256} (proves effective rank ~512) |
| `concat_readout_probe.py` | Bypass the fusion head to compare readout strategies (readout vs representation issue) |
| `late_fusion_probe.py` | Late fusion at the ranking/distance level (vs feature-level concat) |
| `fit_fusion_weights.py` | Isolated fit of learnable fusion weights (fit on train set, eval on test, avoids optimistic bias) |

### Single-ViT-specific evaluation

| Script | Role |
|---|---|
| `b11_singlevit_eval.py` | Single ViT (DINOv3ViTReID) cross-camera mAP + PCA (**verified single-ViT cross-cam 67.03**) |
| `b12_cos_singlevit.py` | Single-ViT intra/inter cosine + discriminability (closed-form) |
| `b13_deploy_gains_wild.py` | Single-ViT deployment-side gain ablation (flip-TTA / PCA / rerank stacked) |

### Cosine distributions / cross-domain

| Script | Role |
|---|---|
| `cos_dist_optionA.py` | Option A intra/inter cosine distribution (d-prime, quantiles, separability); **gallery-feature cache source** |
| `pt_nocam_optionA.py` | Option A no-camera mAP (verifies TRT zero precision loss, same protocol as eval_trt_wild) |
| `test776_optionA.py` | **Cross-domain** mAP of wild-trained Option A on VeRi-776 (66.02) |
| `branch_collapse_776.py` | 776 in-domain branch-collapse probe (per-branch mAP + paired-difference CI + Jaccard) |

### Statistical rigor (paper confidence intervals)

| Script | Role |
|---|---|
| `bootstrap_ci_optionA.py` | Query-level bootstrap 95% CI of Option A mAP/R1 (§4.2 parity argument) |
| `bootstrap_paired_diff.py` | **Paired-difference** bootstrap (concat−best head) + exact per-branch mAP (review R1) |
| `bootstrap_diversity.py` | Bootstrap CI of diversity metrics (Jaccard / occlusion Pearson·IoU) |

### Cross-backbone control / cost / figures

| Script | Role |
|---|---|
| `cka_backbone_control.py` | 4-backbone (DINOv3/ImageNet × ConvNeXt/ViT) linear-CKA matrix (§5.4 shared-teacher control) |
| `bench_flops_latency.py` | Single- vs dual-backbone Params/FLOPs/Latency (§6: 2.8× latency / 3× FLOPs) |
| `plot_training_logs.py` | Parses `logs/*.log` into 4 statistical figures (anti-collapse overlay, LR-vs-mAP-vs-loss, component losses, peak/final bars) |
| `draw_mfu_arch.py` / `draw_mfu_fusion_arch.py` | matplotlib architecture diagrams of MFU / MFU-Fusion |

---

## Model Modules (`models/`)

| File | Role |
|---|---|
| `backbone_convnext.py` | DINOv3 ConvNeXt-B wrapper (frozen prefix + trainable tail + adapter; `use_full_depth` uses stage3) |
| `backbone_vit.py` | DINOv3 ViT-L/16 wrapper + `LoRALinear` injection (layers 21-24) |
| `branch_r50.py` | ResNet50 Stage4 Bottleneck branch + BNNeck (optional ArcFace) |
| `branch_bot.py` | BoT (Bottleneck Transformer) branch, 2D relative-position MHSA |
| `branch_convnext.py` | ConvNeXt-block conv head (Option A conv side, replaces the R50 head) |
| `dinov3_mbr.py` | Dual-backbone 4-branch MBR (original) |
| `dinov3_convnext_mbr.py` | **Option A** single ConvNeXt + 4 heads |
| `dinov3_vit_reid.py` | Single ViT + single BoT branch |
| `dinov3_mfu.py` | MFU cross-attention mutual fusion |
| `dinov3_mfu_fusion.py` | v7 dual-backbone full FT + learnable fusion head |
| `dinov3_htm.py` | Full FusionReID HTM (LRU + shared SEU + token MFU + stacking) |
| `losses.py` | CrossEntropyLabelSmooth / CenterLoss / CircleLoss |

---

## Configuration (`config.py`)

All tunable parameters are nested dataclasses (`convnext` / `vit` / `branch` / `loss` / `data` / `train`). Three losses combined: CE (LabelSmooth ε=0.1, weight 0.6) + CenterLoss (0.005, separate SGD lr=0.5) + CircleLoss (1.0, margin=0.25, γ=256). **Override paths via CLI, do not edit config.**

---

## Checkpoints

| Path | Model | Metric |
|---|---|---|
| `output_wild/optionA_Phase2_Full_Train_Wild_best.pth` | Option A sealed | wild 88.19 / 77.47 (578 keys) |
| `base_model_output/Phase2_Full_Train_best_8545.pth` | 776 dual-backbone | 85.46 unfiltered / 82.42 cross-camera |
| `veri_wild_baseline/SingleViT_Wild_best.pth` | Single ViT | wild cross-cam 67.03 (epoch 26) |
| `veri_wild_baseline/Phase2_Full_Train_Wild_best.pth` | wild dual-backbone baseline | — |
| `output_wild_mfu/Phase1_BoT_Align_Wild_best.pth` | v7 Phase1 align | frozen-backbone state |

---

## Documentation

| Document | Contents |
|---|---|
| [`CLAUDE.md`](CLAUDE.md) | Codebase navigation, commands, architecture overview |
| [`planB.md`](planB.md) | Architecture documentation (mermaid diagrams) |
| [`DINOv3-MBR_架构诊断与重构.md`](DINOv3-MBR_架构诊断与重构.md) | Option A / Option B diagnostic source document |
| [`实验数据.md`](实验数据.md) | Experiment data summary (single source of truth) |
| [`embedding_explaing/结论沉淀.md`](embedding_explaing/结论沉淀.md) | **21-section conclusions TL;DR** (protocol / branch decomposition / fusion / deployment / architecture limits / single ViT / Option A / backbone comparison…) |

---

> Protocol note: all mAP figures default to the **official cross-camera** protocol (removing gallery same-ID + same-camera junk). Omitting this filter systematically inflates results by **3–4 mAP** (see `b5_crosscam_776.py` / `b6_crosscam_wild.py`); always confirm the protocol matches before comparing numbers.