# Open-AISP

**Repository Path**: lintwins155_cortexclaw/Open-AISP

## Basic Information

- **Project Name**: Open-AISP
- **Description**: Open-AISP is a toy-level, and open-source AI-ISP (Artificial Intelligence Image Signal Process) pipeline framework for beginners. 
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2026-09-18
- **Last Updated**: 2026-09-18

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# 🌟 Open-AISP

![Python](https://img.shields.io/badge/Python-3.11+-blue.svg)
![License](https://img.shields.io/badge/License-MIT-green.svg)
![Status](https://img.shields.io/badge/Status-Active-success.svg)

**Open-AISP** is a toy-level, open-source AI-ISP (Artificial Intelligence Image Signal Processor) pipeline framework for beginners. The workflow follows a staged ISP bring-up: realistic RAW degradation, neural RAW reconstruction, traditional static HDR fusion from RAW/DNG data, then AI tone mapping and post enhancement.

## 📚 Documentation

| Language | Entry |
| :---: | :--- |
| 🌐 English | [English Documentation](https://www.coolsyn.top/Open-AISP/) |
| 🇨🇳 中文 | [中文文档](https://www.coolsyn.top/Open-AISP/zh/) |

## 🗓️ Timeline

| Date | Update |
| :--- | :--- |
| 2026-04-27 | Add bilingual Mkdocs documentation for Open-AISP|
| 2026-04-25 | Init repo with `raw-sim` and `JDD` modules|

---

## 🧩 Core Modules

### 📸 1. Raw Simulation (raw-sim)


This module focuses on generating highly realistic degraded raw data that mimics physical sensor characteristics, which is essential for training robust AI-ISP models.
- 🔄 **Unprocess Pipeline**: Reverses high-quality RGB images (DIV2K & Flickr2K) back to the raw domain, inlcuding *4x4 quadbayer* and *2x2 binning* color filter format.
- 🎛️ **Gaussian-Poisson Mixed Noise Model**: Accurately simulates the physical noise introduced during the photoelectric conversion process, including shot noise and read noise with calibrated noise model.
- **Optical PSF Degradation**: Introduces Point Spread Function (PSF) degradation to mimic the *optical blurring* and *chromatic aberration* of real-world camera lenses.

| Input SRGB (DIV2K) | Output Raw (ISO6400)  |
| :---: | :---: |
| <img src="./raw-sim/assets/sample_gt_linear_rgb.png" width="400" /> | <img src="./raw-sim/assets/sample_input_noisy_raw.png" width="400" /> |

### 🧠 2. Joint Denoising and Demosaicing (MF-JDD)
The JDD module handles the early-stage core reconstruction tasks in the ISP pipeline.
- 📊 **Brust Image denoise**: Utilizes deep learning architectures to learn the complex mapping from **multi-frame raw inputs** to a high-quality **single-frame linear RGB** image. 
- ⚙️ **Hardware Noise Estimation**: Estimate the noise map based on analog/digital gain and noise parameter calibration, and guide the JDD model to perform denoising.
- **[Todo] Burst Image alignment**: efficient multi-frame image alignment and warp.

| Opencv Demosaic | JDD Output  | Ground truth    |
| :---: | :---: | :---: |
| <img src="./JDD/assets/opencv_demosaic.png" width="400" /> | <img src="./JDD/assets/pred.png" width="400" /> | <img src="./JDD/assets/gt.png" width="400" /> |

### 🌅 3. Multi-frame HDR Synthesis (HDR)
The current HDR module is a traditional static fusion baseline. It uses MIT5K DNG images, decodes them to linear RGB, dynamically estimates over/normal/under exposure brackets from luminance distribution, and fuses them with photometric weights plus Gaussian/Laplacian pyramids.

- **Static multi-exposure fusion**: Estimate scene-dependent brackets from one MIT5K DNG image and fuse them with Gaussian/Laplacian pyramids.
- **No AI-HDR yet**: DeepHDR-style moving bursts and neural HDR merging are reserved for the next HDR iteration.

| Under exposure | Normal exposure | Over exposure | Fused HDR (TM) |
| :---: | :---: | :---: | :---: |
| <img src="./HDR-fusion/assets/a0003_under.png" width="220" /> | <img src="./HDR-fusion/assets/a0003_norm.png" width="220" /> | <img src="./HDR-fusion/assets/a0003_over.png" width="220" /> | <img src="./HDR-fusion/assets/a0003_fused.png" width="220" /> |


### 🎨 4. [TODO] Learning-based Tonemapping (AITM)

Data-driven AI Local Tonemapping Module

### 🪄 5. [TODO] Diffusion-based Image Post Enhancement (DiffIPE)

Image Post-processing and Enhancement Based on Large-Scale Pre-training and Adversarial Distillation of Single-Step Diffusion Models

---

## 📅 Project Roadmap
- **✅ Raw Image Simulation (`raw-sim`)**
  - [x] Basic unprocess pipeline (sRGB to Raw)
  - [x] Standard Bayer / QuadBayer / Binning sensor formats
  - [x] Calibrated Gaussian-Poisson mixed noise modeling
  - [x] Lens optical degradation (PSF)
- **✅ Joint Denoise & Demosaic (`JDD`)**
  - [x] Neural network architecture setup
  - [x] Multi-frame fusion implementation
  - [x] Hardware noise estimation integration
  - [ ] Burst image alignment registration (WIP)
- **⏳ Multi-frame HDR Synthesis (`MF-HDR`)**
  - [x] Traditional static multi-exposure simulation and fusion from MIT5K DNG images
  - [ ] AI-HDR fusion for moving multi-exposure bursts
- **⏳ Deep Local Tonemapping (`LTM`)**
  - [ ] Specialized linear RGB to sRGB dataset preparation
  - [ ] End-to-end neural tonemapping model implementation
- **⏳ Diffusion Post-Enhancement (`IPE`)**
  - [ ] Latent diffusion baseline setup for dark-light / noise extreme scenarios
  - [ ] Inference efficiency optimization (Distillation / Single-Step)

---

## 🚀 Quick Checks

Run commands from the repository root.

Generate one RAW simulation sample:

```bash
python raw-sim/scripts/generate_raw.py \
  --input raw-sim/examples/input/sample.png \
  --output raw-sim/simu_pairs \
  --camera-json raw-sim/configs/cameras/example_camera_10bit_RGGB_binning.json \
  --patch-size 128 \
  --num-patches 1
```

Run quick JDD inference with the bundled checkpoint:

```bash
python JDD/scripts/infer.py \
  --checkpoint JDD/assets/latest.pth \
  --input raw-sim/examples/input/sample.png \
  --output JDD/runs/quick_infer \
  --camera-module-json JDD/configs/camera_module_10bit_binning_precali_noise_rggb_ag1to64.json \
  --mode full \
  --max-images 1
```

Download a small MIT5K DNG subset:

```bash
python HDR-fusion/scripts/download_mit5k.py --ids 3
```

Run static HDR fusion:

```bash
PYTHONPATH=HDR-fusion python -m hdr_fusion.cli \
  --input HDR-fusion/datasets/MIT5K/dng/a0003.dng \
  --output HDR-fusion/outputs/a0003_fused.hdr \
  --metrics HDR-fusion/outputs/a0003_metrics.json
```

## 🔗 References & Related Projects
- [openISP](https://github.com/cruxopen/openISP)
- [fast-openISP](https://github.com/QiuJueqin/fast-openISP)
- [ISPLab](https://github.com/yuqing-liu-dut/ISPLab)