# par-protein

**Repository Path**: ByteDance-Seed/par-protein

## Basic Information

- **Project Name**: par-protein
- **Description**: (ICML 2026 Oral) Protein Autoregressive Modeling via Multiscale Structure Generation, Official Implementation from ByteDance Seed
- **Primary Language**: Unknown
- **License**: Apache-2.0
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2026-09-15
- **Last Updated**: 2026-09-17

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# PAR: Protein Autoregressive Modeling via Multi-Scale Structure Generation

<div align="center">
  <h2><span style="color:#d73a49;">🏆 ICML 2026 Oral Presentation</span></h2>
</div>

<div align="center">
    <img width="600" alt="teaser" src="assets/generation_over_scale_with_rmsd_4_samples.png"/>
</div>

*Abstract.* We present protein autoregressive modeling (PAR), the first multi-scale autore-
gressive framework for protein backbone generation via coarse-to-fine next-scale
prediction. Using the hierarchical nature of proteins, PAR generates structures that
mimic sculpting a statue, forming a coarse topology and refining structural details
over scales. To achieve this, PAR consists of three key components: (i) multi-
scale downsampling operations that represent protein structures across multiple
scales during training; (ii) an autoregressive transformer that encodes multi-scale
information and produces conditional embeddings to guide structure generation;
(iii) a flow-based backbone decoder that generates backbone atoms conditioned
on these embeddings. Moreover, autoregressive models suffer from exposure bias,
caused by the training and the generation procedure mismatch, and substantially de-
grades structure generation quality. We effectively alleviate this issue by adopting
noisy context learning and scheduled sampling, enabling robust backbone gener-
ation. Notably, PAR exhibits strong zero-shot generalization, supporting flexible
human-prompted conditional generation and motif scaffolding without requiring
fine-tuning. On the unconditional generation benchmark, PAR effectively learns
protein distributions and produces backbones of high design quality, and exhibits
favorable scaling behavior. Together, these properties establish PAR as a promising
framework for protein structure generation.

## Setup 🚀

You can use the pre-configured [proteina environment](https://github.com/NVIDIA-Digital-Bio/proteina?tab=readme-ov-file#setup), or build the environment using the following script:

```bash
# 1. make sure conda is available before running the following

# 2. change DATA_PATH and CKPT_HOME to your own path in prepare_env.sh
DATA_PATH=/your/own/data
CKPT_HOME=/your/own/ckpt/home

# 3. create conda env and activate it
source prepare_env.sh && conda activate PAR

# 4. install MPNN and ESMFold for inference
source script_utils/download_pmpnn_weghts.sh  # Prepare PMPNN weights for designability metrics
```

For training and evaluation, additional meta files are required. Please refer to the [proteina setup](https://github.com/NVIDIA-Digital-Bio/proteina?tab=readme-ov-file#setup) for more details.

## Download Model Checkpoints 🤗

PAR model checkpoints are available on HuggingFace:

<div align="center">
  <a href="https://huggingface.co/ByteDance-Seed/PAR">
    <img src="https://img.shields.io/badge/🤗%20Hugging%20Face-ByteDance--Seed%2FPAR-ffcc4d?style=for-the-badge" alt="Hugging Face Checkpoints"/>
  </a>
</div>

Download the checkpoints from [ByteDance-Seed/PAR](https://huggingface.co/ByteDance-Seed/PAR) and set `pretrain_ckpt_path` to the downloaded checkpoint path for finetuning or inference.

### Download D_FS And PDB Data 📦

For the D_FS dataset, you need to download the `d_fs_index.txt` following [proteina setup](https://github.com/NVIDIA-Digital-Bio/proteina?tab=readme-ov-file#setup).
If you properly download the data files as in [proteina setup](https://github.com/NVIDIA-Digital-Bio/proteina?tab=readme-ov-file#setup), you will find the `d_FS_index.txt` under `$DATA_PATH/D_FS/`, which contains the indices of the D_FS dataset as created by Proteina team.

Check the data config `configs/datasets_config/afdb/d_FS.yaml` and make sure `overwrite: False` to avoid overwriting your processed data.

See [example_download_data.ipynb](example_download_data.ipynb) for more details.

`Statistics:`
The D_FS dataset used for pretraining contains 0.6M structures, as processed by Proteina. This dataset ensures both high quality (pLDDT > 80) and structural diversity, with sequence lengths ranging from 32 to 256 residues. We follow Proteina and split it by 98:19:1 for training, validation and testing.
The PDB dataset used for finetuning contains 21K designable (sc-RMSD <= 2) single chains after filtering. For detailed criterion, please refer to the Proteina paper.


## Unconditional Pretraining 🧠

We use hydra to compose the training config. Taking `configs/experiment_config/training_ca_par_60m.yaml` as the example, we introduce the config structure as follows:

```yaml
# training_ca_par_60m.yaml

...
# the following lines will load the model and data config
defaults:
  - /model: ca_par_60m  # caflow or frameflow
  - _self_
  - /datasets_config: afdb/d_FS

# by default, we use self conditioning, scheduled sampling, and noisy context learning for training
training:
  self_cond: true
  fold_cond: false
  schedule_sampling: true
  noisy_context_learning: true

```

The model config is located at `configs/model/ca_par_60m.yaml`

```yaml
# ca_par_60m.yaml

...
# the following lines will load ca_af3_60M_notri_par as the transformer encoder and caflow as the flow decoder, which follow the same architecture as Proteina
# by default, we do not use aux loss and pair representation
# browse the configs/model/nn and configs/atom_diff/nn folders for more options
defaults:
  - _self_
  - nn: ca_af3_60M_notri_par
  - /atom_diff: caflow_60m

# the default scale configuration
protein_decomposer:
  structure_dim: 3  # Ca-only
  num_local_centroids: [64, 128, 256]
```

You can simply reproduce our 60m pretraining using the following lines:
```bash
source prepare_env.sh && conda activate PAR
python proteinfoundation/train_par.py
```
This will train a 60m model on D_FS dataset for 200k steps with learning rate 0.0001.

We also provide a detailed training script `run/run_training_par.sh` for you to customize training.
```bash
# Enviroments handled inside
bash run/run_training_par.sh
```

## PDB Finetuning 🔧

Change the following line in `configs/experiment_config/finetune_ca_par_400m.yaml` to specify the pretrained checkpoint.
```yaml
pretrain_ckpt_path: /your/pretrained/model.ckpt
```

Then run the following lines to reproduce our PDB finetuning experiments:
```bash
source prepare_env.sh && conda activate PAR
python proteinfoundation/train_par.py experiment_config=finetune_ca_par_400m
```

By default, we only use the designable PDB structures for finetuning, as configured in `configs/datasets_config/pdb/pdb_train.yaml`:
```yaml
filter_designable: True
```
This requires a pre-computed designable set of PDB codes, specified in `proteinfoundation/datasets/pdb_data.py`.
```python
designable_path = 'configs/datasets_config/pdb/pdb_train_designable.pkl'
```
You can also custimize the filtering criterion by modifying the parameters in `configs/datasets_config/pdb/pdb_train.yaml`.

You can also modify parameters in `run/run_training_par.sh` for finetuning.
Since the PDB dataset is much smaller than D_FS, we recommend using a smaller learning rate and fewer steps for finetuning.
```yaml
opt.lr=0.00001
opt.max_steps=10000
opt.val_check_interval=1000
log.checkpoint_every_n_steps=1000
```

## Sampling & Evaluation 📊

Use `run/run_inference_par.sh` to sample and evaluate the model. This script will execute the following steps:

1. Sample 500 structures in multiple gpus for computing designability.
2. Run the MPNN-ESMFold pipeline for folding the sampled structures in multiple gpus.
3. Sample 5000 structures in multiple gpus for computing FPSD and fS as proposed in [Proteina](https://github.com/NVIDIA-Digital-Bio/proteina).
4. Aggregate the folding results and compute sc-RMSD and designability.
5. Collect the sampled structures and compute FPSD and fS.

We provide example outputs under `example_output`. After uncompressing `rmsd.tar.gz` and `fid.tar.gz`, you will find files in the following structure:
```text
example_output/
  - inference_fid_par/samples_fid
    - *.pdb
  - inference_rmsd_par/
    - **/*.pdb
  - results_inference_fid_par.csv
  - results_inference_rmsd_par.csv
```

To evaluate designability of your own samples, you can use `fold_inference_data.py` to run the MPNN-ESMFold pipeline for folding the sampled structures.
```bash
# args: 
# 1: number_of_gpus
# 2: gpu_id
# 3: inference_path
python fold_inference_data.py 1 0 your_inference_path
```
This code will split the .pdb samples under `your_inference_path` into `number_of_gpus` parts, and run folding with one part on `cuda:gpu_id`.

The diversity and secondary structure are computed only for the designable structures. Run `example_compute_metrics.ipynb` to obtain the metrics.


## Generate With Prompt ✨

We provide a notebook `example_prompt_generation.ipynb` with example structures shown in the paper. This notebook will read a structure in PDB format, downsample it to 16 points, and use a 60m 5-scale model to generate the complete structure. You can also feed your 16 point prompt to the model directly. Before running, uncompress `example_prompt/prompt.tar.gz`, which contains the prompt structures used in the paper (fig. 3).

```text
barrel/
  - 1_all_scales.pdb
  - 1_ref.pdb
  - 1_rmsd_xxx.pdb
```

The `*_rmsd_xxx.pdb` files are the generated structures. The `*_ref.pdb` files are the reference structures. The `*_all_scales.pdb` files store the generated structures of all scales.

## 0-Shot Motif Scaffold 🧩

We provide a notebook `example_motif_scaffold.ipynb` to demonstrate how does the model work. In this example, we read the test set and extract the last 20 residues as the motif segment, and generate the rest of the structure.

We also provide a script `run/motif_scaffold_par.sh` to execute the motif benchmark as in RFDiffusion (data located in `./motif_data`), with `compute_motif_success.py` to compute the success rate.
For new motif tasks, you can put the pdb file under `./motif_data` and add corresponding contig string and min/max lengths in `MOTIF_CONTIGS` and `MOTIF_LENGTH` in `proteinfoundation/motif_inference_par.py`.

## Citation

If you use this codebase in your research, please cite the following paper:
```
    @article{qu2026par,
        title={Protein Autoregressive Modeling via Multiscale Structure Generation},
        author={Yanru Qu and Cheng-Yen Hsieh and Zaixiang Zheng and Ge Liu and Quanquan Gu},
        journal={arXiv preprint arXiv:2602.04883},
        year={2026},
        url={https://arxiv.org/abs/2602.04883}, 
    }
```