# HSA-DPO

**Repository Path**: miss-lover/HSA-DPO

## Basic Information

- **Project Name**: HSA-DPO
- **Description**: No description available
- **Primary Language**: Python
- **License**: Apache-2.0
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2025-01-25
- **Last Updated**: 2025-01-25

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

<!-- # magic-edit.github.io -->

<p align="center">
  <h2 align="center">[AAAI 2025] Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback
</h2>
  <p align="center">
    <a><strong>Wenyi Xiao<sup>1*</sup> , </strong></a>
    <a><strong>Ziwei Huang<sup>1*</sup> , </strong></a>
    <a><strong>Leilei Gan<sup>1†</sup> , </strong></a>
    <a><strong>Wanggui He<sup>2</sup>  </strong></a>
    <br>
    <a><strong>Haoyuan Li<sup>2</sup> ,  </strong></a>
    <a><strong>Zhelun Yu<sup>2</sup> , </strong></a>
    <a><strong>Fangxun Shu<sup>2</sup> ,  </strong></a>
    <a><strong>Hao Jiang<sup>2</sup> , </strong></a>
    <a><strong>Linchao Zhu<sup>1</sup>   </strong></a>
    <br>
    <sup>1</sup> Zhejiang University&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<sup>2</sup> Alibaba Group&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp
    <br>
    <sup>*</sup>Equal contribution &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp <sup>†</sup>Corresponding author
    </br>
    </br>
        <a href="https://arxiv.org/pdf/2404.14233">
        <img src='assets/Paper-Arxiv-orange.svg' alt='Paper PDF'></a>
        <a href="https://huggingface.co/datasets/WenyiXiao/HSA-DPO">
        <img src='https://img.shields.io/badge/Dataset-HuggingFace-yellow' alt='Dataset'></a>
        <a href="https://modelscope.cn/models/xiaowenyi/HSA-DPO">
        <img src='https://img.shields.io/badge/Model-ModelScope-blue' alt='Dataset'></a>
        
  </p>
</p>



## Overview

This repository contains the official implementation of the paper "Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback".

![model](assets/1.png)

## Getting Started

### Setup

```bash
git clone https://github.com/Mr-Loevan/HSA-DPO.git
cd HSA-DPO
pip install -r requirements.txt
```

### Dataset
```
pip install -U huggingface_hub
huggingface-cli download --repo-type dataset WenyiXiao/HSA-DPO
```
**For hallucination detection:** The image is sourced from [Visual Genome](https://homes.cs.washington.edu/~ranjay/visualgenome/api.html), and the training dataset can be found in `hsa_dpo_detection.jsonl`.

**For hallucination mitigation:** The image is located in `hsa_dpo_imgs.tar.gz`, and the preferences dataset is available in `hsa_dpo_preference_llava1dot5.jsonl`. Note that in llava1dot5, 'rejected' is generated by llava-v1.5.


### Model LoRA Weight
```
pip install -U modelscope
modelscope download --model xiaowenyi/HSA-DPO
```
Refer to [LLaVA repo](https://github.com/haotian-liu/LLaVA) to install inference requirements and use inference code.


### Training Code

The code is currently undergoing internal review. Please stay tuned!



## Todo List

- [x] paper
- [x] detection & mitigation datasets 
- [x] model weights
- [ ] training code