# MJHQ-30K

**Repository Path**: hf-datasets/MJHQ-30K

## Basic Information

- **Project Name**: MJHQ-30K
- **Description**: Mirror of https://huggingface.co/datasets/playgroundai/MJHQ-30K
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2024-05-23
- **Last Updated**: 2024-06-09

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

---
dataset_info:
  features:
  - name: image
    dtype: image
  - name: label
    dtype:
      class_label:
        names:
          '0': animals
          '1': art
          '2': fashion
          '3': food
          '4': indoor
          '5': landscape
          '6': logo
          '7': people
          '8': plants
          '9': vehicles
  splits:
  - name: test
    num_bytes: 9764107710
    num_examples: 30000
  download_size: 8838465412
  dataset_size: 9764107710
configs:
- config_name: default
  data_files:
  - split: test
    path: data/test-*
task_categories:
- text-to-image
language:
- en
size_categories:
- 10K<n<100K
tags:
- text-to-image
- playground
---
# MJHQ-30K Benchmark

| Model                                 | Overall FID   |
| ------------------------------------- | ----- |
| SDXL-1-0-refiner                      | 9.55  |
| [playground-v2-1024px-aesthetic](https://huggingface.co/playgroundai/playground-v2-1024px-aesthetic)        | **7.07**  |

![image/png](https://cdn-uploads.huggingface.co/production/uploads/63855d851769b7c4b10e1f76/o3Bt62qFsTO9DkeX2yLua.png)


We introduce a new benchmark, **MJHQ-30K**, for automatic evaluation of a model’s aesthetic quality. The benchmark computes FID on a high-quality dataset to gauge aesthetic quality.

We curate the high-quality dataset from Midjourney with 10 common categories, each category with 3K samples. Following common practice, we use aesthetic score and CLIP score to ensure high image quality and high image-text alignment. Furthermore, we take extra care to make the data diverse within each category.

For Playground v2, we report both the overall FID and per-category FID. (All FID metrics are computed at resolution 1024x1024.) From the results, our model outperforms SDXL-1-0-refiner in overall FID and all the categories FID, especially in people and fashion categories. This is inline with the results of the user study, which indicates the correlation between human preferences and the FID score of the MJHQ30K benchmark.

We release this benchmark to the public and encourage the community to adopt it for benchmarking their models’ aesthetic quality.

Please see our [blog](https://blog.playgroundai.com/playground-v2/) for more details.

### Dataset Download
First, download `mjhq30k_imgs.zip`

```python
from huggingface_hub import hf_hub_download

hf_hub_download(
  repo_id="playgroundai/MJHQ-30K", 
  filename="mjhq30k_imgs.zip", 
  local_dir="path/to/folder",
  repo_type="dataset"
)
```

Unzip `mjhq30k_imgs.zip` into its per-category folder structure.
```
root
├── animals
├── art
├── fashion
├── food
├── indoor
├── landscape
├── logo
├── people
├── plants
└── vehicles
```
    
`meta_data.json` contains metadata including its category and the promp for all the image.

Here is one example. Note that the key is the name of the image file.
```json
"126c23ae9a879fdc05f355f9a72b418d589f3926": {
    "category": "plants",
    "prompt": "beautiful British garden5, clear sky, unreal engine, detailed, tropical plants ,strong ambient lighting, volumetric lighting, forest plants and leaves, strong light, a lot of leaves in font, 3D effect, 8k render, bright color rendering "
}
```

### Measure FID
To benchmark your model's performance, you need to first generate images using the same prompts in `meta_data.json`.

We calculate our FID using [clean-fid](https://github.com/GaParmar/clean-fid). You can measure the FID between the generated images and the reference images using
```python
from cleanfid import fid
score = fid.compute_fid(ref_dir, gen_dir)
```

### Contributor
Dataset curated by: [Playground](https://playground.com/) Research Team