# grounding_dataset

**Repository Path**: hf-datasets/grounding_dataset

## Basic Information

- **Project Name**: grounding_dataset
- **Description**: Mirror of https://huggingface.co/datasets/Salesforce/grounding_dataset
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2025-10-17
- **Last Updated**: 2025-10-17

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

---
dataset_info:
  features:
  - name: dataset
    dtype: string
  - name: uuid
    dtype: string
  - name: image
    dtype: image
  - name: bbox
    sequence: int64
    length: 4
  - name: instruction
    dtype: string
  - name: description
    dtype: string
  - name: function
    dtype: string
  - name: combine
    dtype: string
  - name: org_caption
    dtype: string
  splits:
  - name: train
    num_bytes: 37185031827
    num_examples: 70688
  download_size: 34691500772
  dataset_size: 37185031827
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
task_categories:
- image-text-to-text
language:
- en
tags:
- Grounding
- GUI
size_categories:
- 10K<n<100K
---
# Grounding Dataset

A comprehensive, high-quality dataset for GUI element grounding tasks, curated from multiple authoritative sources to provide diverse, well-annotated interface interactions.

## Overview

This dataset combines and standardizes annotations from five major GUI interaction datasets:
- **[Aria-UI](https://github.com/AriaUI/Aria-UI)**
- **[OmniAct](https://huggingface.co/datasets/Writer/omniact)**
- **[Widget Caption](https://huggingface.co/datasets/rootsautomation/RICO-WidgetCaptioning)**
- **[UI-Vision](https://huggingface.co/datasets/ServiceNow/ui-vision)**
- **[OS-Atlas](https://huggingface.co/datasets/OS-Copilot/OS-Atlas-data)**

## Dataset Schema

Each sample contains the following fields:

| Field | Type | Description | Example |
|-------|------|-------------|---------|
| `dataset` | string | Source dataset identifier | "ariaui", "omniact", "widget_caption", "ui_vision", "os_altas" |
| `uuid` | string | Unique sample identifier | "0ce7f27b-0d76-4276-a624-39fc1836b46e" |
| `image` | PIL.Image | Screenshot/interface image | RGB image object |
| `bbox` | list[int] | Bounding box coordinates [x1, y1, x2, y2] | [33, 75, 534, 132] |
| `instruction` | string | Action-focused instruction | "Tap the Search Maps field" |
| `description` | string | Visual element description | "Dark gray, rounded search bar with magnifying glass icon" |
| `function` | string | Functional purpose | "Use this input field to find a specific location" |
| `combine` | string | Comprehensive instruction | "At the top of the left sidebar, tap the dark gray search bar..." |
| `org_caption` | string | Original caption from source | "search maps" |

## Dataset Characteristics

### Domain Coverage
- **Desktop Applications**: Native desktop software interfaces
- **Web Interfaces**: Browser-based applications and websites
- **Mobile Interfaces**: Touch-based mobile applications
- **Operating Systems**: System-level interface interactions

## Applications

This dataset supports research and development in:

### Model Training
- **Vision-Language Models**: Training models to understand GUI screenshots
- **Grounding Models**: Learning to locate elements based on natural language
- **Multimodal Understanding**: Combining visual and textual information

## Usage Examples

```python
from datasets import load_dataset

# Load the dataset
dataset = load_dataset("Salesforce/grounding_dataset")

# Access a sample
sample = dataset['train'][0]
image = sample['image']  # PIL Image
bbox = sample['bbox']    # [x1, y1, x2, y2]
instruction = sample['instruction']
```

## Licensing

This dataset inherits licenses from its constituent sources:

| Source Dataset | License |
|---------------|---------|
| Aria-UI | Apache License 2.0 |
| OmniAct | MIT License |
| Widget Caption | Creative Commons Attribution 4.0 |
| UI-Vision | MIT License |
| OS-Atlas | Apache License 2.0 |

**Important**: Each component dataset retains its original license. Please refer to the original repositories for complete licensing terms and conditions.

## Citation

If you use this dataset in your research, please cite our work:
```markdown
@article{yang2025gta1guitesttimescaling,
      title={GTA1: GUI Test-time Scaling Agent}, 
      author={Yan Yang and Dongxu Li and Yutong Dai and Yuhao Yang and Ziyang Luo and Zirui Zhao and Zhiyuan Hu and Junzhe Huang and Amrita Saha and Zeyuan Chen and Ran Xu and Liyuan Pan and Silvio Savarese and Caiming Xiong and Junnan Li},
      year={2025},
      eprint={2507.05791},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2507.05791}, 
}
```
Please cite the original datasets:

```bibtex
@misc{yang2025ariauivisualgroundinggui,
      title={Aria-UI: Visual Grounding for GUI Instructions}, 
      author={Yuhao Yang and Yue Wang and Dongxu Li and Ziyang Luo and Bei Chen and Chao Huang and Junnan Li},
      year={2025},
      eprint={2412.16256},
      archivePrefix={arXiv},
      primaryClass={cs.HC},
      url={https://arxiv.org/abs/2412.16256}, 
}
@misc{kapoor2024omniactdatasetbenchmarkenabling,
      title={OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web}, 
      author={Raghav Kapoor and Yash Parag Butala and Melisa Russak and Jing Yu Koh and Kiran Kamble and Waseem Alshikh and Ruslan Salakhutdinov},
      year={2024},
      eprint={2402.17553},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2402.17553}, 
}
@misc{li2020widgetcaptioninggeneratingnatural,
      title={Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements}, 
      author={Yang Li and Gang Li and Luheng He and Jingjie Zheng and Hong Li and Zhiwei Guan},
      year={2020},
      eprint={2010.04295},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2010.04295}, 
}

@misc{nayak2025uivisiondesktopcentricguibenchmark,
      title={UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction}, 
      author={Shravan Nayak and Xiangru Jian and Kevin Qinghong Lin and Juan A. Rodriguez and Montek Kalsi and Rabiul Awal and Nicolas Chapados and M. Tamer Özsu and Aishwarya Agrawal and David Vazquez and Christopher Pal and Perouz Taslakian and Spandana Gella and Sai Rajeswar},
      year={2025},
      eprint={2503.15661},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2503.15661}, 
}

@misc{wu2024osatlasfoundationactionmodel,
      title={OS-ATLAS: A Foundation Action Model for Generalist GUI Agents}, 
      author={Zhiyong Wu and Zhenyu Wu and Fangzhi Xu and Yian Wang and Qiushi Sun and Chengyou Jia and Kanzhi Cheng and Zichen Ding and Liheng Chen and Paul Pu Liang and Yu Qiao},
      year={2024},
      eprint={2410.23218},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2410.23218}, 
}
```