# RS5M

**Repository Path**: hf-datasets/RS5M

## Basic Information

- **Project Name**: RS5M
- **Description**: Mirror of https://huggingface.co/datasets/Zilun/RS5M
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2023-10-30
- **Last Updated**: 2024-06-09

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

---
license: cc-by-nc-4.0
language:
- en
size_categories:
- 1M<n<10M
---


# RS5M

## File Explaination

### 1. pub11_NER_geolocation_info.csv
* This file provides extracted geolocation entities in caption. we discovered that the captions from the PUB11 dataset contain a significant amount of location information. As a result, we executed a NER (Named Entity Recognition) extraction on the PUB11 subset.

* We hypothesize that the location information in the captions is closely related to the image's content and its shooting location. While this might introduce some noise, given that most PUB11 images originate from the internet and the paired text's purpose is to supplement the image, we believe most of the location data is useful.

* We specifically extracted entities labeled as "GPE" (geopolitical entities). However, most of these entities are country or city names, not UTM zones or latitude/longitude details. While city names can be readily converted to UTM zones, captions containing only country names provide us with coarse spatial information. Nonetheless, this is a valuable addition to our analysis of RS5M's geographic distribution.

* Out of the dataset, 880,354 images have captions with location information. We took the NER tool from the NLTK implementation. We also tried Stanford NER models, but the estimated processing time was 900 hours. In the future, we plan to develop an algorithm to convert extracted GPEs to UTM zones if applicable.

  |img_name|	text|	entity|
  |:---:|:---:|:---:|
  |laion2b_0_0	|Aerial photography Pattern on the Earth Field Corn Farm Abstract Harvest Season|	[]|
  |laion2b_0_2|	San AntonioTexas  suburban housing development neighborhood - aerial view stock photo|	['San AntonioTexas']
  |laion2b_0_4|	Aerial view of historical orthodox monasteries on the top of meteors cliffs|	[]
  |laion2b_0_7|	Aerial view of Albert Park and the Melbourne skyline, Australia|	['Melbourne', 'Australia']
  |laion2b_0_9|	Aerial photo taken on Oct. 6, 2019 shows tourists viewing pink muhly grass in the Fenghuanggou scenic area during the National Day holiday in Nanchang, capital of east China's Jiangxi Province. (Xinhua/Peng Zhaozhi)|	['Fenghuanggou', 'Nanchang', 'China', 'Jiangxi Province']


### 2. pub11_metacap_country_month.pkl

* Only data from YFCC14M (with 7,841 pairs) have info on "meta_caption", "country", and "month".

* **img_name**: Image name
* **text**: Image caption
* **url**: Image download url
* **download_status**: If the image was downloaded successfully.
* **meta_caption**: Meta caption generated by the template with absolute corrected information from the dataset. (month, country, date, shooting angle, etc.)
* **country**: Which country the image was shot.
* **month**: When the image was shot.

  |img_name|text|url|download_status|meta_caption|country| month|
  |:---:|:---:|:---:|:---:|:---:|:---:|:---:|
  |laion2b_0_0|        Aerial photography Pattern on the Earth Field Corn Farm Abstract Harvest Season|                                                               https://image.shutterstock.com/image-photo/stock-photo-aerial-photography-pattern-on-the-earth-field-corn-farm-abstract-harvest-season-450w-702011665.jpg|         SUCCESS|          NaN|     NaN|   NaN|
  |laion2b_0_2|  San AntonioTexas  suburban housing development neighborhood - aerial view stock photo|  http://media.istockphoto.com/photos/san-antoniotexas-suburban-housing-development-neighborhood-aerial-picture-id170094223?k=6&amp;m=170094223&amp;s=170667a&amp;w=0&amp;h=53-MMobWRGSl29N1E3oQa8FVsv53FL2D9eqfLn5hvl0=|         SUCCESS|          NaN|     NaN|   NaN|
  |laion2b_0_4|            Aerial view of historical orthodox monasteries on the top of meteors cliffs|                                                                         https://cdn.kated.com/wp-content/uploads/2020/06/Grc39v-Aerial-view-of-historical-orthodox-monasteries-on-the-top-of-meteors-cliffs-150x150.jpg|         SUCCESS|          NaN|     NaN|   NaN|
  |yfcc14m_27147| The canyon on the right contains Araster Spring, the canyon on the left is Edgar Canyon on the USGS maps.  |http://farm3.staticflickr.com/2404/13175374033_8445b81b3f.jpg|         SUCCESS   |             bearing a timestamp of 20 o'clock, March 12, 2014, this shot offers a glimpse of Spring in San Manuel, United States.|  United States|   March|
  |yfcc14m_27148| Echo Rock at 7870 feet "Observation Rock and Echo Rock are dissected satellitic volcanoes, which erupted  olivine andesite  upon Mount Rainier's northwestern flank late in its history. "  - from Fisk, 1963, Geology of Mount Rainier National Park, Washington: USGS Professional Paper 444   Link to More Info   i082006 326|      http://farm1.staticflickr.com/82/220990009_cfc85da775.jpg      |   SUCCESS   |             taken in the heart of Summer within Buckley, United States, this image's timestamp reads 13 o'clock, August 19, 2006.|  United States|  August|
  |yfcc14m_27151 |There are lots of unique stones laying around.  A few distinct types... numerous white round stones, as seen on the right.  Also there's lots of crystall-y flakey rock, like the large one on the lower left.  It doesn't come out in this photo, but it reflects light quite well.  It's clearly man-made.  Look at it in the satellite imagery (link below).  Seriously.  There used to be a building on the top.  You can tell it's man made by the way the rocks and dirt are piled.   None of the locals I've interviewed have any idea who put it there.  It's been there for as long as any of them have a verbal history.    What we do know is there's lots of pottery shards... thousands of them.   Last time we were here some of the locals told us after it rains kids search the mound and sometimes find old coins.  Further questioning got nowhere.  Dave and I like to hike here, it's got a great view of the valley.  This time we found a guy at the top digging for artifacts, and he found one while we were there.  It was a large clay bead with designs in it.  He offered to sell it to us... and while I wanted to I didn't because we 1) don't want to reward that behavior and 2) it's bad form and if it isn't it should be illegal.  (34.44809567863388, 70.395348072052) maps.google.com/maps?f=q&source=s_q&hl=en&geo...  | http://farm4.staticflickr.com/3605/3393504467_5b7bf7a058.jpg     |    SUCCESS  |     captured in Jalālābād, Afghanistan, this image highlights the beauty of Spring and is timestamped at 8 o'clock, March 7, 2008.   | Afghanistan |  March
  

### 3. pub11_train_metadata.csv, pub11_validation_metadata.csv, rs3_train_val_metadata.csv

* For pub11 subset, we split the train file and validation file.

  * **file_name**: Image name
  * **text**: Image caption

  |file_name|	text|
  |:---:|:---:|
  |laion2b_0_0.jpg|	Aerial photography Pattern on the Earth Field Corn Farm Abstract Harvest Season
  |laion2b_0_2.jpg|	San AntonioTexas  suburban housing development neighborhood - aerial view stock photo
  |laion2b_0_4.jpg|	Aerial view of historical orthodox monasteries on the top of meteors cliffs
  |laion2b_0_5.jpg|	Overhead view of a car parking entrance road. Aerial view....
  |laion2b_0_7.jpg|	Aerial view of Albert Park and the Melbourne skyline, Australia

* For rs3 subset, we did not split the train and validation files at this stage.

  * **img_name**: Image name
  * **subset_name**: fmow/ben/milllionaid
  * **top1_cap_vanilla**: The rank1 image caption filtered by VLMs. Generated by BLIP2-opt6.7B.
  * **ssl_cap_vanilla**: The rank1 rotation invariant image caption. Generated by BLIP2-opt6.7B.
  * **top1_cap_ft**: The rank1 image caption filtered by VLMs. Generated by fine-tuned BLIP2-opt6.7B.
  * **ssl_cap_ft**: The rank1 rotation invariant image caption. Generated by fine-tuned BLIP2-opt6.7B.
  * **country**: Which country the image was shot.
  * **month**: When the image was shot.
  * **meta_caption**: Meta caption generated by the template with absolute corrected information from the dataset. (month, country, date, shooting angle, etc.)

  img_name|	subset_name|	top1_cap_vanilla|	ssl_cap_vanilla|	top1_cap_ft|	ssl_cap_ft|	country	|month|	meta_caption|
  |:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
  |P0000001.jpg|	millionaid|	a google earth aerial view of a commercial building and some parking lots|	view of the roof top parking lot from a satellite image, with an overhead view of the building|	a parking lot next to some businesses with cars in them. There are many cars parked in it. There is one in the front and many on the sides. They are parked in front of a building as it has a parking lot in it|	There are many yellow buildings next to a big highway. There is a white line dividing the buildings from the rest of the parking lot. The buildings seem to be in different rooms yet there are roads connecting each other. There is also an intersection	||||		
  |293813.jpg|	ben|	an old satellite photo of some land that has been covered in dirt|	the satellite view of a dirt area, with no grass|	This area is not very large but has a number of houses. This is located in an area of land between two rivers. This area is not very large but has a number of houses.|	This area is not very large but has a number of houses. This is located in an area of land between two rivers. This area is not very large but has a number of houses. |	|	August|	a depiction from Summer, this satellite image showcases 'non-irrigated arable land, land principally occupied by agriculture, with significant areas of natural vegetation, transitional woodland/shrub' and is affiliated with utm zone 29N, timestamped 11 o'clock, August 18, 2017.
  |fmow/train/zoo/zoo_90/zoo_90_1_msrgb.jpg|        fmow|                               a satellite photo shows the river next to buildings|           the satellite image shows an area that is near the river|                                                                                     The path is very close to the river. there are trees in the middle of the path. a small park sits close to the path. a small town is also close to the path|                   there are several roads next to a river. A river is the source of various plants and animals. Many trees are around the river. The ground is fertile and provides people with plenty of food|       Argentina|  December|                          a peak into Rawson, Argentina during its Winter showcases zoo at the center and top-center blocks. the clarity comes from a ground sample distance of 2.15 meters, and it belongs to the series from the utm zone 20G, time-marked on 14 o'clock, December 24, 2016.
  

### 4. rs3_train_val_metadata_clean.pkl
* There are many repeated expression in the caption generated by the fine-tuned model (see file "rs3_train_val_metadata.csv"). We performed a dedulication on sentences with similar meaning.


### 5. pub11_train_intermediate.csv, pub11_val_intermediate.csv, rs3_train_intermediate.csv, rs3_val_intermediate.csv
* This file contains the final captions we made/selected for generating RS5M data with webdataset format.

* For pub11 subset
  * **img_name**: Image name
  * **caption**: Image caption

  img_name|	caption
  |:---:|:---:|
  laion2b_0_29.jpg|	Aerial view of Teahwhit Head and James Island.jpg
  laion2b_0_94.jpg|	Aerial view of boats. Top view of yachts from flying drone stock video
  laion2b_0_109.jpg|	This aerial view of the exhibit shows the barn, yard, and water feature.


* For rs3 subset
  * **base_name**: Image name, path related
  * **subset_name**: fmow/ben/millionaid
  * **save_name**: Image name, unique name, path unrelated
  * **caption**: Image caption

  base_name|	subset_name|	save_name|	caption|
  |:---:|:---:|:---:|:---:|
  fmow/train/place_of_worship/place_of_worship_2222/place_of_worship_2222_0_rgb.jpg|	fmow|	fmow_place_of_worship_2222_0_rgb.jpg|	the Fall aura of Tanjung, Indonesia gets a voice with this image of place of worship at the center and top-left blocks. its clarity, a result of the ground sample distance of 2.25 meters, ensures its place in the utm zone 50L, timestamped 2 o'clock, September 22, 2015. a satellite image of a large village surrounded by trees|
  P0294281.jpg|	millionaid|	millionaid_P0294281.jpg|	there is a very large water area next to the road|
  66994.jpg|	ben|	ben_66994.jpg|	originating from utm zone 32N in the Summer season, this satellite image showcasing 'complex cultivation patterns, coniferous forest, inland marshes' is timestamped 10 o'clock, August 18, 2017. a satellite photo of the area near the train tracks