# Falcon **Repository Path**: liukejia/Falcon ## Basic Information - **Project Name**: Falcon - **Description**: RS_VLM - **Primary Language**: Unknown - **License**: MIT - **Default Branch**: csying-patch-1 - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 0 - **Forks**: 0 - **Created**: 2025-04-05 - **Last Updated**: 2025-04-05 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README

Falcon: A Remote Sensing Vision-Language Foundation Model

[\[🚀 Quick Start\]](#quick-start-with-Falcon)

![opencompass](assets/radar_graph.png)

## Introduction We are excited to introduce Falcon, which offers a unified, prompt-based paradigm that effectively executes comprehensive and complex remote sensing vision tasks. Falcon demonstrates powerful understanding and reasoning abilities at the image, region, and pixel levels. Specifically, given simple natural language instructions and remote sensing images, Falcon can produce impressive results in text form across 14 distinct tasks, i.e., image classification, object detection, segmentation, image captioning, and etc. \ ## News 🚀🚀🚀 - `2024/11/27`: Falcon has been released. The model checkpoints is now available on HuggingFace, and both training / evaluation data and scripts are open-sourced. ## Model Zoo

Model Name	Model Size	HF Link	MS Link
Falcon-Single-Instruction-Large	0.7B	🤗 link	🤖 link
Falcon-Multi-Instruction-Large	0.7B	🤗 link	🤖 link
Falcon-Single-Instruction-Base	0.3B	🤗 link	🤖 link

## What can Falcon do? ![opencompass](assets/task_example.png) ## Quick Start with Falcon

Installation (click to expand)

You can use the following script to install the environment： ```bash conda create -n falon python=3.10 conda activate falcon pip install -r requirements.txt ``` Optionally, we also provide a Docker image in [here](https://hub.docker.com/r/tianhuilab/falcon/tags) for fast deployment of the environment.

Datasets preperation (click to expand)

Training Falcon with FCD-78M (click to expand)

1. Download the checkpoints you want and place them at the root path of this repo. The directory structure should be as follows: ```bash |-model_checkpoints |----Falcon-Single-Instruction-0.7B | |---pytorch_model.bin | ... |----Falcon-Multi-Instruction-0.7B | |---pytorch_model.bin | ... |... ``` 2. Here we give an example of a training script used for single instruction training. You may runing this script on master machine node and every slave machine node you have. Note that some parameters in this script should be modified according to the machine node on which it is running. ```bash RANK=0 # The node idx of current machine node WORLD_SIZE=1 # The total number of machine node GPU_NUM=8 # The number of gpu in each machine node MASTER_ADDR=localhost # The IP address of the master machine node MASTER_PORT=12355 # The port of the master machine node python multi_node_distributed_train.py \ --node_rank $RANK \ --local_size $GPU_NUM --world_size $(($GPU_NUM*$WORLD_SIZE)) \ --master_addr $MASTER_ADDR \ --master_port $MASTER_PORT \ --checkpoint_path model_checkpoints/ \ # choose the checkpoint you want --dataset FCD-78M \ --label_json FCD/json_train_taskall/train_task14_all.json \ --num_workers 2 \ --batch_size 7 \ # Adjust this value according to your GPU memory --epochs 3 \ --run_name # edit the name of this training task ```

Evaluating Falcon with FCD-78M (click to expand)

1. Here we provide an example of the evaluation program to evaluate Falcon using FCD-78M dataset with the json annotation file. ```bash GPU=0 CUDA_VISIBLE_DEVICES=$GPU python single_gpu_inference_eval.py \ --model-path model_checkpoints/ \ --eval-file FCD//test/Annotation_test.json \ --dataset-path FCD-78M \ --model-name Falcon \ --result-path ./ \ --batch_size 8 \ # Adjust this value according to your GPU memory --num_workers 2 \ ``` 2. Running Evaluation Scripts for Single File and Batch Processing. To calculate evaluation metrics using the evaluation.py script, follow the commands below depending on whether you want to process a single file or all files in a folder. - `Process a Single Evaluation File` Run the command below, replacing "eval/tmp/model/falcon_CLS.json" with the path to your evaluation file and "falcon" with your model name: ```bash python eval/evaluation.py \ --evaluation-file eval/tmp/model/falcon_CLS.json \ --model_name falcon ``` - `Process All Evaluation Files in a Folder` Run the command below, replacing "eval/tmp/model/" with the path to the folder containing your evaluation files and "falcon" with your model name: ```bash python eval/evaluation.py \ --evaluation-folder eval/tmp/model/ \ --model_name falcon ```

## License This project is released under the [MIT license](LICENSE). Parts of this project contain code and models from other sources, which are subject to their respective licenses. ## Citation If you find this project useful in your research, please consider cite: ```BibTeX @article{yao2025falcon, title={Falcon: A Remote Sensing Vision-Language Foundation Model}, author={kelu, Yao and Nuo, Xu and Rong, Yang and Yingying, Xu and Titinunt, Kitrungrotsakul and Zhuoyan, Gao and yi, Ren and Jin, Wang and Ning, Wei and Chao, Li}, journal={arXiv preprint arXiv:XXXX.XXXXX}, year={2025} } ``` ## Acknowledgement Falcon is built with reference to the code of the following projects: [Florence-2-base-ft](https://huggingface.co/microsoft/Florence-2-base-ft), [Florence-2-large-ft](https://huggingface.co/microsoft/Florence-2-large-ft), [florence2-finetuning](https://github.com/andimarafioti/florence2-finetuning). Thanks for their awesome work!