# AstroLayout

**Repository Path**: AstroTJU/AstroLayout

## Basic Information

- **Project Name**: AstroLayout
- **Description**: A Redistribution Tool for Long-Term Archive of Astronomical Observation Data
- **Primary Language**: C++
- **License**: GPL-2.0
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2020-03-28
- **Last Updated**: 2021-04-29

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

## AstroLayout

### Introduction

AstroLayout is a tool to redistribute long-term observation data in astronomy by spatially aggregating,
and it is implemented as a command-line program in Linux platform.
The purpose of AstroLayout is to improve the performance of data access for time domain astronomy research.

The core algorithm of AstroLayout is based on graph partition method
which calculates an optimal data placement solution of long-term astronomical observation data.
Then, AstroLayout can make a copy for long-term archive in the target storage system following the solution.
It supports any type of storage media for the target storage system, such as optical disks, tapes, hard disks, etc.

AstroLayout can logging the execution progress, the error message and the copy result of each FITS file.
And it can serialize the status of the program and restore from the status.

### Prerequisites

* inihcpp: Simple .INI file parser in C++
* CFITSIO: A FITS File Subroutine Library
* HEALPix: Hierarchical Equal Area Iso Latitude pixelation
* METIS: Serial Graph Partitioning and Fill-reducing Matrix Ordering
* cereal: A C++11 library for serialization

### Build

We use CMake to build the executable program, and you can build it as following:

```sh
mkdir build
cd build
cmake ..
make
cd -
```

Then, an executable file "AstroLayout" will be created in the directory 'build'.

### Configuration

A configuration file (config.ini) must be prepared in the woring directory to run AstroLayout.
The configuration file is in configuration format.
A demo can be found in the source. All supported options have been listed in demo config.ini.
```
[GLOBAL]
logfile = run.log       ; for log
nside = 64              ; the parameter NSIDE used for healpix
cereal = status.cereal  ; serialize the status of the system

[SOURCE]                ; configurations of source storage
dirs = ./fits/obslog    ; source directories of source files (delimited by semicolon)
from_obs_log = false    ; read information from the observation log files

[FITS]                  ; configurations of FITS file
ra_keys = RA;ra         ; the potential keywords of right ascension (delimited by semicolon)
dec_keys = DEC;dec      ; the potential keywords of declination (delimited by semicolon)

[OBSLOG]                ; configurations of observation log
obs_log_ext = .txt      ; the extension of the observation log files
header_line = true      ; whether have the header line
filenname_column = 1    ; the column of filename
ra_column = 3           ; the column of right ascension
dec_column = 4          ; the column of declination
default_file_size = 200 ; default size of source files (MB)

[TARGET]                ; configurations of target storage
dirs = /mnt             ; mount points of target devices (delimited by semicolon)
media = tape            ; media type of target device (supported disk/optical/tape)
capacity = 1000000      ; the maximum capacity of each target device (MB)

[DISTRIBUTE]            ; configurations for distribution progress
command = cp            ; copy command
reserve_filepath = true ; whether reserving source path when distribution (only filename if false)

```

Several necessary options are listed here.
* Option 'dirs' in 'SOURCE' set the directories of FITS file to recursively read (delimited by semicolon).
* Option 'from_obs_log' in 'SOURCE' indicates whether to read the informations of FITS files from observation logs or from the FITS files directly.
* Options 'ra_keys' and 'dec_keys' in 'FITS' indicate the keys representing right ascension and declination in the header of FITS files (delimited by semicolon).
* Options in 'OBSLOG' defines the configurations of observation log
* Option 'dirs' in 'TARGET' set the mount points of target devices (delimited by semicolon).
* Option 'capacicty' in 'TARGET' points out the maximum size of FITS file a disk can store.

#### Usage
After configuring config.ini, this tool can execute in command-line. This command shows execute this tool in source directory rather than the build directory.
```sh
./build/astrolayout
```
The program will traversal the directories to collect the spatial information of all FITS files,
generate an undirected weighted graph according to spatial information,
and then use a graph partition tool called METIS to
partition the graph into several balanced parts.
And the data placement can be obtained from the result assignment.

Then, some devices will be required to archive this FITS files.
Mount the devices to target mount points specified,
and distribute the source FITS files into target devices according to the generated data placement.

The process of redistribution may take a long time depending on data size.
The '--resume' parameter let the tool resume from last checkpoint:
```sh
./build/astrolayout --resume
```
If accidents occurred, such as power failure, it can restore the running status and the operation will continue.