# Baseline-recmat

**Repository Path**: yuntiy/baseline-recmat

## Basic Information

- **Project Name**: Baseline-recmat
- **Description**: Repository for performance evaluation of Semi-StructMG against hypre's BoomerAMG, SSAMG, Split-PFMG/SMG.
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 1
- **Created**: 2025-09-16
- **Last Updated**: 2025-09-16

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# Performance Evaluation of Semi-Structured Multigrids

This repository is used to compare the performance of various multigrid methods in solving semi-structured grid problems.

Multigrids:
- Split (the underlying structured component can be SMG or PFMG)
- SSAMG
- BoomerAMG
- Semi-StructMG

**Split, SSAMG, and Semi-StructMG are semi-structured-specific multigrids, while BoomerAMG is fully unstructured.**

Six problems are evaluated in this repository. They are listed here and the detailed descriptions are in the following.
- case3: benchmark problem of multi-block structured mesh (MBSM), denoted as "bench-mbsm"
- case4: benchmark problem of structured adaptive-mesh-refinement (SAMR), denoted as "bench-samr"
- oil: petroleum reservoir simulation. 
- kvlcc2: ship manufacturing.
- gfs: numerical weather prediction. Data is from CMA-GFS (global forecasting system of Chinese Meteorological Administration).
- ocean: ocean modeling.


Each problem has two test source files, prefixed with "base_" and "test_," respectively.
Split, SSAMG, and BoomerAMG are from *hypre*, and the source files to invoke them have the prefix "base_", and the corresponding wrapper functions are in "wrapper_hypre.\*".
Semi-StructMG is a recently-developed multigrid, and the source files to invoke it have the prefix "test_", and the corresponding wrapper functions are in "wrapper_sstructmg.\*.

**Since base_\${case_name}.cpp and test_\${case_name}.cpp solve the same problem with input data in the same format using *hypre*'s multigrids and Semi-StructMG, respectively, a comparison between the two source files can help users quickly understand how to use Semi-StructMG effectively.**
Users can also refer to Semi-StructMG's user guide for more information (available at https://gitee.com/zongy17/semi-struct-mg/blob/master/doc/UserGuide.pdf).
**The user guide of Semi-StructMG documents the equivalent API to *hypre* for setting up semi-structured grids, vectors, and matrices**.

### Useful External links
- *hypre*'s recmat branch where SSAMG is available: https://github.com/hypre-space/hypre/tree/recmat
- Semi-StructMG's repository: https://gitee.com/zongy17/semi-struct-mg
- Test data in suitable formats for the source files in this repository: https://cloud.tsinghua.edu.cn/d/995bc2b2929a4aa780d0. Please check the md5sum for the real-world problems listed in the following. The data of benchmark problems are not included, since they could be constructed on-the-fly within the program.




## bench-mbsm: base/test_case3.cpp

This benchmark problem was designed based on idealized Laplace 7pt problem when SSAMG was proposed and evaluated in https://doi.org/10.1137/21M1434118.
Its grid consists of three blocks of the same size that share a common intersection edge, and is a simplified version of the cubed-sphere grid in FV3 (https://www.gfdl.noaa.gov/fv3/fv3-grids/).

Since the settings of this benchmark problem strictly follow the literature, its parameters and the way they construct the corresponding problem size may seem a bit unintuitive.
The problem grid consists of three cube blocks,  with each block involving $p^3$ processes for computation.
Each process computes a box region of size $M^3$.
Therefore, the total length of each dimension in a block `${Tot}` equals to $p * M$.
The total problem size (i.e., total number of rows in the matrix) is $3 * (p * M)^3$, and the total number of processes is $3*p^3$.
The command line parameters are as follows:
```bash
# np: number of processes
# Mi: the size along each dimension of the box that each process computes
# pi: number of processes distributed along each dimension of a block
mpirun -n ${np} test_case3.exe ${M} ${p} CG GMG config/mg.case3.${Tot}.json # Semi-StructMG
mpirun -n ${np} base_case3.exe ${M} ${p} CG ${hypre_prec} # hypre's multigrids
# hypre_prec can be SSAMG-base, AMG, Split-PFMG, where AMG denotes BoomerAMG
```

### Strong-scaling tests
The total problem sizes are unchanged through strong-scaling tests.
The above `${np}` and `${M}`, `${p}`, `${Tot}` for strong-scaling tests are listed in the table.
| np | M | p | Tot |
| -- | --| --| --- |
| 375	| 72 |  5 | 360 |
| 648	| 60 |  6 | 360 |
| 1536	| 45 |  8 | 360 |
| 3000	| 36 | 10 | 360 |
| 5184	| 30 | 12 | 360 |
| 10125	| 24 | 15 | 360 |
### Weak-scaling tests
The total problem sizes increases proportionally with the degrees of parallelism.
The above `${np}` and `${M}`, `${p}`, `${Tot}` for weak-scaling tests are listed in the table.
| np | M | p | Tot |
| -- | --| --| --- |
|  192 | 64 |  4 | 256 |
|  375 | 64 |  5 | 320 |
|  648 | 64 |  6 | 384 |
| 1029 | 64 |  7 | 448 |
| 1536 | 64 |  8 | 512 |
| 2187 | 64 |  9 | 576 |
| 3000 | 64 | 10 | 640 |
| 3993 | 64 | 11 | 704 |






## bench-samr: base/test_case4.cpp

This benchmar problem, similar to bench-mbsm, was designed based on idealized Laplace 7pt problem when SSAMG was proposed and evaluated in https://doi.org/10.1137/21M1434118.
Its grid consists of two blocks of the same size.
The inner block with finer resolution is nested at the center of the outer block with coarser resolution.

Since the settings of this benchmark problem strictly follow the literature, its parameters and the way they construct the corresponding problem size may seem a bit unintuitive.
The problem grid consists of two cube blocks,  with each block involving $p^3$ processes for computation.
Each process computes a box region of size $M^3$.
Therefore, the total length of each dimension in a block `${Tot}` equals to $p * M$.
The total problem size (i.e., total number of rows in the matrix) is $2 * (p * M)^3$, and the total number of processes is $2*p^3$.
The command line parameters are as follows:
```bash
# np: number of processes
# Mi: the size along each dimension of the box that each process computes
# pi: number of processes distributed along each dimension of a block
mpirun -n ${np} test_case4.exe ${M} ${p} CG GMG config/mg.case4.${Tot}.json # Semi-StructMG
mpirun -n ${np} base_case4.exe ${M} ${p} CG ${hypre_prec} # hypre's multigrids
# hypre_prec can be SSAMG-base, AMG, Split-PFMG, where AMG denotes BoomerAMG
```

### Strong-scaling tests
The total problem sizes are unchanged through strong-scaling tests.
The above `${np}` and `${M}`, `${p}`, `${Tot}` for strong-scaling tests are listed in the table.
| np | M | p | Tot |
| -- | --| --| --- |
| 128 | 96 |  4 | 384 |
| 432 | 64 |  6 | 384 |
|1024 | 48 |  8 | 384 |
|3456 | 32 | 12 | 384 |
|8192 | 24 | 16 | 384 |
### Weak-scaling tests
The total problem sizes increases proportionally with the degrees of parallelism.
The above `${np}` and `${M}`, `${p}`, `${Tot}` for weak-scaling tests are listed in the table.
| np | M | p | Tot |
| -- | --| --| --- |
|  128 | 64 | 4  | 256 |
|  250 | 64 | 5  | 320 |
|  432 | 64 | 6  | 384 |
|  686 | 64 | 7  | 448 |
| 1024 | 64 | 8  | 512 |
| 2662 | 64 | 11 | 704 |
| 3456 | 64 | 12 | 768 |







## oil: base/test_oil.cpp
MD5 of oil.tar.gz: ef2299b8bc25fdc949a1e8bf9781bfe9

Oil is from petroleum reservoir simulation.
Settings of SPE1 and SPE10 benchmarks (https://www.spe.org/web/csp/datasets/set02.htm) are combined to generate larger cases via OpenCAEPoro (https://github.com/OpenCAEPlus/OpenCAEPoro/tree/main/examples/spe10).
There are 5 small blocks representing oil wells, and 1 large block representing the ground in the grid.
Inter-block connections (ITBC) reflect the interaction between oil/gas activities and the ground.

The command line parameters are as follows:
```bash
# np: number of processes
# DATA_PATH: path to data files decompressed from the oil.tar.gz
# px: number of processes along the x-dimension of the main structured block
# py: number of processes along the y-dimension of the main structured block
# pz: number of processes along the y-dimension of the main structured block
mpirun -n ${np} test_oil.exe ${px} ${py} ${pz} ${DATA_PATH} GMRES GMG config/mg.oil.json # Semi-StructMG
mpirun -n ${np} base_oil.exe ${px} ${py} ${pz} ${DATA_PATH} GMRES ${hypre_prec} # hypre's multigrids
# hypre_prec can be SSAMG-base, AMG, Split-SMG, where AMG denotes BoomerAMG
```

### Strong-scaling tests
The above `${np}` and `${px}`, `${py}` for strong-scaling tests are listed in the table.
| np   | px  | py  | pz  |
| ---- | --- | --- | --- |
| 120  |  2  | 10  |   6 |
| 240  |  2  | 10  |  12 |
| 480  |  4  | 10  |  12 |
| 960  |  4  | 20  |  12 |
|1920  |  4  | 20  |  24 |
|3840  |  8  | 20  |  24 |






## kvlcc2: base/test_kvlcc2.cpp
MD5 of kvlcc2.tar.gz: e5e24ea8af5e9280d639d3826d4deb71

Kvlcc2 (KRISO Very Large Crude Carrier, https://doi.org/10.5957/jsr.2003.47.1.24) is a standard ship type and widely used in naval architecture and marine engineering.
Its grid consists of 7 blocks to capture the detailed geometry of its hull shape, which includes the bow, stern, and midship sections.
NaviiX (https://taihulab.com.cn/einfo/210.html) software runs the unsteady simulations, where the pressure equation of SIMPLE algorithm is well-suited for multigrid solution.

The command line parameters are as follows:
```bash
# np: number of processes
# DATA_PATH: path to data files decompressed from the kvlcc2.tar.gz
# PROC_ASSIGN: assignment file that specifies the computational domain for each process
mpirun -n ${np} test_kvlcc2.exe ${DATA_PATH} ${PROC_ASSIGN} CG GMG config/mg.kvlcc2.json # Semi-StructMG
mpirun -n ${np} base_kvlcc2.exe ${DATA_PATH} ${PROC_ASSIGN} CG ${hypre_prec} # hypre's multigrids
# hypre_prec can be SSAMG-base, AMG, Split-PFMG, where AMG denotes BoomerAMG
```

### Strong-scaling tests
The above `${np}` and `${PROC_ASSIGN}` for strong-scaling tests are listed in the table.
| np   | PROC_ASSIGN |
| ---- | ----------- |
| 67   | assign.kvlcc2.P67   |
| 130  | assign.kvlcc2.P130  |
| 242  | assign.kvlcc2.P242  |
| 498  | assign.kvlcc2.P498  |
| 996  | assign.kvlcc2.P996  |
| 1992 | assign.kvlcc2.P1992 |







## gfs: base/test_gfs.cpp
MD5 of gfs.tar.gz: 4600becaa1f06b5940d0e92fdbe585db

Gfs arises from numerical weather prediction, provided by the dynamic core of CMA-GFS (https://doi.org/10.1007/s13351-023-3002-0) running in August 2021 with a resolution of 25km.
The latitude-longitude grid of the Earth can be represented by a four-block semi-structured grid.
The cross-polar irregularity occurs in polar regions where two blocks are coupled with the other two blocks, respectively, on the opposite side. 

The command line parameters are as follows:
```bash
# np: number of processes
# DATA_PATH: path to data files decompressed from the gfs.tar.gz
# px: number of processes along the x-dimension of each block
# py: number of processes along the y-dimension of each block
mpirun -n ${np} test_gfs.exe 720 1440 89 ${px} ${py} 1 ${DATA_PATH} GMRES GMG config/mg.gfs.json # Semi-StructMG
mpirun -n ${np} base_gfs.exe 720 1440 89 ${px} ${py} 1 ${DATA_PATH} GMRES ${hypre_prec} # hypre's multigrids
# hypre_prec can be SSAMG-base, AMG, Split-SMG, where AMG denotes BoomerAMG
```

### Strong-scaling tests
The above `${np}` and `${px}`, `${py}` for strong-scaling tests are listed in the table.
| np   | px  | py  |
| ---- | --- | --- |
| 240  |  10 |   6 |
| 480  |  12 |  10 |
| 960  |  60 |   4 |
|1960  |  60 |   8 |
|3840  | 120 |   8 |
|7680  | 240 |   8 |






## ocean: base/test_ocean.cpp
MD5 of ocean.tar.gz: 1767f354900132091258ceb30073f35d

Ocean arises from ocean modeling that solves the incompressible Boussinesq Navier-Stokes equations over the pan-Arctic region.
The discretization of the ocean system on the curvilinear grid, using an implicit linear free-surface, eventually forms a Helmholtz equation (https://doi.org/10.1029/96JC02775).
To represent the intricate geometry of land-sea boundaries, the sea surface is partitioned with a high-resolution Cartesian grid, in which only the oceanic regions are active in computations.

The command line parameters are as follows:
```bash
# np: number of processes
# DATA_PATH: path to data files decompressed from the ocean.tar.gz 
# Tx: tile size (i.e., box size) along the x-dimension of each process
# Ty: tile size (i.e., box size) along the y-dimension of each process
mpirun -n ${np} test_ocean.exe 14580 11520 ${Tx} ${Ty} ${DATA_PATH} GMRES GMG config/mg.ocean.json # Semi-StructMG
mpirun -n ${np} base_ocean.exe 14580 11520 ${Tx} ${Ty} ${DATA_PATH} GMRES ${hypre_prec} # hypre's multigrids
# hypre_prec can be AMG, PFMG, SMG, where AMG denotes BoomerAMG
```

### Strong-scaling tests
The above `${np}` and `${Tx}`, `${Ty}` for strong-scaling tests are listed in the table. Note that `${Tx}` and `{Ty}` are not numbers of processes, but the sizes that each process computes. They become smaller when more degrees of parallelism are used to solve the problem. To be specific, `${Tx}*${Ty}` is the box size that each process computes. 
| np   | Tx  | Ty  |
| ---- | --- | --- |
|228   | 540 | 960 |
|421   | 540 | 480 |
|812   | 270 | 480 |
|1199  | 180 | 480 |
|2397  | 108 | 384 |
|4536  | 108 | 192 |
|8714  | 108 | 96  |