# ph-cmarl
**Repository Path**: zhessiah/ph-cmarl
## Basic Information
- **Project Name**: ph-cmarl
- **Description**: No description available
- **Primary Language**: Python
- **License**: MIT
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No
## Statistics
- **Stars**: 2
- **Forks**: 3
- **Created**: 2025-08-14
- **Last Updated**: 2025-11-13
## Categories & Tags
**Categories**: Uncategorized
**Tags**: None
## README
# Physics-Informed Multi-Agent Reinforcement Learning for Distributed Multi-Robot Problems
This repository contains all the supplementary material
associated to the paper with title ["Physics-Informed Multi-Agent
Reinforcement Learning for Distributed Multi-Robot Problems"](http://arxiv.org/abs/2401.00212).
Please check out our project website for more details: https://eduardosebastianrodriguez.github.io/phMARL/.
## Quick start
Our code is tested with ````Ubuntu 24.04```` and ````Python 3.10````. It depends on the Python packages from `requirements.txt`.
Recommended: use conda to create an isolated Python 3.10 environment and install packages with pip.
1. Create and activate conda environment
```bash
conda create -n phmarl python=3.10 -y
conda activate phmarl
```
2. Install Python dependencies
```bash
python -m pip install --upgrade pip
pip install -r requirements.txt # -i https://pypi.tuna.tsinghua.edu.cn/simple/
# If you plan to run the Multi‑Agent MuJoCo experiments, also install mamujoco requirements:
# cd mamujoco
# pip install -r requirements.txt
# cd ..
```
3. Optional external simulators
- Follow official instructions to install/configure MuJoCo or Robotarium if you need those experiments (see the "Dependencies" section below and links in this README).
4. Start training and evaluation
```bash
# Train with default parameters (see parse_args.py to customize)
python training.py
# Evaluate trained policies and generate animations
python evaluation.py
```
Notes:
- For GPU training ensure CUDA and appropriate PyTorch/CUDA packages are installed.
- Edit parse_args.py to change training/evaluation settings (scenario, number of robots, seeds, paths, etc.).
- To persist environment and packages you can export conda environment:
```bash
conda env export -n phmarl > phmarl-environment.yml
```
## Dependencies
The paper includes experiments using the [Multi-Agent MuJoCo platform](https://robotics.farama.org/envs/MaMuJoCo/). The code to replicate the results associated to those experiments adapts the [repository](https://github.com/cyanrain7/TRPO-in-MARL/tree/master) of the paper "[Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning](https://arxiv.org/pdf/2109.11251)", and it can be found in the ```mamujoco``` folder. The requirements for running this part of the code can be found in the folder. For further instructions on how to set up [Multi-Agent MuJoCo](https://robotics.farama.org/envs/MaMuJoCo/), see [https://github.com/openai/mujoco-py](https://github.com/openai/mujoco-py) and [https://github.com/schroederdewitt/multiagent_mujoco](https://github.com/schroederdewitt/multiagent_mujoco).
On the other hand, the paper also includes experiments using [Robotarium](https://www.robotarium.gatech.edu/).The code to replicate the results associated to those experiments can be found in the ```robotarium``` folder. In particular, once Robotarium is configured, one must simply copy the files and paste them inside the ```robotarium_python_simulator``` folder. For further instructions on how to set up [Robotarium for Python](https://www.robotarium.gatech.edu/downloads), see [https://github.com/robotarium/robotarium_python_simulator](https://github.com/robotarium/robotarium_python_simulator).
## Some qualitative results
We propose a novel MARL approach to learn distributed-by-design control policies for general cooperative/competitive multi-robot tasks.
The solution has three key characteristics: (1) we use a port-Hamiltonian description of the multi-robot system and task that respects the network
topology and robot energy conservation laws, leading to a scalable and sampling efficient parameterization
of the control policy; (2) we parameterize the control policy using self-attention neural networks that handle the time-varying
information available at each robot, so we are able to learn the task while achieving invariance with respect to the number of
robots in the multi-robot team; and (3) we propose modifications over a soft actor-critic to overcome non-stationarity issues while
allowing robots to interact with other members of the team, and also avoid value factorization/approximations.
In the following we show some qualitative results from six cooperative/competitive scenarios. They cover a wide variety of cooperative and competitive behaviors such as collision avoidance, navigation, transport, evasion and monitoring.
All the policies are trained with 4 robots.
### Reverse transport
| 4 robots | 8 robots | 12 robots | 16 robots |
|:-------------------------------------------------:|:-----------------------------------------------:|:-----------------------------------------------:|:----------------------------------------------------:|
| 
| 
| 
| 
|
### Sampling
| 3 robots | 5 robots | 7 robots | 9 robots |
|:-------------------------------------------------:|:-----------------------------------------------:|:-----------------------------------------------:|:----------------------------------------------------:|
| 
| 
| 
| 
|
### Navigation
| 4 robots | 5 robots | 6 robots | 8 robots |
|:-------------------------------------------------:|:-----------------------------------------------:|:-----------------------------------------------:|:----------------------------------------------------:|
| 
| 
| 
| 
|
### Food collection
| 3 robots | 6 robots | 12 robots | 24 robots |
|:-------------------------------------------------:|:-----------------------------------------------:|:-----------------------------------------------:|:----------------------------------------------------:|
| 
| 
| 
| 
|
### Grassland
| 6 robots | 12 robots | 24 robots | 48 robots |
|:-------------------------------------------------:|:-----------------------------------------------:|:-----------------------------------------------:|:----------------------------------------------------:|
| 
| 
| 
| 
|
### Adversarial
| 6 robots | 12 robots | 24 robots | 48 robots |
|:-------------------------------------------------:|:-----------------------------------------------:|:-----------------------------------------------:|:----------------------------------------------------:|
| 
| 
| 
| 
|
### Half Cheetah
### Robotarium
| 4 robots (perfect comms) | 8 robots (imperfect comms) | 12 robots (imperfect comms) | 16 robots (imperfect comms) |
|:-------------------------------------------------:|:-----------------------------------------------:|:-----------------------------------------------:|:----------------------------------------------------:|
| 
| 
| 
| 
|
| 
| 
| 
| 
|
## Code
The code is based on ```gym``` and makes use of some of the scenarios from [VMAS: Vectorized Multi-Agent Simulator](https://github.com/proroklab/VectorizedMultiAgentSimulator/tree/main).
````sampling```` scenario is reimplemented to fit the version used in the paper. Besides, the
scenarios with name ````food collection````, ```grassland``` and ```adversaries``` are implementations
of the scenarios used in the paper "[DARL1N: Distributed multi-Agent Reinforcement Learning with One-hop Neighbors](https://github.com/BaoqianWang/IROS22_DARL1N/tree/master)", but parallelized as in the ```vmas``` simulator.
The file ````instructions_scenarios```` includes instructions about how to modify [VMAS](https://github.com/proroklab/VectorizedMultiAgentSimulator/tree/main) to include
the ````sampling````, ````food collection````, ```grassland``` and ```adversaries``` scenarios.
We provide the weights of our policies for the different scenarios in the
folder ````data/````.
You can train your own policies by executing ````python training.py````, tuning
the training parameters in the file ````parse_args.py````. You can run ````python evaluation.py````
to evaluate your trained policies and get some cool animations. The evaluation parameters can also be tuned
in the file ````parse_args.py````.
You can train the policies for the Multi-Agent MuJoCo scenarios by executing the ````train_mujoco.sh```` file. To choose the port-Hamiltonian policy as the actor parameterization, go to ````mamujoco/algorithms/actor_critic```` and turn the
constructor parameter ````ph_MARL```` to ````True````.
## Citation
If you find our papers/code useful for your research, please cite our work as follows.
E. Sebastian, T. Duong, N. Atanasov, E. Montijano, C. Sagues. [Physics-Informed Multi-Agent Reinforcement Learning for Distributed Multi-Robot Problems](https://eduardosebastianrodriguez.github.io/phMARL/). IEEE Transactions on Robotics, 2025
```bibtex
@article{sebastian24phMARL,
author = {Eduardo Sebasti\'{a}n AND Thai Duong AND Nikolay Atanasov AND Eduardo Montijano AND Carlos Sag\"{u}\'{e}s},
title = {{Physics-Informed Multi-Agent Reinforcement Learning for Distributed Multi-Robot Problems}},
journal = {IEEE Transactions on Robotics},
pages={1--19},
year = {2025}
}
```