# PPOGAE

**Repository Path**: nonomi-3/ppogae

## Basic Information

- **Project Name**: PPOGAE
- **Description**: 强化学习作业
实现PPO算法，添加了GAE的内容作对比。
- **Primary Language**: Unknown
- **License**: MIT
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2026-06-22
- **Last Updated**: 2026-07-06

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# PPO PyTorch Implementation

A PyTorch-based implementation of the Proximal Policy Optimization (PPO) reinforcement learning algorithm, supporting both discrete and continuous action spaces.

## Project Overview

This project implements the PPO algorithm and its variants using the PyTorch framework, designed for training agents across various reinforcement learning environments. It includes comprehensive training, testing, and visualization functionalities.

## Key Features

- **PPO Algorithm**: Implements the standard PPO algorithm with an Actor-Critic architecture.
- **GAE Support**: Supports Generalized Advantage Estimation (GAE) for advantage estimation.
- **Multi-Environment Support**: Compatible with various environments from OpenAI Gym and Gymnasium.
- **Continuous/Discrete Actions**: Supports both continuous and discrete action spaces.
- **Model Save/Load**: Enables saving and loading of model checkpoints during training.
- **Visualization**: Generates training curves and GIF animations.

## Core Components

### PPO.py / PPO_GAE.py

- `ActorCritic`: Actor-Critic neural network model.
- `RolloutBuffer`: Buffer for storing trajectory data.
- `PPO`: Main PPO algorithm class containing training and inference logic.

### train.py

Training script to train a PPO agent in a specified environment.

### test.py

Testing script to evaluate the performance of a trained PPO agent.

### make_gif.py

Generates GIF animations from environment renderings during training.

### plot_graph.py

Plots training curves based on logged data.

## Requirements

- Python 3.x
- PyTorch
- Gymnasium or OpenAI Gym
- NumPy
- Matplotlib (for visualization)

## Quick Start

### Train an Agent

```python
python train.py --config configs/LunarLander-v3.json
```

### Test an Agent

```python
python test.py --config configs/LunarLander-v3.json
```

### Generate GIF

```python
python make_gif.py --env_name LunarLander-v3
```

## Configuration Details

Configuration files (in JSON format) include the following parameters:

| Parameter | Description |
|-----------|-------------|
| env_name | Environment name |
| has_continuous_action_space | Whether the action space is continuous |
| max_ep_len | Maximum number of steps per episode |
| K_epochs | Number of epochs per update |
| eps_clip | PPO clipping range |
| gamma | Discount factor |
| lr_actor | Actor learning rate |
| lr_critic | Critic learning rate |
| action_std | Action standard deviation |

## Pretrained Models

Pretrained models are provided in the `PPO_preTrained/` directory:

- CartPole-v1
- LunarLander-v2 / LunarLander-v3
- BipedalWalker-v2
- MountainCar-v0

## Project Structure

```
.
├── PPO.py              # PPO algorithm implementation
├── PPO_GAE.py          # PPO + GAE implementation
├── train.py            # Training script
├── test.py             # Testing script
├── make_gif.py         # GIF generation script
├── plot_graph.py       # Plotting script
├── configs/           # Configuration files
├── PPO_preTrained/    # Pretrained models
├── PPO_logs/          # Training logs
├── PPO_figs/          # Training curves
└── PPO_gifs/          # GIF animations
```

## License

MIT License