# optimizers

**Repository Path**: mirrors_facebookresearch/optimizers

## Basic Information

- **Project Name**: optimizers
- **Description**: For optimization algorithm research and development.
- **Primary Language**: Unknown
- **License**: BSD-3-Clause
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2022-06-16
- **Last Updated**: 2026-05-30

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# Optimizers

[![Python3.12](https://img.shields.io/badge/python-3.12-blue.svg)](https://www.python.org/downloads/)
[![tests](https://github.com/facebookresearch/optimizers/actions/workflows/tests.yaml/badge.svg)](https://github.com/facebookresearch/optimizers/actions/workflows/tests.yaml?query=branch%3Amain)
[![gpu-tests](https://github.com/facebookresearch/optimizers/actions/workflows/gpu-tests.yaml/badge.svg)](https://github.com/facebookresearch/optimizers/actions/workflows/gpu-tests.yaml?query=branch%3Amain)
[![pre-commit](https://github.com/facebookresearch/optimizers/actions/workflows/pre-commit.yaml/badge.svg)](https://github.com/facebookresearch/optimizers/actions/workflows/pre-commit.yaml?query=branch%3Amain)
[![type-checking](https://github.com/facebookresearch/optimizers/actions/workflows/type-check.yaml/badge.svg)](https://github.com/facebookresearch/optimizers/actions/workflows/type-check.yaml?query=branch%3Amain)
[![examples](https://github.com/facebookresearch/optimizers/actions/workflows/examples.yaml/badge.svg)](https://github.com/facebookresearch/optimizers/actions/workflows/examples.yaml?query=branch%3Amain)
[![license](https://img.shields.io/badge/license-BSD--Clause-lightgrey.svg)](./LICENSE)


*Copyright (c) Meta Platforms, Inc. and affiliates.
All rights reserved.*

## Description
Optimizers is a Github repository of PyTorch optimization algorithms. It is designed for external collaboration and development.

Currently includes the optimizers:
- Distributed Shampoo
- GPA-AdamW (Generalized Primal Averaging)

See the [CONTRIBUTING](CONTRIBUTING.md) file for how to help out.

## License
Optimizers is released under the [BSD license](LICENSE).

## Installation and Dependencies
Install with all dependencies:
```bash
git clone git@github.com:facebookresearch/optimizers.git
cd optimizers
pip install .
```
If you also want to try the examples ([Distributed Shampoo](./distributed_shampoo/examples/), [GPA-AdamW](./gpa/examples/)), replace the last line with `pip install ".[examples]"`.

## Usage

### Distributed Shampoo

```python
import torch
from distributed_shampoo import AdamPreconditionerConfig, DistributedShampoo

model = ...  # Instantiate model

optim = DistributedShampoo(
    model.parameters(),
    lr=1e-3,
    betas=(0.9, 0.999),
    epsilon=1e-8,
    grafting_config=AdamPreconditionerConfig(
        beta2=0.999,
        epsilon=1e-8,
    ),
)
```

For more, please see the [additional documentation here](./distributed_shampoo/README.md) and especially the [How to Use](./distributed_shampoo/README.md#how-to-use) section.

### GPA-AdamW

```python
import torch
from gpa.gpa_adamw import GPAAdamW

model = ...  # Instantiate model

optim = GPAAdamW(
    model.parameters(),
    lr=1e-3,
    train_interp_coeff=0.7,
    eval_interp_coeff=0.9967,  # GPA mode (DiLoCo-equivalent of 32 local steps)
)
```

GPA-AdamW requires switching between train and eval modes via `optimizer.train()` /
`optimizer.eval()` so that gradients are computed on the y-sequence and evaluation
runs on the x-sequence. For full details (Schedule-Free vs GPA mode, hyperparameter
tuning, distributed training), see the [GPA-AdamW README](./gpa/README.md).