# The Lightweight OpenAI API Server
### ๐ Local Inference Without Dependencies ๐
[](https://opensource.org/licenses/MIT)
[](https://github.com/Michael-A-Kuykendall/shimmy/security)
[](https://crates.io/crates/shimmy)
[](https://crates.io/crates/shimmy)
[](https://rustup.rs/)
[](https://github.com/Michael-A-Kuykendall/shimmy/stargazers)
[](https://github.com/sponsors/Michael-A-Kuykendall)
**Shimmy will be free forever.** No asterisks. No "free for now." No pivot to paid.
### ๐ Support Shimmy's Growth
๐ **If Shimmy helps you, consider [sponsoring](https://github.com/sponsors/Michael-A-Kuykendall) โ 100% of support goes to keeping it free forever.**
- **$5/month**: Coffee tier โ - Eternal gratitude + sponsor badge
- **$25/month**: Bug prioritizer ๐ - Priority support + name in [SPONSORS.md](SPONSORS.md)
- **$100/month**: Corporate backer ๐ข - Logo placement + monthly office hours
- **$500/month**: Infrastructure partner ๐ - Direct support + roadmap input
[**๐ฏ Become a Sponsor**](https://github.com/sponsors/Michael-A-Kuykendall) | See our amazing [sponsors](SPONSORS.md) ๐
---
## Drop-in OpenAI API Replacement for Local LLMs
Shimmy is a **single-binary** that provides **100% OpenAI-compatible endpoints** for GGUF models. Point your existing AI tools to Shimmy and they just work โ locally, privately, and free.
**๐ NEW in v1.9.0**: One download, all GPU backends included! No compilation, no backend confusion - just download and run.
## Developer Tools
Whether you're forking Shimmy or integrating it as a service, we provide complete documentation and integration templates.
### Try it in 30 seconds
```bash
# 1) Download pre-built binary (includes all GPU backends)
# Windows:
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-windows-x86_64.exe -o shimmy.exe
./shimmy.exe serve &
# Linux:
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-linux-x86_64 -o shimmy && chmod +x shimmy
./shimmy serve &
# macOS (Apple Silicon):
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-macos-arm64 -o shimmy && chmod +x shimmy
./shimmy serve &
# 2) See models and pick one
./shimmy list
# 3) Smoke test the OpenAI API
curl -s http://127.0.0.1:11435/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model":"REPLACE_WITH_MODEL_FROM_list",
"messages":[{"role":"user","content":"Say hi in 5 words."}],
"max_tokens":32
}' | jq -r '.choices[0].message.content'
```
## ๐ Compatible with OpenAI SDKs and Tools
**No code changes needed** - just change the API endpoint:
- **Any OpenAI client**: Python, Node.js, curl, etc.
- **Development applications**: Compatible with standard SDKs
- **VSCode Extensions**: Point to `http://localhost:11435`
- **Cursor Editor**: Built-in OpenAI compatibility
- **Continue.dev**: Drop-in model provider
### Use with OpenAI SDKs
- Node.js (openai v4)
```ts
import OpenAI from "openai";
const openai = new OpenAI({
baseURL: "http://127.0.0.1:11435/v1",
apiKey: "sk-local", // placeholder, Shimmy ignores it
});
const resp = await openai.chat.completions.create({
model: "REPLACE_WITH_MODEL",
messages: [{ role: "user", content: "Say hi in 5 words." }],
max_tokens: 32,
});
console.log(resp.choices[0].message?.content);
```
- Python (openai>=1.0.0)
```python
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:11435/v1", api_key="sk-local")
resp = client.chat.completions.create(
model="REPLACE_WITH_MODEL",
messages=[{"role": "user", "content": "Say hi in 5 words."}],
max_tokens=32,
)
print(resp.choices[0].message.content)
```
## โก Zero Configuration Required
- **Automatically finds models** from Hugging Face cache, Ollama, local dirs
- **Auto-allocates ports** to avoid conflicts
- **Auto-detects LoRA adapters** for specialized models
- **Just works** - no config files, no setup wizards
## ๐ง Advanced MOE (Mixture of Experts) Support
**Run 70B+ models on consumer hardware** with intelligent CPU/GPU hybrid processing:
- **๐ CPU MOE Offloading**: Automatically distribute model layers across CPU and GPU
- **๐งฎ Intelligent Layer Placement**: Optimizes which layers run where for maximum performance
- **๐พ Memory Efficiency**: Fit larger models in limited VRAM by using system RAM strategically
- **โก Hybrid Acceleration**: Get GPU speed where it matters most, CPU reliability everywhere else
- **๐๏ธ Configurable**: `--cpu-moe` and `--n-cpu-moe` flags for fine control
```bash
# Enable MOE CPU offloading during installation
cargo install shimmy --features moe
# Run with MOE hybrid processing
shimmy serve --cpu-moe --n-cpu-moe 8
# Automatically balances: GPU layers (fast) + CPU layers (memory-efficient)
```
**Perfect for**: Large models (70B+), limited VRAM systems, cost-effective inference
## ๐ฏ Perfect for Local Development
- **Privacy**: Your code never leaves your machine
- **Cost**: No API keys, no per-token billing
- **Speed**: Local inference, sub-second responses
- **Reliability**: No rate limits, no downtime
## Quick Start (30 seconds)
### Installation
**โจ v1.9.0 NEW**: Download pre-built binaries with ALL GPU backends included!
#### **๐ฅ Pre-Built Binaries (Recommended - Zero Dependencies)**
Pick your platform and download - no compilation needed:
```bash
# Windows x64 (includes CUDA + Vulkan + OpenCL)
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-windows-x86_64.exe -o shimmy.exe
# Linux x86_64 (includes CUDA + Vulkan + OpenCL)
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-linux-x86_64 -o shimmy && chmod +x shimmy
# macOS ARM64 (includes MLX for Apple Silicon)
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-macos-arm64 -o shimmy && chmod +x shimmy
# macOS Intel (CPU-only)
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-macos-intel -o shimmy && chmod +x shimmy
# Linux ARM64 (CPU-only)
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-linux-aarch64 -o shimmy && chmod +x shimmy
```
**That's it!** Your GPU will be detected automatically at runtime.
#### **๐ ๏ธ Build from Source (Advanced)**
Want to customize or contribute?
```bash
# Basic installation (CPU only)
cargo install shimmy --features huggingface
# Kitchen Sink builds (what pre-built binaries use):
# Windows/Linux x64:
cargo install shimmy --features huggingface,llama,llama-cuda,llama-vulkan,llama-opencl,vision
# macOS ARM64:
cargo install shimmy --features huggingface,llama,mlx,vision
# CPU-only (any platform):
cargo install shimmy --features huggingface,llama,vision
```
> **โ ๏ธ Build Notes**:
> - **Windows**: Install [LLVM](https://releases.llvm.org/download.html) first for libclang.dll
> - **Recommended**: Use pre-built binaries to avoid dependency issues
> - **Advanced users only**: Building from source requires C++ compiler + CUDA/Vulkan SDKs
### GPU Acceleration
**โจ NEW in v1.9.0**: One binary per platform with automatic GPU detection!
> **โ ๏ธ IMPORTANT - Vision Feature Performance**:
> CPU-based vision inference (MiniCPM-V) is **5-10x slower** than GPU acceleration.
> **CPU**: 15-45 seconds per image | **GPU (CUDA/Vulkan)**: 2-8 seconds per image
> **For production vision workloads, GPU acceleration is strongly recommended.**
#### **๐ฅ Download Pre-Built Binaries (Recommended)**
No compilation needed! Each binary includes ALL GPU backends for your platform:
| Platform | Download | GPU Support | Auto-Detects |
|----------|----------|-------------|--------------|
| **Windows x64** | [shimmy-windows-x86_64.exe](https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-windows-x86_64.exe) | CUDA + Vulkan + OpenCL | โ
|
| **Linux x86_64** | [shimmy-linux-x86_64](https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-linux-x86_64) | CUDA + Vulkan + OpenCL | โ
|
| **macOS ARM64** | [shimmy-macos-arm64](https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-macos-arm64) | MLX (Apple Silicon) | โ
|
| **macOS Intel** | [shimmy-macos-intel](https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-macos-intel) | CPU only | N/A |
| **Linux ARM64** | [shimmy-linux-aarch64](https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-linux-aarch64) | CPU only | N/A |
**How it works**: Download one file, run it. Shimmy automatically detects and uses your GPU!
```bash
# Windows example
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-windows-x86_64.exe -o shimmy.exe
./shimmy.exe serve --gpu-backend auto # Auto-detects CUDA/Vulkan/OpenCL
# Linux example
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-linux-x86_64 -o shimmy
chmod +x shimmy
./shimmy serve --gpu-backend auto # Auto-detects CUDA/Vulkan/OpenCL
# macOS ARM64 example
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy-macos-arm64 -o shimmy
chmod +x shimmy
./shimmy serve # Auto-detects MLX on Apple Silicon
```
#### **๐ฏ GPU Auto-Detection**
Shimmy uses intelligent GPU detection with this priority order:
1. **CUDA** (NVIDIA GPUs via nvidia-smi)
2. **Vulkan** (Cross-platform GPUs via vulkaninfo)
3. **OpenCL** (AMD/Intel GPUs via clinfo)
4. **MLX** (Apple Silicon via system detection)
5. **CPU** (Fallback if no GPU detected)
**No manual configuration needed!** Just run with `--gpu-backend auto` (default).
#### **๐ง Manual Backend Override**
Want to force a specific backend? Use the `--gpu-backend` flag:
```bash
# Auto-detect (default - recommended)
shimmy serve --gpu-backend auto
# Force CPU (for testing or compatibility)
shimmy serve --gpu-backend cpu
# Force CUDA (NVIDIA GPUs only)
shimmy serve --gpu-backend cuda
# Force Vulkan (AMD/Intel/Cross-platform)
shimmy serve --gpu-backend vulkan
# Force OpenCL (AMD/Intel alternative)
shimmy serve --gpu-backend opencl
```
**๐ก๏ธ Error Handling & Robustness**: If you force an unavailable backend (e.g., `--gpu-backend cuda` on AMD GPU), Shimmy will:
1. โ
Display clear error message explaining the issue
2. โ
Automatically fallback to next available backend in priority order
3. โ
Log which backend was actually used (check with `--verbose`)
4. โ
Continue serving requests (graceful degradation, no crashes)
5. โ
Support environment variable override: `SHIMMY_GPU_BACKEND=cuda`
**Common scenarios**:
- `--gpu-backend cuda` on non-NVIDIA โ Falls back to Vulkan or OpenCL
- `--gpu-backend vulkan` without drivers โ Falls back to OpenCL or CPU
- `--gpu-backend invalid` โ Clear error + fallback to auto-detection
- No GPU detected โ Runs on CPU with performance warning
**Environment Variable**: Set `SHIMMY_GPU_BACKEND=cuda` to override default without CLI flags.
#### **๐ Check GPU Support**
```bash
# Show detected GPU backends
shimmy gpu-info
# Check which backend is being used
shimmy serve --gpu-backend auto --verbose
```
#### **โก Binary Sizes**
- **GPU-enabled binaries** (Windows/Linux x64, macOS ARM64): ~40-50MB
- **CPU-only binaries** (macOS Intel, Linux ARM64): ~20-30MB
Trade-off: Slightly larger binaries for zero compilation and automatic GPU detection.
#### **๐ ๏ธ Build from Source (Advanced)**
Want to customize or contribute? Build from source:
- Multiple backends can be compiled in, best one selected automatically
- Use `--gpu-backend