# BentoMoshi

**Repository Path**: qq89287286/BentoMoshi

## Basic Information

- **Project Name**: BentoMoshi
- **Description**: No description available
- **Primary Language**: Unknown
- **License**: Apache-2.0
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2025-10-24
- **Last Updated**: 2025-10-24

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

<div align="center">
    <h1 align="center">Self-host Moshi with BentoML</h1>
</div>

[Moshi](https://github.com/kyutai-labs/moshi) is a speech-text foundation model and full-duplex spoken dialogue framework. This is a BentoML example project, showing you how to serve and deploy Moshi with BentoML. Specifically, it creates a real-time voice chat application by implementing a WebSocket endpoint for bi-directional audio streaming.

Here is the workflow after you start the server:

1. You speak into your microphone. The client records the audio and sends it to the server in real-time via a WebSocket connection.
2. The server uses the Mimi model to process the audio and the Moshi language model to generate both text and audio responses.
3. The server sends the generated text and audio back to the client.
4. The client plays the audio through your speakers and displays the text in the terminal.

Check out [the full list](https://docs.bentoml.com/en/latest/examples/overview.html) of example projects to explore more BentoML use cases.

## Prerequisites

If you want to test the Service locally, we recommend you use an Nvidia GPU with at least 24Gb VRAM.

## Instructions

1. Install `uv`.

   ```bash
   curl -LsSf https://astral.sh/uv/install.sh | sh
   ```

2. Clone the project directory.

   ```bash
   git clone https://github.com/bentoml/BentoMoshi.git && cd BentoMoshi
   ```

3. Try local serving:

   ```bash
   # option 1: bentoml serve [RECOMMENDED]
   uvx --with-editable . bentoml serve . --debug

   # option 2: script
   uvx --from . server
   ```

4. The server will be running at `http://localhost:3000`. To connect to the WebSocket endpoint, use the following:

   ```bash
   URL=http://localhost:3000 uvx --from . client
   ```

## Deploy to BentoCloud

You can deploy this project to BentoCloud for better management and scalability. [Sign up](https://www.bentoml.com/) if you haven't got a BentoCloud account.

Make sure you have [logged in to BentoCloud](https://docs.bentoml.com/en/latest/bentocloud/how-tos/manage-access-token.html).

```bash
bentoml cloud login
```

Deploy it to BentoCloud.

```bash
uvx --with-editable . bentoml deploy .
```

After deployment, specify the `URL` on BentoCloud and use the client:

```bash
# option 1: uvx [RECOMMENDED]
URL=<bentocloud-endpoint> uv run --with-editable . bentomoshi/client.py

# option 2: using python
URL=<bentocloud-endpoint> python bentomoshi/client.py
```

**Note**: For custom deployment in your own infrastructure, use [BentoML to generate an OCI-compliant image](https://docs.bentoml.com/en/latest/guides/containerization.html).