# SLAM-LLM **Repository Path**: zhanghanweii/SLAM-LLM ## Basic Information - **Project Name**: SLAM-LLM - **Description**: No description available - **Primary Language**: Unknown - **License**: MIT - **Default Branch**: main - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 0 - **Forks**: 0 - **Created**: 2024-06-21 - **Last Updated**: 2024-06-21 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README

SLAM-LLM

SLAM-LLM is a deep learning toolkit that allows researchers and developers to train custom multimodal large language model (MLLM), focusing on Speech, Language, Audio, Music processing. We provide detailed recipes for training and high-performance checkpoints for inference.

SLAM-LLM Logo

version version python mit
# Table of Contents 1. [News](#news) 2. [Installation](#installation) 3. [Uasge](#uasge) - [List of Recipes](#list-of-recipes) - [Configuration Priority](#configuration-priority) 4. [Features](#features) 5. [Acknowledge](#acknowledge) # News - [Update Jun. 12, 2024] Recipes for [MaLa-ASR](examples/mala_asr_slidespeech/README.md) has been supported. - **[CALL FOR EXAMPLE]** We sincerely invite developers and researchers to develop new applications, conduct academic research based on SLAM-LLM, and pull request your examples! We also acknowledge engineering PR (such as improving and speeding up multi-node training). - [Update May. 22, 2024] Please join [slack](https://join.slack.com/t/slam-llm/shared_invite/zt-2jbuiyqgi-O83DteBG36xYWcjCNcLWqQ) or [WeChat group](./docs/Wechat.jpg). We will sync our updates and Q&A here. - [Update May. 21, 2024] Recipes for [Spatial Audio Understanding](examples/seld_spatialsoundqa/README.md) has been supported. - [Update May. 20, 2024] Recipes for [music caption (MC)](examples/mc_musiccaps/README.md) has been supported. - [Update May. 8, 2024] Recipes for [visual speech recognition (VSR)](examples/vsr_LRS3/README.md) has been supported. - [Update May. 4, 2024] Recipes for [zero-shot text-to-speech (TTS)](examples/vallex/README.md) has been supported. - [Update Apr. 28, 2024] Recipes for [automated audio captioning (AAC)](examples/aac_audiocaps/README.md) has been supported. - [Update Mar. 31, 2024] Recipes for [automatic speech recognition (ASR)](examples/asr_librispeech/README.md) has been supported. # Installation ```bash git clone https://github.com/huggingface/transformers.git cd transformers git checkout tags/v4.35.2 pip install -e . cd .. git clone https://github.com/huggingface/peft.git cd peft git checkout tags/v0.6.0 pip install -e . cd .. pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 --index-url https://download.pytorch.org/whl/cu118 git clone https://github.com/ddlBoJack/SLAM-LLM.git cd SLAM-LLM pip install -e . ``` For some examples, you may need to use `fairseq`, the command line is as follows: ``` # you need to install fairseq before SLAM-LLM git clone https://github.com/pytorch/fairseq cd fairseq pip install --editable ./ ``` We also provide a docker image for convenience: ```shell # build docker image docker build -t slam-llm:latest . # run docker image with gpu docker run -it --gpus all --name slam --shm-size=256g slam-llm:latest /bin/bash ``` # Usage ## List of Recipes We provide reference implementations of various LLM-based speech, audio, and music tasks: - **Speech Task** - [Automatic Speech Recognition (ASR)](examples/asr_librispeech/README.md) - [Text-to-Speech (TTS)](examples/vallex/README.md) - [Visual Speech Recognition (VSR)](examples/vsr_LRS3/README.md) - **Audio Task** - [Automated Audio Captioning (AAC)](examples/aac_audiocaps/README.md) - [Spatial Audio Understanding](examples/seld_spatialsoundqa/README.md) - **Music Task** - [Music Caption (MC)](examples/mc_musiccaps/README.md) ## Configuration Priority We provide hierarchical configuration inheritance relationships as follows: ``` command-line (shell file) > Hydra configuration (yaml file) > dataclass configuration (Python file) ``` # Features - Easily extend to new models and tasks. - Detailed recipes for training and high-performance checkpoints for inference. - Mixed precision training which trains faster with less GPU memory on NVIDIA tensor cores. - Multi-GPU training with data and model parallel, supporting [DDP](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html), [FSDP](https://pytorch.org/tutorials/intermediate/FSDP_tutorial.html) and [deepspeed](https://github.com/microsoft/DeepSpeed) (still need to be improved). - Flexible configuration based on [Hydra](https://github.com/facebookresearch/hydra) and [dataclass](https://docs.python.org/3/library/dataclasses.html) allowing a combination of code, command-line and file based configuration. # Acknowledge - We borrow code from [Llama-Recipes](https://github.com/meta-llama/llama-recipes) for the training process. - We borrow code from [Fairseq](https://github.com/facebookresearch/fairseq) for deepspeed configuration. - We thank the contributors for providing diverse recipes.