代码拉取完成，页面将自动刷新
SpeechBrain is an open-source and all-in-one conversational AI toolkit based on PyTorch.
The goal is to create a single, flexible, and user-friendly toolkit that can be used to easily develop state-of-the-art speech technologies, including systems for speech recognition, speaker recognition, speech enhancement, speech separation, language identification, multi-microphone signal processing, and many others.
Please, star our project on github (see top-right corner) if you appreciate our contribution to the community!
SpeechBrain is currently in beta.
| Tutorials | Website | Documentation | Contributing | HuggingFace |
SpeechBrain provides various useful tools to speed up and facilitate research on speech and language technologies:
(HuggingFace) in our official organization account. These models are coupled with easy-inference interfaces that facilitate their use. To help everyone replicate our results, we also provide all the experimental results and folders (including logs, training curves, etc.) in a shared Google Drive folder.
Brain class is a fully-customizable tool for managing training and evaluation loops over data. The annoying details of training loops are handled for you while retaining complete flexibility to override any part of the process when needed.SpeechBrain supports state-of-the-art methods for end-to-end speech recognition:
dataset to facilitate the training over a large text dataset.SpeechBrain provides efficient (GPU-friendly) speech augmentation and feature extraction pipelines:
SpeechBrain provides different models for speaker recognition, identification, and diarization on different datasets:
We have models for converting characters into a sequence of phonemes. In particular, we have Transformer- and RNN-based models operating at the sentence level (i.e, converting a full sentence into a corresponding sequence of phonemes). The models are trained with both data from Wikipedia and LibriSpeech.
SpeechBrain provides different models for language identification. In particular, our best model is based on an ECAPA-TDNN trained with the voxlingua107 dataset.
Combining multiple microphones is a powerful approach to achieving robustness in adverse acoustic environments:
The recipes released with speechbrain implement speech processing systems with competitive or state-of-the-art performance. In the following, we report the best performance achieved on some popular benchmarks:
| Dataset | Task | System | Performance |
|---|---|---|---|
| LibriSpeech | Speech Recognition | wav2vec2 | WER=1.90% (test-clean) |
| LibriSpeech | Speech Recognition | CNN + Transformer | WER=2.26% (test-clean) |
| TIMIT | Speech Recognition | CRDNN + distillation | PER=13.1% (test) |
| TIMIT | Speech Recognition | wav2vec2 + CTC/Att. | PER=8.04% (test) |
| CommonVoice (English) | Speech Recognition | wav2vec2 + CTC | WER=15.69% (test) |
| CommonVoice (French) | Speech Recognition | wav2vec2 + CTC | WER=9.96% (test) |
| CommonVoice (Italian) | Speech Recognition | wav2vec2 + seq2seq | WER=9.86% (test) |
| CommonVoice (Kinyarwanda) | Speech Recognition | wav2vec2 + seq2seq | WER=18.91% (test) |
| AISHELL (Mandarin) | Speech Recognition | wav2vec2 + seq2seq | CER=5.58% (test) |
| Fisher-callhome (spanish) | Speech translation | conformer (ST + ASR) | BLEU=48.04 (test) |
| VoxCeleb2 | Speaker Verification | ECAPA-TDNN | EER=0.80% (vox1-test) |
| AMI | Speaker Diarization | ECAPA-TDNN | DER=3.01% (eval) |
| VoiceBank | Speech Enhancement | MetricGAN+ | PESQ=3.08 (test) |
| WSJ2MIX | Speech Separation | SepFormer | SDRi=22.6 dB (test) |
| WSJ3MIX | Speech Separation | SepFormer | SDRi=20.0 dB (test) |
| WHAM! | Speech Separation | SepFormer | SDRi= 16.4 dB (test) |
| WHAMR! | Speech Separation | SepFormer | SDRi= 14.0 dB (test) |
| Libri2Mix | Speech Separation | SepFormer | SDRi= 20.6 dB (test-clean) |
| Libri3Mix | Speech Separation | SepFormer | SDRi= 18.7 dB (test-clean) |
| LibryParty | Voice Activity Detection | CRDNN | F-score=0.9477 (test) |
| IEMOCAP | Emotion Recognition | wav2vec | Accuracy=79.8% (test) |
| CommonLanguage | Language Recognition | ECAPA-TDNN | Accuracy=84.9% (test) |
| Timers and Such | Spoken Language Understanding | CRDNN | Intent Accuracy=89.2% (test) |
| SLURP | Spoken Language Understanding | CRDNN | Intent Accuracy=87.54% (test) |
| VoxLingua 107 | Identification | ECAPA-TDNN | Sentence Accuracy=93.3% (test) |
For more details, take a look at the corresponding implementation in recipes/dataset/.
Beyond providing recipes for training the models from scratch, SpeechBrain shares several pre-trained models (coupled with easy-inference functions) on HuggingFace. In the following, we report some of them:
| Task | Dataset | Model |
|---|---|---|
| Speech Recognition | LibriSpeech | CNN + Transformer |
| Speech Recognition | LibriSpeech | CRDNN |
| Speech Recognition | CommonVoice(English) | wav2vec + CTC |
| Speech Recognition | CommonVoice(French) | wav2vec + CTC |
| Speech Recognition | CommonVoice(Italian) | wav2vec + CTC |
| Speech Recognition | CommonVoice(Kinyarwanda) | wav2vec + CTC |
| Speech Recognition | AISHELL(Mandarin) | wav2vec + CTC |
| Text-to-Speech | LJSpeech | Tacotron2 |
| Speaker Recognition | Voxceleb | ECAPA-TDNN |
| Speech Separation | WHAMR! | SepFormer |
| Speech Enhancement | Voicebank | MetricGAN+ |
| Speech Enhancement | WHAMR! | SepFormer |
| Spoken Language Understanding | Timers and Such | CRDNN |
| Language Identification | CommonLanguage | ECAPA-TDNN |
The full list of pre-trained models can be found on HuggingFace
SpeechBrain is designed to speed up the research and development of speech technologies. Hence, our code is backed-up with different levels of documentation:
We are currently implementing speech synthesis pipelines and real-time speech processing pipelines. An interface with the Finite State Transducers (FST) implemented by the Kaldi 2 team is under development.
SpeechBrain has been presented at Interspeech 2021 and 2022 as well as ASRU 2021. When possible, we will provide some ressources here:
SpeechBrain is constantly evolving. New features, tutorials, and documentation will appear over time. SpeechBrain can be installed via PyPI. Moreover, a local installation can be used by those users that what to run experiments and modify/customize the toolkit. SpeechBrain supports both CPU and GPU computations. For most all the recipes, however, a GPU is necessary during training. Please note that CUDA must be properly installed to use GPUs.
Once you have created your Python environment (Python 3.7+) you can simply type:
pip install speechbrain
Then you can access SpeechBrain with:
import speechbrain as sb
Once you have created your Python environment (Python 3.7+) you can simply type:
git clone https://github.com/speechbrain/speechbrain.git
cd speechbrain
pip install -r requirements.txt
pip install --editable .
Then you can access SpeechBrain with:
import speechbrain as sb
Any modification made to the speechbrain package will be automatically interpreted as we installed it with the --editable flag.
Please, run the following script to make sure your installation is working:
pytest tests
pytest --doctest-modules speechbrain
In SpeechBrain, you can run experiments in this way:
> cd recipes/<dataset>/<task>/
> python experiment.py params.yaml
The results will be saved in the output_folder specified in the yaml file. The folder is created by calling sb.core.create_experiment_directory() in experiment.py. Both detailed logs and experiment outputs are saved there. Furthermore, less verbose logs are output to stdout.
As a community-based and open-source project, SpeechBrain needs the help of its community to grow in the right direction. Opening the roadmap to our users enables the toolkit to benefit from new ideas, new research axes, or even new technologies. The roadmap will be available in our GitHub Discussions and will list all the changes and updates that need to be done in the current version of SpeechBrain. Users are more than welcome to propose new items via new Discussions topics!
We provide users with different resources to learn how to use SpeechBrain:
SpeechBrain is released under the Apache License, version 2.0. The Apache license is a popular BSD-like license. SpeechBrain can be redistributed for free, even for commercial purposes, although you can not take off the license headers (and under some circumstances, you may have to distribute a license document). Apache is not a viral license like the GPL, which forces you to release your modifications to the source code. Note that this project has no connection to the Apache Foundation, other than that we use the same license terms.
We constantly update the community using Twitter. Feel free to follow us
Please, cite SpeechBrain if you use it for your research or business.
@misc{speechbrain,
title={{SpeechBrain}: A General-Purpose Speech Toolkit},
author={Mirco Ravanelli and Titouan Parcollet and Peter Plantinga and Aku Rouhe and Samuele Cornell and Loren Lugosch and Cem Subakan and Nauman Dawalatabad and Abdelwahab Heba and Jianyuan Zhong and Ju-Chieh Chou and Sung-Lin Yeh and Szu-Wei Fu and Chien-Feng Liao and Elena Rastorgueva and François Grondin and William Aris and Hwidong Na and Yan Gao and Renato De Mori and Yoshua Bengio},
year={2021},
eprint={2106.04624},
archivePrefix={arXiv},
primaryClass={eess.AS},
note={arXiv:2106.04624}
}
此处可能存在不合适展示的内容，页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容，可点击提交进行申诉，我们将尽快为您处理。
