# PaddleNLP **Repository Path**: t-_-f/PaddleNLP ## Basic Information - **Project Name**: PaddleNLP - **Description**: Easy-to-use and powerful NLP library with Awesome model zoo, supporting wide-range of NLP tasks from research to industrial applications (Neural Search/QA/IE/Sentiment Analysis) - **Primary Language**: Python - **License**: Apache-2.0 - **Default Branch**: develop - **Homepage**: http://paddlenlp.readthedocs.io - **GVP Project**: No ## Statistics - **Stars**: 0 - **Forks**: 174 - **Created**: 2024-05-13 - **Last Updated**: 2024-06-04 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README [็ฎ€ไฝ“ไธญๆ–‡๐Ÿ€„](./README.md) | **English๐ŸŒŽ**

------------------------------------------------------------------------------------------

Features | Installation | Quick Start | API Reference | Community **PaddleNLP** is a NLP library that is both **easy to use** and **powerful**. It aggregates high-quality pretrained models in the industry and provides a **plug-and-play** development experience, covering a model library for various NLP scenarios. With practical examples from industry practices, PaddleNLP can meet the needs of developers who require **flexible customization**. ## News ๐Ÿ“ข * **2024.01.04 [PaddleNLP v2.7](https://github.com/PaddlePaddle/PaddleNLP/releases/tag/v2.7.0)**: The LLM experience is fully upgraded, and the tool chain LLM entrance is unified. Unify the implementation code of pre-training, fine-tuning, compression, inference and deployment to the `PaddleNLP/llm` directory. The new [LLM Toolchain Documentation](https://paddlenlp.readthedocs.io/zh/latest/llm/finetune.html) provides one-stop guidance for users from getting started with LLM to business deployment and launch. The full breakpoint storage mechanism Unified Checkpoint greatly improves the versatility of LLM storage. Efficient fine-tuning upgrade supports the simultaneous use of efficient fine-tuning + LoRA, and supports QLoRA and other algorithms. * **2023.08.15 [PaddleNLP v2.6](https://github.com/PaddlePaddle/PaddleNLP/releases/tag/v2.6.0)**: Release [Full-process LLM toolchain](./llm) , covering all aspects of pre-training, fine-tuning, compression, inference and deployment, providing users with end-to-end LLM solutions and one-stop development experience; built-in [4D parallel distributed Trainer](./docs/trainer.md ), [Efficient fine-tuning algorithm LoRA/Prefix Tuning](./llm#33-lora), [Self-developed INT8/INT4 quantization algorithm](./llm#6-quantization), etc.; fully supports [LLaMA 1/2](./llm/llama), [BLOOM](.llm/bloom), [ChatGLM 1/2](./llm/chatglm), [GLM](./llm/glm), [OPT](./llm/opt) and other mainstream LLMs. ## Installation ### Prerequisites * python >= 3.7 * paddlepaddle >= 2.6.0 More information about PaddlePaddle installation please refer to [PaddlePaddle's Website](https://www.paddlepaddle.org.cn/install/quick?docurl=/documentation/docs/zh/install/conda/linux-conda.html). ### Python pip Installation ``` pip install --upgrade paddlenlp ``` or you can install the latest develop branch code with the following command: ```shell pip install --pre --upgrade paddlenlp -f https://www.paddlepaddle.org.cn/whl/paddlenlp.html ``` ## Features #### ๐Ÿ“ฆ Out-of-Box NLP Toolset #### ๐Ÿค— Awesome Chinese Model Zoo #### ๐ŸŽ›๏ธ Industrial End-to-end System #### ๐Ÿš€ High Performance Distributed Training and Inference ### Out-of-Box NLP Toolset Taskflow aims to provide off-the-shelf NLP pre-built task covering NLU and NLG technique, in the meanwhile with extremely fast inference satisfying industrial scenario. ![taskflow1](https://user-images.githubusercontent.com/11793384/159693816-fda35221-9751-43bb-b05c-7fc77571dd76.gif) For more usage please refer to [Taskflow Docs](./docs/model_zoo/taskflow.md). ### Awesome Chinese Model Zoo #### ๐Ÿ€„ Comprehensive Chinese Transformer Models We provide **45+** network architectures and over **500+** pretrained models. Not only includes all the SOTA model like ERNIE, PLATO and SKEP released by Baidu, but also integrates most of the high-quality Chinese pretrained model developed by other organizations. Use `AutoModel` API to **โšกSUPER FASTโšก** download pretrained models of different architecture. We welcome all developers to contribute your Transformer models to PaddleNLP! ```python from paddlenlp.transformers import * ernie = AutoModel.from_pretrained('ernie-3.0-medium-zh') bert = AutoModel.from_pretrained('bert-wwm-chinese') albert = AutoModel.from_pretrained('albert-chinese-tiny') roberta = AutoModel.from_pretrained('roberta-wwm-ext') electra = AutoModel.from_pretrained('chinese-electra-small') gpt = AutoModelForPretraining.from_pretrained('gpt-cpm-large-cn') ``` Due to the computation limitation, you can use the ERNIE-Tiny light models to accelerate the deployment of pretrained models. ```python # 6L768H ernie = AutoModel.from_pretrained('ernie-3.0-medium-zh') # 6L384H ernie = AutoModel.from_pretrained('ernie-3.0-mini-zh') # 4L384H ernie = AutoModel.from_pretrained('ernie-3.0-micro-zh') # 4L312H ernie = AutoModel.from_pretrained('ernie-3.0-nano-zh') ``` Unified API experience for NLP task like semantic representation, text classification, sentence matching, sequence labeling, question answering, etc. ```python import paddle from paddlenlp.transformers import * tokenizer = AutoTokenizer.from_pretrained('ernie-3.0-medium-zh') text = tokenizer('natural language processing') # Semantic Representation model = AutoModel.from_pretrained('ernie-3.0-medium-zh') sequence_output, pooled_output = model(input_ids=paddle.to_tensor([text['input_ids']])) # Text Classificaiton and Matching model = AutoModelForSequenceClassification.from_pretrained('ernie-3.0-medium-zh') # Sequence Labeling model = AutoModelForTokenClassification.from_pretrained('ernie-3.0-medium-zh') # Question Answering model = AutoModelForQuestionAnswering.from_pretrained('ernie-3.0-medium-zh') ``` #### Wide-range NLP Task Support PaddleNLP provides rich examples covering mainstream NLP task to help developers accelerate problem solving. You can find our powerful transformer [Model Zoo](./model_zoo), and wide-range NLP application [examples](./examples) with detailed instructions. Also you can run our interactive [Notebook tutorial](https://aistudio.baidu.com/aistudio/personalcenter/thirdview/574995) on AI Studio, a powerful platform with **FREE** computing resource.
PaddleNLP Transformer model summary (click to show details)
| Model | Sequence Classification | Token Classification | Question Answering | Text Generation | Multiple Choice | | :----------------- | ----------------------- | -------------------- | ------------------ | --------------- | --------------- | | ALBERT | โœ… | โœ… | โœ… | โŒ | โœ… | | BART | โœ… | โœ… | โœ… | โœ… | โŒ | | BERT | โœ… | โœ… | โœ… | โŒ | โœ… | | BigBird | โœ… | โœ… | โœ… | โŒ | โœ… | | BlenderBot | โŒ | โŒ | โŒ | โœ… | โŒ | | ChineseBERT | โœ… | โœ… | โœ… | โŒ | โŒ | | ConvBERT | โœ… | โœ… | โœ… | โŒ | โœ… | | CTRL | โœ… | โŒ | โŒ | โŒ | โŒ | | DistilBERT | โœ… | โœ… | โœ… | โŒ | โŒ | | ELECTRA | โœ… | โœ… | โœ… | โŒ | โœ… | | ERNIE | โœ… | โœ… | โœ… | โŒ | โœ… | | ERNIE-CTM | โŒ | โœ… | โŒ | โŒ | โŒ | | ERNIE-Doc | โœ… | โœ… | โœ… | โŒ | โŒ | | ERNIE-GEN | โŒ | โŒ | โŒ | โœ… | โŒ | | ERNIE-Gram | โœ… | โœ… | โœ… | โŒ | โŒ | | ERNIE-M | โœ… | โœ… | โœ… | โŒ | โŒ | | FNet | โœ… | โœ… | โœ… | โŒ | โœ… | | Funnel-Transformer | โœ… | โœ… | โœ… | โŒ | โŒ | | GPT | โœ… | โœ… | โŒ | โœ… | โŒ | | LayoutLM | โœ… | โœ… | โŒ | โŒ | โŒ | | LayoutLMv2 | โŒ | โœ… | โŒ | โŒ | โŒ | | LayoutXLM | โŒ | โœ… | โŒ | โŒ | โŒ | | LUKE | โŒ | โœ… | โœ… | โŒ | โŒ | | mBART | โœ… | โŒ | โœ… | โŒ | โœ… | | MegatronBERT | โœ… | โœ… | โœ… | โŒ | โœ… | | MobileBERT | โœ… | โŒ | โœ… | โŒ | โŒ | | MPNet | โœ… | โœ… | โœ… | โŒ | โœ… | | NEZHA | โœ… | โœ… | โœ… | โŒ | โœ… | | PP-MiniLM | โœ… | โŒ | โŒ | โŒ | โŒ | | ProphetNet | โŒ | โŒ | โŒ | โœ… | โŒ | | Reformer | โœ… | โŒ | โœ… | โŒ | โŒ | | RemBERT | โœ… | โœ… | โœ… | โŒ | โœ… | | RoBERTa | โœ… | โœ… | โœ… | โŒ | โœ… | | RoFormer | โœ… | โœ… | โœ… | โŒ | โŒ | | SKEP | โœ… | โœ… | โŒ | โŒ | โŒ | | SqueezeBERT | โœ… | โœ… | โœ… | โŒ | โŒ | | T5 | โŒ | โŒ | โŒ | โœ… | โŒ | | TinyBERT | โœ… | โŒ | โŒ | โŒ | โŒ | | UnifiedTransformer | โŒ | โŒ | โŒ | โœ… | โŒ | | XLNet | โœ… | โœ… | โœ… | โŒ | โœ… |
For more pretrained model usage, please refer to [Transformer API Docs](./docs/model_zoo/index.rst). ### Industrial End-to-end System We provide high value scenarios including information extraction, semantic retrieval, question answering high-value. For more details industrial cases please refer to [Applications](./applications). #### ๐Ÿ” Neural Search System
For more details please refer to [Neural Search](./applications/neural_search). #### โ“ Question Answering System We provide question answering pipeline which can support FAQ system, Document-level Visual Question answering system based on [๐Ÿš€RocketQA](https://github.com/PaddlePaddle/RocketQA).
For more details please refer to [Question Answering](./applications/question_answering) and [Document VQA](./applications/document_intelligence/doc_vqa). #### ๐Ÿ’Œ Opinion Extraction and Sentiment Analysis We build an opinion extraction system for product review and fine-grained sentiment analysis based on [SKEP](https://arxiv.org/abs/2005.05635) Model.
For more details please refer to [Sentiment Analysis](./applications/sentiment_analysis). #### ๐ŸŽ™๏ธ Speech Command Analysis Integrated ASR Model, Information Extraction, we provide a speech command analysis pipeline that show how to use PaddleNLP and [PaddleSpeech](https://github.com/PaddlePaddle/PaddleSpeech) to solve Speech + NLP real scenarios.
For more details please refer to [Speech Command Analysis](./applications/speech_cmd_analysis). ### High Performance Distributed Training and Inference #### โšก FastTokenizer: High Performance Text Preprocessing Library
```python AutoTokenizer.from_pretrained("ernie-3.0-medium-zh", use_fast=True) ``` Set `use_fast=True` to use C++ Tokenizer kernel to achieve 100x faster on text pre-processing. For more usage please refer to [FastTokenizer](./fast_tokenizer). #### โšก FastGeneration: High Performance Generation Library
```python model = GPTLMHeadModel.from_pretrained('gpt-cpm-large-cn') ... outputs, _ = model.generate( input_ids=inputs_ids, max_length=10, decode_strategy='greedy_search', use_fast=True) ``` Set `use_fast=True` to achieve 5x speedup for Transformer, GPT, BART, PLATO, UniLM text generation. For more usage please refer to [FastGeneration](./fast_generation). #### ๐Ÿš€ Fleet: 4D Hybrid Distributed Training
For more super large-scale model pre-training details please refer to [GPT-3](./examples/language_model/gpt-3). ## Quick Start **Taskflow** aims to provide off-the-shelf NLP pre-built task covering NLU and NLG scenario, in the meanwhile with extremely fast inference satisfying industrial applications. ```python from paddlenlp import Taskflow # Chinese Word Segmentation seg = Taskflow("word_segmentation") seg("็ฌฌๅๅ››ๅฑŠๅ…จ่ฟไผšๅœจ่ฅฟๅฎ‰ไธพๅŠž") >>> ['็ฌฌๅๅ››ๅฑŠ', 'ๅ…จ่ฟไผš', 'ๅœจ', '่ฅฟๅฎ‰', 'ไธพๅŠž'] # POS Tagging tag = Taskflow("pos_tagging") tag("็ฌฌๅๅ››ๅฑŠๅ…จ่ฟไผšๅœจ่ฅฟๅฎ‰ไธพๅŠž") >>> [('็ฌฌๅๅ››ๅฑŠ', 'm'), ('ๅ…จ่ฟไผš', 'nz'), ('ๅœจ', 'p'), ('่ฅฟๅฎ‰', 'LOC'), ('ไธพๅŠž', 'v')] # Named Entity Recognition ner = Taskflow("ner") ner("ใ€Šๅญคๅฅณใ€‹ๆ˜ฏ2010ๅนดไนๅทžๅ‡บ็‰ˆ็คพๅ‡บ็‰ˆ็š„ๅฐ่ฏด๏ผŒไฝœ่€…ๆ˜ฏไฝ™ๅ…ผ็พฝ") >>> [('ใ€Š', 'w'), ('ๅญคๅฅณ', 'ไฝœๅ“็ฑป_ๅฎžไฝ“'), ('ใ€‹', 'w'), ('ๆ˜ฏ', '่‚ฏๅฎš่ฏ'), ('2010ๅนด', 'ๆ—ถ้—ด็ฑป'), ('ไนๅทžๅ‡บ็‰ˆ็คพ', '็ป„็ป‡ๆœบๆž„็ฑป'), ('ๅ‡บ็‰ˆ', 'ๅœบๆ™ฏไบ‹ไปถ'), ('็š„', 'ๅŠฉ่ฏ'), ('ๅฐ่ฏด', 'ไฝœๅ“็ฑป_ๆฆ‚ๅฟต'), ('๏ผŒ', 'w'), ('ไฝœ่€…', 'ไบบ็‰ฉ็ฑป_ๆฆ‚ๅฟต'), ('ๆ˜ฏ', '่‚ฏๅฎš่ฏ'), ('ไฝ™ๅ…ผ็พฝ', 'ไบบ็‰ฉ็ฑป_ๅฎžไฝ“')] # Dependency Parsing ddp = Taskflow("dependency_parsing") ddp("9ๆœˆ9ๆ—ฅไธŠๅˆ็บณ่พพๅฐ”ๅœจไบš็‘Ÿยท้˜ฟไป€็ƒๅœบๅ‡ป่ดฅไฟ„็ฝ—ๆ–ฏ็ƒๅ‘˜ๆข…ๅพท้Ÿฆๆฐๅคซ") >>> [{'word': ['9ๆœˆ9ๆ—ฅ', 'ไธŠๅˆ', '็บณ่พพๅฐ”', 'ๅœจ', 'ไบš็‘Ÿยท้˜ฟไป€็ƒๅœบ', 'ๅ‡ป่ดฅ', 'ไฟ„็ฝ—ๆ–ฏ', '็ƒๅ‘˜', 'ๆข…ๅพท้Ÿฆๆฐๅคซ'], 'head': [2, 6, 6, 5, 6, 0, 8, 9, 6], 'deprel': ['ATT', 'ADV', 'SBV', 'MT', 'ADV', 'HED', 'ATT', 'ATT', 'VOB']}] # Sentiment Analysis senta = Taskflow("sentiment_analysis") senta("่ฟ™ไธชไบงๅ“็”จ่ตทๆฅ็œŸ็š„ๅพˆๆต็•…๏ผŒๆˆ‘้žๅธธๅ–œๆฌข") >>> [{'text': '่ฟ™ไธชไบงๅ“็”จ่ตทๆฅ็œŸ็š„ๅพˆๆต็•…๏ผŒๆˆ‘้žๅธธๅ–œๆฌข', 'label': 'positive', 'score': 0.9938690066337585}] ``` ## API Reference - Support [LUGE](https://www.luge.ai/) dataset loading and compatible with Hugging Face [Datasets](https://huggingface.co/datasets). For more details please refer to [Dataset API](https://paddlenlp.readthedocs.io/zh/latest/data_prepare/dataset_list.html). - Using Hugging Face style API to load 500+ selected transformer models and download with fast speed. For more information please refer to [Transformers API](https://paddlenlp.readthedocs.io/zh/latest/model_zoo/index.html). - One-line of code to load pre-trained word embedding. For more usage please refer to [Embedding API](https://paddlenlp.readthedocs.io/zh/latest/model_zoo/embeddings.html). Please find all PaddleNLP API Reference from our [readthedocs](https://paddlenlp.readthedocs.io/). ## Community ### Slack To connect with other users and contributors, welcome to join our [Slack channel](https://paddlenlp.slack.com/). ### WeChat Scan the QR code below with your Wechatโฌ‡๏ธ. You can access to official technical exchange group. Look forward to your participation.
## Citation If you find PaddleNLP useful in your research, please consider cite ``` @misc{=paddlenlp, title={PaddleNLP: An Easy-to-use and High Performance NLP Library}, author={PaddleNLP Contributors}, howpublished = {\url{https://github.com/PaddlePaddle/PaddleNLP}}, year={2021} } ``` ## Acknowledge We have borrowed from Hugging Face's [Transformers](https://github.com/huggingface/transformers)๐Ÿค— excellent design on pretrained models usage, and we would like to express our gratitude to the authors of Hugging Face and its open source community. ## License PaddleNLP is provided under the [Apache-2.0 License](./LICENSE).