GLUE-QNLI

A repository comparing the inference accuracy of MindNLP and Transformer on the GLUE QNLI dataset

• Dataset

• The QNLI (Question Natural Language Inference) dataset is part of the GLUE benchmark. It is converted from the Stanford Question Answering Dataset (SQuAD).

• Getting the Dataset

• 1. Visit GLUE Benchmark Tasks
• 2. Register/Login to download the GLUE data
• 3. Download and extract the QNLI dataset
• 4. Place the following files in the mindnlp/benchmark/GLUE-QNLI/ directory:
• • dev.tsv (Development set)
• • test.tsv (Test set)
• • train.tsv (Training set)
• The QNLI task is a binary classification task derived from SQuAD, where the goal is to determine whether a given context sentence contains the answer to a given question.

Quick Start

Installation

To get started with this project, follow these steps:

1. Create a conda environment (optional but recommended):

   conda create -n mindnlp python==3.9
   conda activate mindnlp
   
2. Install the dependencies: Please note that mindnlp is in the Ascend environment, while transformers is in the GPU environment, and the required dependencies are in the requirements of their respective folders.

   pip install -r requirements.txt
   
3. Usage Once the installation is complete, you can choose use differnet models to start inference. Here's how to run the inference:

   # Evaluate specific model using default dataset (dev.tsv)
   python model_QNLI.py --model bart
   
   # Evaluate with custom dataset
   python model_QNLI.py --model bart --data ./QNLI/test.tsv
   
   Supported model options: bart, bert, roberta, xlm-roberta, gpt2, t5, distilbert, albert, llama, opt

Accuracy Comparsion

Our reproduced model performance on QNLI/dev.tsv is reported as follows. Experiments are tested on ascend 910* with mindspore 2.3.1 graph mode. All fine-tuned models are derived from open-source models provided by huggingface.