0 Star 0 Fork 0

科大讯飞/TextPruner

加入 Gitee
与超过 1200万 开发者一起发现、参与优秀开源项目,私有仓库也完全免费 :)
免费加入
文件
克隆/下载
measure_performance.py 1.05 KB
一键复制 编辑 原始数据 按行查看 历史
Ziqing Yang 提交于 3年前 . init commit
import logging
logging.basicConfig(level = logging.INFO,format = '%(asctime)s - %(name)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)
from transformers import XLMRobertaForSequenceClassification,XLMRobertaTokenizer
import sys, os
sys.path.insert(0, os.path.abspath('..'))
from classification_utils.my_dataset import MultilingualNLIDataset
from classification_utils.predict_function import predict
model_path = sys.argv[1]
taskname = 'pawsx'
data_dir = '../datasets/pawsx'
split = 'test'
max_seq_length=128
eval_langs = ['en']
batch_size=32
device = 'cuda'
# Re-initialze the tokenizer
model = XLMRobertaForSequenceClassification.from_pretrained(model_path).to(device)
tokenizer = XLMRobertaTokenizer.from_pretrained(model_path)
eval_dataset = MultilingualNLIDataset(
task=taskname, data_dir=data_dir, split=split, prefix='xlmr',
max_seq_length=max_seq_length, langs=eval_langs, tokenizer=tokenizer)
eval_datasets = [eval_dataset.lang_datasets[lang] for lang in eval_langs]
predict(model, eval_datasets, eval_langs, device, batch_size)
Loading...
马建仓 AI 助手
尝试更多
代码解读
代码找茬
代码优化
Python
1
https://gitee.com/iflytek/TextPruner.git
git@gitee.com:iflytek/TextPruner.git
iflytek
TextPruner
TextPruner
main

搜索帮助