【1.2.0】【bugfix】修复internlm2 tokenizer解码时多出空格的问题 · Pull Request !3674 · MindSpore/mindformers

### 相关的Issue
无

### 原因（目的、解决的问题等）
internlm2 tokenizer官方实现有问题，结果相较internlm2 tokenizer fast多出不符合预期的空格
使用hf的internlm2 tokenizer做实验
internlm2 tokenizer
![输入图片说明](https://foruda.gitee.com/images/1722908589288249874/fa7af6c1_9674352.png "屏幕截图")
internlm2 tokenizer fast
![输入图片说明](https://foruda.gitee.com/images/1722908638418265618/57137229_9674352.png "屏幕截图")

### 描述（做了什么，变更了什么）
修改仓上internlm2 tokenizer实现，使之结果与预期（hf的internlm2 tokenizer fast）对齐
修改前
![输入图片说明](https://foruda.gitee.com/images/1722908720821305095/3e7d6d23_9674352.png "屏幕截图")
修改后
![输入图片说明](https://foruda.gitee.com/images/1722908743418563506/c0dd93d6_9674352.png "屏幕截图")

附上验证代码
mf
```python
from internlm2_tokenizer import InternLM2Tokenizer

tokenizer = InternLM2Tokenizer(vocab_file="tokenizer.model")
print("test 1")
print(tokenizer.decode([92543, 77230, 60477, 69259, 74010, 68417, 68364, 61076, 60504, 92542], skip_special_tokens=True))
print("test 2")
print(tokenizer.decode([92543, 77230, 60477, 69259, 74010, 68417, 68364, 61076, 60504, 92542]))
print("test 3")
print(tokenizer.decode([92543, 92542], skip_special_tokens=True))
print("test 4")
print(tokenizer.decode([92543, 92542]))
print("test 5")
print(tokenizer.decode([1, 92543, 77230, 60477, 69259, 74010, 68417, 68364, 61076, 60504, 92542], skip_special_tokens=True))
print("test 6")
print(tokenizer.decode([1, 92543, 77230, 60477, 69259, 74010, 68417, 68364, 61076, 60504, 92542]))
```

hf
```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("internlm/internlm2-chat-7b")
print("test 1")
print(tokenizer.decode([92543, 77230, 60477, 69259, 74010, 68417, 68364, 61076, 60504, 92542], skip_special_tokens=True))
print("test 2")
print(tokenizer.decode([92543, 77230, 60477, 69259, 74010, 68417, 68364, 61076, 60504, 92542]))
print("test 3")
print(tokenizer.decode([92543, 92542], skip_special_tokens=True))
print("test 4")
print(tokenizer.decode([92543, 92542]))
print("test 5")
print(tokenizer.decode([1, 92543, 77230, 60477, 69259, 74010, 68417, 68364, 61076, 60504, 92542], skip_special_tokens=True))
print("test 6")
print(tokenizer.decode([1, 92543, 77230, 60477, 69259, 74010, 68417, 68364, 61076, 60504, 92542]))
```

### check list

#### 是否完成方案评审或问题根因分析（Y/N）Y

#### 是否完成了功能模块的UT/ST，并执行通过，附上结果（Y/N）N

#### 是否涉及公共组件或对外接口修改，涉及时需给出修改范围和影响评估（Y/N）N

#### 是否涉及资料修改，涉及时需同步修改（Y/N）N

GVPMindSpore/mindformers

搜索帮助

GVP MindSpore/mindformers