代码拉取完成,页面将自动刷新
torchtext.data.utils.ngrams_iterator(
token_list,
ngrams
)
For more information, see torchtext.data.utils.ngrams_iterator.
class mindspore.dataset.text.Ngram(
n,
left_pad=("", 0),
right_pad=("", 0),
separator=" "
)
For more information, see mindspore.dataset.text.Ngram.
PyTorch: Returns an iterator that generates the given tokens and ngrams.
MindSpore: TensorOp generates n-grams from a one-dimensional string tensor.
from mindspore.dataset import text
from torchtext.data.utils import ngrams_iterator
# In MindSpore, output numpy.ndarray type n-gram.
ngram_op = text.Ngram(3, separator="-")
output = ngram_op(["WildRose Country", "Canada's Ocean Playground", "Land of Living Skies"])
print(output)
# Out:
# ["WildRose Country-Canada's Ocean Playground-Land of Living Skies"]
# In torch, return an iterator that yields the given tokens and their ngrams.
token_list = ['here', 'we', 'are']
print(list(ngrams_iterator(token_list, 2)))
# Out:
# ['here', 'we', 'are', 'here we', 'we are']
此处可能存在不合适展示的内容,页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容,可点击提交进行申诉,我们将尽快为您处理。