Preprocessing Chinese Text.ipynb · d_benhua/Preprocessing_Chinese_Text

加入 Gitee

与超过 1200万开发者一起发现、参与优秀开源项目，私有仓库也完全免费：）

该仓库未声明开源许可证文件（LICENSE），使用请关注具体项目描述及其代码上游依赖。

克隆/下载

Preprocessing Chinese Text.ipynb 48.38 KB

{
 "cells": [
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "# 一、中文文本预处理\n",
 "\n",
 "参考链接：<https://zhuanlan.zhihu.com/p/46704845>"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {
 "toc": true
 },
 "source": [
 "<h1>Contents/目录</h1>\n",
 "<div class=\"toc\"><ul class=\"toc-item\"><li><a href=\"#前言\" data-toc-modified-id=\"前言-0\">前言</a></li><li><a href=\"#预处理步骤\" data-toc-modified-id=\"预处理步骤-1\">预处理步骤</a></li><li><a href=\"#一、去除指定无用符号\" data-toc-modified-id=\"一、去除指定无用符号-2\">一、去除指定无用符号</a><ul class=\"toc-item\"><li><a href=\"#知识点\" data-toc-modified-id=\"知识点-2.1\">知识点</a><ul class=\"toc-item\"><li><a href=\"#1.map函数\" data-toc-modified-id=\"1.map函数-2.1.1\">1.map函数</a><ul class=\"toc-item\"><li><a href=\"#语法：\" data-toc-modified-id=\"语法：-2.1.1.1\">语法：</a></li><li><a href=\"#参数\" data-toc-modified-id=\"参数-2.1.1.2\">参数</a></li></ul></li><li><a href=\"#2.lambda匿名函数\" data-toc-modified-id=\"2.lambda匿名函数-2.1.2\">2.lambda匿名函数</a><ul class=\"toc-item\"><li><a href=\"#语法\" data-toc-modified-id=\"语法-2.1.2.1\">语法</a></li><li><a href=\"#参数\" data-toc-modified-id=\"参数-2.1.2.2\">参数</a></li></ul></li><li><a href=\"#3.str.replace(old,new[,max])方法\" data-toc-modified-id=\"3.str.replace(old,new[,max])方法-2.1.3\">3.str.replace(old,new[,max])方法</a><ul class=\"toc-item\"><li><a href=\"#语法\" data-toc-modified-id=\"语法-2.1.3.1\">语法</a></li><li><a href=\"#参数\" data-toc-modified-id=\"参数-2.1.3.2\">参数</a></li><li><a href=\"#返回值\" data-toc-modified-id=\"返回值-2.1.3.3\">返回值</a></li></ul></li></ul></li></ul></li><li><a href=\"#二、去除非中文字符\" data-toc-modified-id=\"二、去除非中文字符-3\">二、去除非中文字符</a><ul class=\"toc-item\"><li><a href=\"#知识点\" data-toc-modified-id=\"知识点-3.1\">知识点</a><ul class=\"toc-item\"><li><a href=\"#1.-unicode编码[\\u4e00-\\u9fa5]匹配所有中文\" data-toc-modified-id=\"1.-unicode编码[\\u4e00-\\u9fa5]匹配所有中文-3.1.1\">1. unicode编码[\\u4e00-\\u9fa5]匹配所有中文</a></li><li><a href=\"#2.-list.append(obj)方法\" data-toc-modified-id=\"2.-list.append(obj)方法-3.1.2\">2. list.append(obj)方法</a><ul class=\"toc-item\"><li><a href=\"#语法\" data-toc-modified-id=\"语法-3.1.2.1\">语法</a></li><li><a href=\"#参数\" data-toc-modified-id=\"参数-3.1.2.2\">参数</a></li><li><a href=\"#返回值\" data-toc-modified-id=\"返回值-3.1.2.3\">返回值</a></li></ul></li></ul></li></ul></li><li><a href=\"#三、对文本进行jieba分词\" data-toc-modified-id=\"三、对文本进行jieba分词-4\">三、对文本进行jieba分词</a><ul class=\"toc-item\"><li><a href=\"#知识点\" data-toc-modified-id=\"知识点-4.1\">知识点</a><ul class=\"toc-item\"><li><a href=\"#1.-第三方组件、包的导入与使用\" data-toc-modified-id=\"1.-第三方组件、包的导入与使用-4.1.1\">1. 第三方组件、包的导入与使用</a></li></ul></li></ul></li><li><a href=\"#四、去除停用词\" data-toc-modified-id=\"四、去除停用词-5\">四、去除停用词</a><ul class=\"toc-item\"><li><a href=\"#知识点\" data-toc-modified-id=\"知识点-5.1\">知识点</a><ul class=\"toc-item\"><li><a href=\"#1.-停用词\" data-toc-modified-id=\"1.-停用词-5.1.1\">1. 停用词</a></li></ul></li></ul></li><li><a href=\"#五、转化文本为tfidf向量\" data-toc-modified-id=\"五、转化文本为tfidf向量-6\">五、转化文本为tfidf向量</a><ul class=\"toc-item\"><li><a href=\"#知识点\" data-toc-modified-id=\"知识点-6.1\">知识点</a><ul class=\"toc-item\"><li><a href=\"#1.-gensim库\" data-toc-modified-id=\"1.-gensim库-6.1.1\">1. gensim库</a></li><li><a href=\"#2.-corpora.dictionary-–-Construct-word<->id-mappings\" data-toc-modified-id=\"2.-corpora.dictionary-–-Construct-word<->id-mappings-6.1.2\">2. <a href=\"https://radimrehurek.com/gensim/corpora/dictionary.html\" target=\"_blank\"><code>corpora.dictionary</code> – Construct word&lt;-&gt;id mappings</a></a></li><li><a href=\"#3.--models.tfidfmodel-–-TF-IDF-model\" data-toc-modified-id=\"3.--models.tfidfmodel-–-TF-IDF-model-6.1.3\">3. <a href=\"https://radimrehurek.com/gensim/models/tfidfmodel.html#id2\" target=\"_blank\"><code>models.tfidfmodel</code> – TF-IDF model</a></a></li><li><a href=\"#4.-pprint\" data-toc-modified-id=\"4.-pprint-6.1.4\">4. <a href=\"https://docs.python.org/3/library/pprint.html\" target=\"_blank\">pprint</a></a></li></ul></li></ul></li><li><a href=\"#六、将原始语料库转化为tfidf向量的过程总结\" data-toc-modified-id=\"六、将原始语料库转化为tfidf向量的过程总结-7\">六、将原始语料库转化为tfidf向量的过程总结</a></li></ul></div>"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## 前言\n",
 "- 机器学习我的理解就是把各种原始的东西变成机器可以理解的东西，然后再用各种机器学习算法来做操作。\n",
 "- 机器可以理解的东西是什么呢？——**向量** 。所以不管是图片还是文字，要用机器学习算法对它们进行处理，就要把它们转为**向量**。\n",
 "\n",
 "> 机器可以理解的东西是向量，NLP的任务是将输入（文本，图像，语音等）变成**向量**，从而对向量进行操作。-- my"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## 预处理步骤\n",
 "- 将文本中指定无用的符号去除\n",
 "- 将文本中的非中文符号去除（前两步目的是得到纯中文文本 -- my）\n",
 "- 对文本进行**jieba**分词\n",
 "- 将分词结果中的**停用词**去除（即留下此文本的关键词 -- my）\n",
 "- 将文本转成**TFIDF**向量"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## 一、去除指定无用符号\n",
 "无用符号可以自己指定，如空格..."
 ]
 },
 {
 "cell_type": "code",
 "execution_count": 8,
 "metadata": {},
 "outputs": [
 {
 "name": "stdout",
 "output_type": "stream",
 "text": [
 "['w', '我', '是', 'd', 'h', '你', '的', '，', '谁', '本', '地', '最', '，', '，', '强', '王', '者', '的', '风', '衣', '大', '哥', '好', '，', '喝', '酒', '，', '吃', '茶', '睡', '觉', '觉', '你', '是', '我', '的', '谁', '？', 'd']\n"
 ]
 }
 ],
 "source": [
 "content = ['w我是 dh 你 的，谁 本地 最 ，， 强 王者 的风 衣',' 大哥 好，喝 酒，吃 茶睡觉 觉你是 我的 谁？ d']\n",
 "\n",
 "#去除空格\n",
 "content_last = []\n",
 "for i in range(len(content)):\n",
 " for item in content[i]:\n",
 " if item == ' ':\n",
 " continue\n",
 " else:\n",
 " content_last += item\n",
 "print(content_last)"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": 9,
 "metadata": {},
 "outputs": [
 {
 "name": "stdout",
 "output_type": "stream",
 "text": [
 "['w我是dh你的，谁本地最，，强王者的风衣', '大哥好，喝酒，吃茶睡觉觉你是我的谁？d']\n"
 ]
 }
 ],
 "source": [
 "content = ['w我是 dh 你 的，谁 本地 最 ，， 强 王者 的风 衣',' 大哥 好，喝 酒，吃 茶睡觉 觉你是 我的 谁？ d']\n",
 "\n",
 "content_last = []\n",
 "for i in range(len(content)):\n",
 " content_tmp = ''\n",
 " for item in content[i]:\n",
 " if item == ' ':\n",
 " continue\n",
 " else:\n",
 " content_tmp += item\n",
 " content_last.append(content_tmp)\n",
 "print(content_last)"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": 10,
 "metadata": {},
 "outputs": [
 {
 "name": "stdout",
 "output_type": "stream",
 "text": [
 "['w我是dh你的，谁本地最，，强王者的风衣', '大哥好，喝酒，吃茶睡觉觉你是我的谁？d']\n"
 ]
 }
 ],
 "source": [
 "content = ['w我是 dh 你 的，谁 本地 最 ，， 强 王者 的风 衣',' 大哥 好，喝 酒，吃 茶睡觉 觉你是 我的 谁？ d']\n",
 "\n",
 "def process(ourdata):\n",
 " new_list = map(lambda s:s.replace(' ',''),ourdata)\n",
 " return list(new_list)\n",
 "print(process(content))"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "### 知识点\n",
 "#### 1.map函数\n",
 "- 根据提供的function对指定的iterable序列做映射 \n",
 "- 创建一个列表，其中包含对指定序列包含的项执行指定函数返回的值\n",
 "\n",
 "##### 语法：\n",
 "- map(function, iterable, ...)\n",
 "\n",
 "##### 参数\n",
 "- function -- 函数\n",
 "- iterable -- 一个或多个序列 \n",
 "\n",
 "\n",
 ">参考链接：<https://www.runoob.com/python/python-func-map.html>"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": 13,
 "metadata": {},
 "outputs": [
 {
 "data": {
 "text/plain": [
 "<map at 0x264d7094dc8>"
 ]
 },
 "execution_count": 13,
 "metadata": {},
 "output_type": "execute_result"
 }
 ],
 "source": [
 "# e.g.实例\n",
 "def square(x): #计算平方数\n",
 " return x**2\n",
 "square(2)\n",
 "map(square,[4]) #计算列表各元素的平方\n",
 "\n",
 "#map(lambda x:x**2,[1,2,3,4,5])"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": 48,
 "metadata": {},
 "outputs": [
 {
 "name": "stdout",
 "output_type": "stream",
 "text": [
 "<map object at 0x000001F5369BA248>\n"
 ]
 }
 ],
 "source": [
 "# e.g.\n",
 "def square(x): #计算平方数\n",
 " return x**2\n",
 "square(2)\n",
 "ls = map(square,[4,2,3,4,5]) #计算列表各元素的平方\n",
 "print(ls)\n",
 "\n",
 "#map(lambda x:x**2,[1,2,3,4,5])"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": 52,
 "metadata": {},
 "outputs": [
 {
 "name": "stdout",
 "output_type": "stream",
 "text": [
 "[16, 4, 9, 16, 25]\n"
 ]
 }
 ],
 "source": [
 "# e.g.\n",
 "def square(x): #计算平方数\n",
 " return x**2\n",
 "square(2)\n",
 "ls = list(map(square,[4,2,3,4,5])) #计算列表各元素的平方\n",
 "print(ls)\n",
 "\n",
 "#map(lambda x:x**2,[1,2,3,4,5])"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": 65,
 "metadata": {},
 "outputs": [
 {
 "ename": "TypeError",
 "evalue": "'map' object is not callable",
 "output_type": "error",
 "traceback": [
 "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
 "\u001b[1;31mTypeError\u001b[0m Traceback (most recent call last)",
 "\u001b[1;32m<ipython-input-65-87744db26f01>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m 4\u001b[0m \u001b[0msquare\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m2\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 5\u001b[0m \u001b[0mls\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mmap\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0msquare\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m4\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;36m2\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;36m3\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;36m4\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;36m5\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;31m#计算列表各元素的平方\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 6\u001b[1;33m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mlist\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mls\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 7\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 8\u001b[0m \u001b[1;31m#map(lambda x:x**2,[1,2,3,4,5])\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
 "\u001b[1;31mTypeError\u001b[0m: 'map' object is not callable"
 ]
 }
 ],
 "source": [
 "# e.g.\n",
 "def square(x): #计算平方数\n",
 " return x**2\n",
 "square(2)\n",
 "ls = map(square,[4,2,3,4,5]) #计算列表各元素的平方\n",
 "print(list(ls))\n",
 "\n",
 "#map(lambda x:x**2,[1,2,3,4,5])"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": 57,
 "metadata": {},
 "outputs": [],
 "source": [
 "#list = map(lambda x:x**2,[1,2,3,4,5])"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": 58,
 "metadata": {},
 "outputs": [
 {
 "ename": "TypeError",
 "evalue": "'map' object is not callable",
 "output_type": "error",
 "traceback": [
 "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
 "\u001b[1;31mTypeError\u001b[0m Traceback (most recent call last)",
 "\u001b[1;32m<ipython-input-58-cb42058c3e12>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m 1\u001b[0m \u001b[0mls\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mmap\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;32mlambda\u001b[0m \u001b[0mx\u001b[0m\u001b[1;33m:\u001b[0m\u001b[0mx\u001b[0m\u001b[1;33m**\u001b[0m\u001b[1;36m2\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;36m2\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;36m3\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;36m4\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;36m5\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 2\u001b[1;33m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mlist\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mls\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
 "\u001b[1;31mTypeError\u001b[0m: 'map' object is not callable"
 ]
 }
 ],
 "source": [
 "ls = map(lambda x:x**2,[1,2,3,4,5])\n",
 "print(list(ls))"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 ">由于上面（In [57]）`list = map(lambda x:x**2,[1,2,3,4,5])`已经定义了`list`变量，所以下面(In [58])会报错。重新运行，取消掉list已定义variable的影响。解决方法参考：https://stackoverflow.com/questions/49998740/error-list-object-is-not-callable-in-map-function"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": 1,
 "metadata": {},
 "outputs": [
 {
 "name": "stdout",
 "output_type": "stream",
 "text": [
 "[1, 4, 9, 16, 25]\n"
 ]
 }
 ],
 "source": [
 "ls = map(lambda x:x**2,[1,2,3,4,5])\n",
 "print(list(ls))"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "#### 2.lambda匿名函数\n",
 "- lambd为表达式，仅仅能在lambda表达式中封装有限的逻辑进去。\n",
 "- lambda 只是一个表达式，函数体比 def 简单很多。\n",
 "- lambda的主体是一个表达式，而不是一个代码块。仅仅能在lambda表达式中封装有限的逻辑进去。\n",
 "- lambda 函数拥有自己的命名空间，且不能访问自己参数列表之外或全局命名空间里的参数。\n",
 "- 虽然lambda函数看起来只能写一行，却不等同于C或C++的内联函数，后者的目的是调用小函数时不占用栈内存从而增加运行效率。\n",
 "\n",
 "##### 语法\n",
 "- lambad [arg1[,arg2,...argn]]:expression\n",
 "\n",
 "##### 参数\n",
 "- arg为参数，[,arg2,...,argn]为可选参数 \n",
 "\n",
 "\n",
 ">参考链接：<https://www.runoob.com/python3/python3-function.html>"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": 1,
 "metadata": {},
 "outputs": [
 {
 "name": "stdout",
 "output_type": "stream",
 "text": [
 "6\n"
 ]
 }
 ],
 "source": [
 "# e.g.实例\n",
 "sum = lambda s: s + 2 #用lambda设置匿名函数并传给sum\n",
 "print(sum(4)) #调用并输出函数值"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": 3,
 "metadata": {},
 "outputs": [
 {
 "name": "stdout",
 "output_type": "stream",
 "text": [
 "10 + 20 = 30\n"
 ]
 }
 ],
 "source": [
 "# e.g.\n",
 "sum = lambda s1,s2: s1 + s2\n",
 "print(\"10 + 20 =\",sum(10,20))"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "#### 3.str.replace(old,new[,max])方法\n",
 "把字符串中的old（旧字符串）替换成new（新字符串）\n",
 "##### 语法\n",
 "- str.replace(old,new[,max])\n",
 "\n",
 "##### 参数\n",
 "- old 被替换子字符串\n",
 "- new 新字符串\n",
 "- max 替换的最大次数，可选参数 \n",
 "\n",
 "##### 返回值\n",
 "- 返回字符串中的 old（旧字符串） 替换成 new(新字符串)后生成的新字符串，如果指定第三个参数max，则替换不超过 max 次。\n",
 "\n",
 ">参考链接：<https://www.runoob.com/python/att-string-replace.html> \n",
 ">Python2字符串参考链接：<https://www.runoob.com/python/python-strings.html> \n",
 ">Python3字符串参考链接：<https://www.runoob.com/python3/python3-string.html>"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": 9,
 "metadata": {},
 "outputs": [
 {
 "name": "stdout",
 "output_type": "stream",
 "text": [
 "thwas was string example....wow!!! thwas was really string\n",
 "thwas was string example....wow!!! thwas is really string\n",
 "this is string example....wow!!! this is really string\n"
 ]
 }
 ],
 "source": [
 "# e.g.实例\n",
 "str = \"this is string example....wow!!! this is really string\";\n",
 "print(str.replace(\"is\", \"was\"))\n",
 "print(str.replace(\"is\", \"was\", 3))\n",
 "print(str)"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "由上可知，replace方法并没有改变原字符串str的内容，而是对其副本进行了替换。"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## 二、去除非中文字符"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "去除所有符号，如：数字、标点、字母等等。"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": 10,
 "metadata": {},
 "outputs": [
 {
 "name": "stdout",
 "output_type": "stream",
 "text": [
 "['我是你睡觉觉的谁好评谁王者本地最强王者的风衣', '大哥风衣好喝酒吃茶睡觉觉你是我的谁', '大哥大嫂过年吃茶好评']\n"
 ]
 }
 ],
 "source": [
 "content = ['w我是 dh 你睡觉觉 的，谁好评 谁王者 本地 最 ，， 强 王者 的风 衣',' 大哥 风衣好，喝 酒，吃 茶睡觉 觉你是 我的 谁？ d','大哥大嫂 过年 hao，guonianhao，吃茶，好评']\n",
 "\n",
 "#判断是否为中文\n",
 "def isChinese(uchar):\n",
 " if uchar >= u'\\u4e00' and uchar <= u'\\u9fa5':\n",
 " return True\n",
 " else:\n",
 " return False\n",
 "\n",
 "#对字符串去除中文\n",
 "def format_str(content):\n",
 " content_str = ''\n",
 " for i in content:\n",
 " if isChinese(i):\n",
 " content_str = content_str + i\n",
 " return content_str\n",
 "\n",
 "#对列表去除中文\n",
 "content_list = []\n",
 "for line in content:\n",
 " content_list.append(format_str(line))\n",
 "print(content_list)"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "### 知识点\n",
 "\n",
 "#### 1. unicode编码[\\u4e00-\\u9fa5]匹配所有中文\n",
 "\n",
 "- \"u4e00\"代表什么意思 \"u9fa5“代表什么意思？\n",
 "\n",
 "- 答：**\\u4e00-\\u9fa5**是用来判断是不是中文的一个条件，采用的是unicode编码\n",
 "\n",
 "- 查了下中文的unicode的中文编码表\n",
 "- 第一个“4e00”\n",
 "- 最后一个“9fa0”\n",
 "- 总共有20901个汉字，中国文化果然博大精深啊。\n",
 "\n",
 "\n",
 ">参考链接： <https://blog.csdn.net/tinyletero/article/details/8201465> \n",
 ">python官网文档：https://docs.python.org/3/howto/unicode.html#the-unicode-type \n",
 ">附中文编码表下载http://download.csdn.net/detail/s_jobs/4786519 \n",
 ">有关中文编码的知识可以参考http://blog.csdn.net/s_jobs/article/details/8197974 "
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "#### 2. list.append(obj)方法\n",
 "在列表末尾添加新的对象\n",
 "##### 语法 \n",
 "- list.append(obj) \n",
 "\n",
 "##### 参数\n",
 "- obj -- 添加到列表末尾的对象 \n",
 "\n",
 "##### 返回值\n",
 "- 无，但会修改原始列表\n",
 "\n",
 ">参考链接：https://www.runoob.com/python3/python3-att-list-append.html"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": 14,
 "metadata": {},
 "outputs": [
 {
 "name": "stdout",
 "output_type": "stream",
 "text": [
 "更新后的列表： ['Google', 'Runoob', 'Taobao', 'Baidu']\n"
 ]
 }
 ],
 "source": [
 "#e.g.实例\n",
 "list1 = ['Google','Runoob','Taobao']\n",
 "list1.append('Baidu')\n",
 "print(\"更新后的列表：\",list1)"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## 三、对文本进行jieba分词"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "使用jieba分词库对文本分词"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": 12,
 "metadata": {},
 "outputs": [
 {
 "name": "stdout",
 "output_type": "stream",
 "text": [
 "[['我', '是', '你', '睡觉觉', '的', '谁', '好评', '谁', '王者', '本地', '最强', '王者', '的', '风衣'], ['大哥', '风衣', '好', '喝酒', '吃', '茶', '睡觉觉', '你', '是', '我', '的', '谁'], ['大哥', '大嫂', '过年', '吃', '茶', '好评']]\n"
 ]
 }
 ],
 "source": [
 "# 采用~~2.去除非中文字符的处理结果：['我是你的谁本地最强王者的风衣', '大哥好喝酒吃茶睡觉觉你是我的谁']\n",
 "\n",
 "text_list = ['我是你睡觉觉的谁好评谁王者本地最强王者的风衣', '大哥风衣好喝酒吃茶睡觉觉你是我的谁', '大哥大嫂过年吃茶好评']\n",
 "# ! pip install jieba #导入包\n",
 "# !pip install --upgrade pip #更新pip\n",
 "# %time !pip install package #查看安装时间\n",
 "# %pip install jieba ##导入包到the current kernel而不是launched the notebook\n",
 "\n",
 "import jieba\n",
 "def fenci(datas):\n",
 " cut_words = map(lambda s: list(jieba.cut(s)),datas)\n",
 " return list(cut_words)\n",
 " \n",
 "print(fenci(text_list))"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "### 知识点\n",
 "#### 1. 第三方组件、包的导入与使用\n",
 "- ！pip isntall package #安装包 \n",
 "- import package #导入包 \n",
 "- package.method() #使用包的功能 \n",
 "- %pip install package In IPython (jupyter) 7.3 and later, there is a magic %pip and %conda command that will install into the current kernel (rather than into the instance of Python that launched the notebook).来自Stack Overflow\n",
 "\n",
 "\n",
 ">jieba组件参考链接：https://github.com/fxsjy/jieba \n",
 ">\n",
 ">第三方Python包（如jieba）在jupyter上的参考链接：https://blog.csdn.net/mymatin1004/article/details/82698422?utm_medium=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-2.control&depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-2.control \n",
 ">\n",
 ">Stack Overflow参考链接：https://stackoverflow.com/questions/38368318/installing-a-pip-package-from-within-a-jupyter-notebook-not-working\n",
 ">\n",
 ">python Jupyter不能导入外部包----解决方案:https://blog.csdn.net/qq_42182367/article/details/82594994?utm_medium=distribute.pc_relevant.none-task-blog-BlogCommendFromBaidu-2.control&depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromBaidu-2.control"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## 四、去除停用词\n",
 "对分好词的文本去除停用词"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": 13,
 "metadata": {},
 "outputs": [
 {
 "name": "stdout",
 "output_type": "stream",
 "text": [
 "[['睡觉觉', '好评', '王者', '本地', '最强', '王者', '风衣'], ['大哥', '风衣', '喝酒', '茶', '睡觉觉'], ['大哥', '大嫂', '过年', '茶', '好评']]\n"
 ]
 }
 ],
 "source": [
 "# 采用~~3.对文本进行jieba分词的处理结果：[['我', '是', '你', '的', '谁', '本地', '最强', '王者', '的', '风衣'], ['大哥', '好', '喝酒', '吃', '茶', '睡觉觉', '你', '是', '我', '的', '谁']]\n",
 "\n",
 "text_list = [['我', '是', '你', '睡觉觉', '的', '谁', '好评', '谁', '王者', '本地', '最强', '王者', '的', '风衣'], ['大哥', '风衣', '好', '喝酒', '吃', '茶', '睡觉觉', '你', '是', '我', '的', '谁'], ['大哥', '大嫂', '过年', '吃', '茶', '好评']]\n",
 "\n",
 "# 停用词表\n",
 "# 此处先自己设置\n",
 "stopwords = ['我', '是', '你', '的', '谁', '的', '好', '吃', '你', '是', '我', '的', '谁']\n",
 "\n",
 "# 去掉分词文本中的停用词\n",
 "def drop_stopwords(content, stopwords):\n",
 " content_clean = []\n",
 " for item in content:\n",
 " line_list = []\n",
 " for line in item:\n",
 " if line in stopwords:\n",
 " continue\n",
 " line_list.append(line)\n",
 " content_clean.append(line_list)\n",
 " return content_clean\n",
 "\n",
 "print(drop_stopwords(text_list, stopwords))"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "### 知识点\n",
 "#### 1. 停用词\n",
 "文本中不具有实体意义的词，这些词在文本处理中可以忽略，从而节约空间和提高处理效率。如：“这”、“那”、“的”、“得”、“地”等等\n",
 "\n",
 ">停用词解释参考: \n",
 ">https://en.wikipedia.org/wiki/Stop_word \n",
 ">\n",
 ">https://baike.baidu.com/item/%E5%81%9C%E7%94%A8%E8%AF%8D \n",
 ">\n",
 ">停用词表下载链接; \n",
 ">https://blog.csdn.net/fontthrone/article/details/74200026 \n",
 ">\n",
 ">https://github.com/goto456/stopwords \n",
 ">\n",
 ">https://mlln.cn/2018/09/11/%E8%87%AA%E7%84%B6%E8%AF%AD%E8%A8%80%E5%A4%84%E7%90%86%E5%81%9C%E7%94%A8%E8%AF%8D%E5%A4%A7%E5%85%A8-%E7%99%BE%E5%BA%A6-%E5%93%88%E5%B7%A5%E5%A4%A7-%E5%9B%9B%E5%B7%9D%E5%A4%A7%E5%AD%A6/#menu \n"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## 五、转化文本为tfidf向量\n",
 "将以上对原始语料库*original_corpus*进行分词并去除停用词得到的分词语料库*processed_corpus*转化成TF-IDF向量。"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": 8,
 "metadata": {
 "scrolled": true
 },
 "outputs": [
 {
 "name": "stdout",
 "output_type": "stream",
 "text": [
 "\n",
 "(词：ID)字典：\n",
 "\n",
 "{'喝酒': 6,\n",
 " '大哥': 7,\n",
 " '大嫂': 9,\n",
 " '好评': 0,\n",
 " '最强': 1,\n",
 " '本地': 2,\n",
 " '王者': 3,\n",
 " '睡觉觉': 4,\n",
 " '茶': 8,\n",
 " '过年': 10,\n",
 " '风衣': 5}\n",
 "\n",
 "\n",
 "[[(0, 1), (1, 1), (2, 1), (3, 2), (4, 1), (5, 1)],\n",
 " [(4, 1), (5, 1), (6, 1), (7, 1), (8, 1)],\n",
 " [(0, 1), (7, 1), (8, 1), (9, 1), (10, 1)]]\n",
 "\n",
 "\n",
 "corpus的TF-IDF向量值：\n",
 "\n",
 "[[(0, 0.1457894663281049),\n",
 " (1, 0.39501820517610303),\n",
 " (2, 0.39501820517610303),\n",
 " (3, 0.7900364103522061),\n",
 " (4, 0.1457894663281049),\n",
 " (5, 0.1457894663281049)],\n",
 " [(4, 0.29693793311264666),\n",
 " (5, 0.29693793311264666),\n",
 " (6, 0.8045566825992793),\n",
 " (7, 0.29693793311264666),\n",
 " (8, 0.29693793311264666)],\n",
 " [(0, 0.23780622519852498),\n",
 " (7, 0.23780622519852498),\n",
 " (8, 0.23780622519852498),\n",
 " (9, 0.6443386523290703),\n",
 " (10, 0.6443386523290703)]]\n",
 "\n",
 "\n",
 "单独获得的Tf-IDF值：\n",
 "\n",
 "[(0, 0.1457894663281049),\n",
 " (1, 0.39501820517610303),\n",
 " (2, 0.39501820517610303),\n",
 " (3, 0.7900364103522061),\n",
 " (4, 0.1457894663281049),\n",
 " (5, 0.1457894663281049)]\n",
 "[(4, 0.29693793311264666),\n",
 " (5, 0.29693793311264666),\n",
 " (6, 0.8045566825992793),\n",
 " (7, 0.29693793311264666),\n",
 " (8, 0.29693793311264666)]\n",
 "[(0, 0.23780622519852498),\n",
 " (7, 0.23780622519852498),\n",
 " (8, 0.23780622519852498),\n",
 " (9, 0.6443386523290703),\n",
 " (10, 0.6443386523290703)]\n"
 ]
 }
 ],
 "source": [
 "corpus = [['睡觉觉', '好评', '王者', '本地', '最强', '王者', '风衣'], ['大哥', '风衣', '喝酒', '茶', '睡觉觉'], ['大哥', '大嫂', '过年', '茶', '好评']]\n",
 "\n",
 "import gensim,pprint\n",
 "\n",
 "from gensim import corpora,models\n",
 "\n",
 "dictionary = corpora.Dictionary(corpus) # 使用gensim.corpora.Dictionary(corpus)类对分词语料库corpus建立（词：ID）一一对应的词典\n",
 "print(\"\\n(词：ID)字典：\\n\")\n",
 "pprint.pprint(dictionary.token2id) # 输出（词：ID）一一对应的元组列表\n",
 "print(\"\\n\")\n",
 "\n",
 "bow_vec = [dictionary.doc2bow(text) for text in corpus] #将词典通过词袋模型的doc2bow转化为bow向量\n",
 "pprint.pprint(bow_vec)\n",
 "\n",
 "# 使用gensim.models.TfidfModel(bow_vec)训练tfidf模型\n",
 "tfidf = models.TfidfModel(bow_vec)\n",
 "\n",
 "# 将训练好的模型持久化到磁盘上\n",
 "tfidf.save(\"my_model.tfidf\") # tfidf.save(custom_name)# 保存训练好的tfidf模型\n",
 "# 使用时载入模型\n",
 "tfidf = models.TfidfModel.load(\"my_model.tfidf\")\n",
 "\n",
 "\n",
 "# 使用TF-IDF模型将bow向量转化为tfidf向量\n",
 "tfidf_vec = []\n",
 "for i in range(len(corpus)):\n",
 " string = corpus[i]\n",
 " # pprint.pprint(string)\n",
 " \n",
 " string_bow = dictionary.doc2bow(string) # 利用词袋模型的doc2bow进行分词\n",
 " # pprint.pprint(string_bow)\n",
 " \n",
 " string_tfidf = tfidf[string_bow] #使用训练好的tfidf模型对单个词进行转化\n",
 " tfidf_vec.append(string_tfidf) #将转化好的每个词的tfidf向量组合成语料库的tfidf向量\n",
 "\n",
 "# 以上任务已完成，以下进行独立验证。\n",
 "\n",
 "print(\"\\n\\ncorpus的TF-IDF向量值：\\n\")\n",
 "pprint.pprint(tfidf_vec) # 输出tfidf向量\n",
 "\n",
 "print(\"\\n\\n单独获得的Tf-IDF值：\\n\")\n",
 "\n",
 "words = \"睡觉觉 好评 王者 本地 最强 王者 风衣\".lower().split()\n",
 "pprint.pprint(tfidf[dictionary.doc2bow(words)])\n",
 "\n",
 "words = \"大哥 风衣 喝酒 茶 睡觉觉\".lower().split()\n",
 "pprint.pprint(tfidf[dictionary.doc2bow(words)])\n",
 "\n",
 "words = \"大哥 大嫂 过年 茶 好评\".lower().split()\n",
 "pprint.pprint(tfidf[dictionary.doc2bow(words)])"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "### 知识点\n",
 "\n",
 "#### 1. gensim库\n",
 "关于tfidf的介绍看另外几篇jupyter笔记**TF-IDF Algorithm** *（关于TD-IDF算法的介绍）*、**Gensim_Example** *（gensim官网例子笔记）*、**Computing TF-IDF** *（计算TD-IDF的各种方法）*。\n",
 "\n",
 ">gensim库参考资料： \n",
 ">- https://radimrehurek.com/gensim/auto_examples/core/run_core_concepts.html\n",
 ">- https://www.jianshu.com/p/9ac0075cc4c0 \n",
 ">- https://www.kesci.com/mw/project/5b7a35c131902f000f551567\n",
 ">\n",
 ">TF-IDF参考资料： \n",
 ">- http://www.ruanyifeng.com/blog/2013/03/tf-idf.html\n",
 ">- https://www.jianshu.com/p/9ac0075cc4c0\n",
 ">- https://www.zybuluo.com/lianjizhe/note/1212780\n",
 "\n",
 "\n",
 "#### 2. **[`corpora.dictionary` – Construct word<->id mappings](https://radimrehurek.com/gensim/corpora/dictionary.html)**\n",
 "\n",
 "**`class gensim.corpora.dictionary.Dictionary(documents = None, prune_at = 2000000)`**\n",
 "- Bases: [`gensim.utils.SaveLoad`](https://radimrehurek.com/gensim/utils.html#gensim.utils.SaveLoad), collections.abc.Mapping \n",
 "\n",
 "- Dictionary encapsulates the mapping between normalized words and their integer ids. \n",
 "\n",
 "- Notable instance （实例） attributes:\n",
 "\n",
 "- - `token2id`\n",
 "- - - token -> tokenId. \n",
 "- - - - Type: dict of (str, int)\n",
 " \n",
 "- - `id2token`\n",
 "- - - Reverse mapping for token2id, initialized in a lazy manner to save memory (not created until needed). \n",
 "- - - - Type: dict of (int, str) \n",
 "\n",
 "- - `doc2bow(document, allow_update=False, return_missing=False)`\n",
 "- - - Convert document into the bag-of-words (BoW) format = list of (token_id, token_count) tuples.\n",
 "\n",
 "- - - Parameters:\n",
 "- - - - **document** (list of str) – Input document.\n",
 "- - - - **allow_update** (bool, optional) – Update self, by adding new tokens from document and updating internal corpus statistics.\n",
 "- - - - **return_missing** (bool, optional) – Return missing tokens (tokens present in document but not in self) with frequencies?\n",
 "\n",
 "- - - Returns:\n",
 "- - - - list of (int, int) – BoW representation of document.\n",
 "- - - - list of (int, int), dict of (str, int) – If return_missing is True, return BoW representation of document + dictionary with missing tokens and their frequencies. \n",
 "\n",
 "- - - Examples\n",
 "```python\n",
 "from gensim.corpora import Dictionary\n",
 "dct = Dictionary([\"máma mele maso\".split(), \"ema má máma\".split()])\n",
 "dct.doc2bow([\"this\", \"is\", \"máma\"])\n",
 "[(2, 1)]\n",
 "dct.doc2bow([\"this\", \"is\", \"máma\"], return_missing=True)\n",
 "([(2, 1)], {u'this': 1, u'is': 1})\n",
 "```"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "#### 3. **[`models.tfidfmodel` – TF-IDF model](https://radimrehurek.com/gensim/models/tfidfmodel.html#id2)**\n",
 "- This module implements functionality related to the Term Frequency - Inverse Document Frequency <https://en.wikipedia.org/wiki/Tf%E2%80%93idf> vector space bag-of-words models.\n",
 "\n",
 "- For a more in-depth exposition of TF-IDF and its various SMART variants (normalization, weighting schemes), see the blog post at https://rare-technologies.com/pivoted-document-length-normalisation/ \n",
 "\n",
 "\n",
 "**`class gensim.models.tfidfmodel.TfidfModel(corpus=None, id2word=None, dictionary=None, wlocal=<function identity>, wglobal=<function df2idf>, normalize=True, smartirs=None, pivot=None, slope=0.25)`** \n",
 "\n",
 "- Bases:` [gensim.interfaces.TransformationABC]`(https://radimrehurek.com/gensim/interfaces.html#gensim.interfaces.TransformationABC)\n",
 "\n",
 "\n",
 "- Objects of this class realize the transformation between word-document co-occurrence matrix (int) into a locally/globally weighted TF-IDF matrix (positive floats). \n",
 "\n",
 "- **Examples** \n",
 "\n",
 "```Python\n",
 "import gensim.downloader as api \n",
 "from gensim.models import TfidfModel \n",
 "from gensim.corpora import Dictionary \n",
 "dataset = api.load(\"text8\") \n",
 "dct = Dictionary(dataset) #fit dictionary \n",
 "corpus = [dct.doc2bow(line) for line in dataset] #convert corpus to BoW format \n",
 "model = TfidfModel(corpus) #fit model \n",
 "vector = model[corpus[0]] #apply model to the first corpus document \n",
 "```\n",
 "\n",
 "- Compute TF-IDF by multiplying a local component (term frequency) with a global component (inverse document frequency), and normalizing the resulting documents to unit length. Formula for non-normalized weight of term i in document j in a corpus of D documents\n",
 "\n",
 "$$ weight_{i,j} = frequency_{i,j} * log_2 \\frac{D}{document\\_freq_{i}} $$\n",
 "\n",
 "- or, more generally\n",
 "\n",
 "$$ weight_{i,j} = wlocal(frequency_{i,j}) * wglobal(document\\_freq_{i}, D) $$\n",
 "\n",
 "- so you can plug in your own custom wlocal and wglobal functions.\n",
 "\n",
 "\n",
 "- - Parameters\n",
 "\n",
 "\n",
 "- - - **corpus** (iterable of iterable of (int, int), optional) – Input corpus\n",
 "\n",
 "\n",
 "- - - **id2word** ({dict, `Dictionary`}, optional) – Mapping token - id, that was used for converting input data to bag of words format.\n",
 "\n",
 "\n",
 "- - - **dictionary** (`Dictionary`) – If dictionary is specified, it must be a corpora.Dictionary object and it will be used. to directly construct the inverse document frequency mapping (then corpus, if specified, is ignored).\n",
 "\n",
 "\n",
 "- - - **wlocals** (callable, optional) – Function for local weighting, default for wlocal is identity() (other options: numpy.sqrt(), lambda tf: 0.5 + (0.5 * tf / tf.max()), etc.).\n",
 "\n",
 "\n",
 "- - - **wglobal** (callable, optional) – Function for global weighting, default is df2idf().\n",
 "\n",
 "\n",
 "- - - **normalize** ({bool, callable}, optional) – Normalize document vectors to unit euclidean length? You can also inject your own function into normalize.\n",
 "\n",
 "\n",
 "- - - **smartirs** (str, optional) – SMART (System for the Mechanical Analysis and Retrieval of Text) Information Retrieval System, a mnemonic scheme for denoting tf-idf weighting variants in the vector space model. The mnemonic for representing a combination of weights takes the form XYZ, for example ‘ntc’, ‘bpn’ and so on, where the letters represents the term weighting of the document vector.\n",
 "\n",
 "- - - **`Term frequency weighing:`**\n",
 "+ + + + b - binary,\n",
 "\n",
 "+ + + + t or n - raw,\n",
 "\n",
 "+ + + + a - augmented,\n",
 "\n",
 "+ + + + l - logarithm,\n",
 "\n",
 "+ + + + d - double logarithm,\n",
 "\n",
 "+ + + + L - log average.\n",
 "\n",
 "- - - **`Document frequency weighting:`**\n",
 "+ + + + x or n - none,\n",
 "\n",
 "+ + + + f - idf,\n",
 "\n",
 "+ + + + t - zero-corrected idf,\n",
 "\n",
 "+ + + + p - probabilistic idf.\n",
 "\n",
 "- - - **`Document normalization:`**\n",
 "+ + + + x or n - none,\n",
 "\n",
 "+ + + + c - cosine,\n",
 "\n",
 "+ + + + u - pivoted unique,\n",
 "\n",
 "+ + + + b - pivoted character length.\n",
 "\n",
 "\n",
 "- - - Default is ‘nfc’. For more information visit [SMART Information Retrieval System](https://en.wikipedia.org/wiki/SMART_Information_Retrieval_System). \n",
 "\n",
 "\n",
 "- - - **pivot** (float or None, optional) – \n",
 "- - - In information retrieval, TF-IDF is biased against long documents 1. Pivoted document length normalization solves this problem by changing the norm of a document to slope * old_norm + (1.0 - slope) * pivot.\n",
 "\n",
 "\n",
 "- - - You can either set the pivot by hand, or you can let Gensim figure it out automatically with the following two steps:\n",
 "\n",
 "+ + + + Set either the u or b document normalization in the smartirs parameter.\n",
 "\n",
 "+ + + + Set either the corpus or dictionary parameter. The pivot will be automatically determined from the properties of the corpus or dictionary.\n",
 "\n",
 "\n",
 "- - - If pivot is None and you don’t follow steps 1 and 2, then pivoted document length normalization will be disabled. Default is None.\n",
 "\n",
 "\n",
 "- - - See also the blog post at https://rare-technologies.com/pivoted-document-length-normalisation/. \n",
 "\n",
 "\n",
 "- - - **slope** (float, optional) –\n",
 "\n",
 "+ + + In information retrieval, TF-IDF is biased against long documents [^1]. Pivoted document length normalization solves this problem by changing the norm of a document to slope * old_norm + (1.0 - slope) * pivot.\n",
 "\n",
 "\n",
 "+ + + Setting the slope to 0.0 uses only the pivot as the norm, and setting the slope to 1.0 effectively disables pivoted document length normalization. Singhal [^2] suggests setting the slope between 0.2 and 0.3 for best results. Default is 0.25.\n",
 "\n",
 "\n",
 "+ + + See also the blog post at https://rare-technologies.com/pivoted-document-length-normalisation/.\n",
 "\n",
 "> *See also*\n",
 "> \n",
 "> ~gensim.sklearn_api.tfidf.TfIdfTransformer : Class that also uses the SMART scheme. resolve_weights : Function that also uses the SMART scheme. \n",
 "\n",
 "\n",
 "**References**\n",
 "\n",
 ">[1] (1,2)\n",
 "[Singhal, A., Buckley, C., & Mitra, M. (1996). Pivoted Document Length Normalization. SIGIR Forum, 51, 176–184.](http://singhal.info/pivoted-dln.pdf)\n",
 ">\n",
 ">[2]\n",
 "[Singhal, A. (2001). Modern information retrieval: A brief overview. IEEE Data Eng. Bull., 24(4), 35–43.](http://singhal.info/ieee2001.pdf) \n",
 "\n",
 "\n",
 "- - `classmethod load(*args, **kwargs)`\n",
 "- - - Load a previously saved TfidfModel class. Handles backwards compatibility from older TfidfModel versions which did not use pivoted document normalization.\n",
 "\n",
 "\n",
 "- - `save(fname_or_handle, separately=None, sep_limit=10485760, ignore=frozenset({}), pickle_protocol=2)`\n",
 "- - - Save the object to a file.\n",
 "\n",
 "- - - Parameters\n",
 "- - - - **fname_or_handle** (str or file-like) – Path to output file or already opened file-like object. If the object is a file handle, no special array handling will be performed, all attributes will be saved to the same file.\n",
 "\n",
 "- - - - **separately** (list of str or None, optional) – If None, automatically detect large numpy/scipy.sparse arrays in the object being stored, and store them into separate files. This prevent memory errors for large objects, and also allows [memory-mapping](https://en.wikipedia.org/wiki/Mmap) the large arrays for efficient loading and sharing the large arrays in RAM between multiple processes.\n",
 "- - - - If list of str: store these attributes into separate files. The automated size check is not performed in this case.\n",
 "\n",
 "- - - - **sep_limit** (int, optional) – Don’t store arrays smaller than this separately. In bytes.\n",
 "\n",
 "- - - - **ignore** (frozenset of str, optional) – Attributes that shouldn’t be stored at all.\n",
 "\n",
 "- - - - **pickle_protocol** (int, optional) – Protocol number for pickle.\n",
 "\n",
 ">See also\n",
 ">\n",
 ">`load()` \n",
 "> Load object from file.\n",
 "\n",
 "**`gensim.models.tfidfmodel.df2idf(docfreq, totaldocs, log_base=2.0, add=0.0)`**\n",
 "- Compute inverse-document-frequency for a term with the given document frequency docfreq: $$ idf = add + log_{log\\_base} \\frac{totaldocs}{docfreq} $$\n",
 "\n",
 "\n",
 "- Parameters:\n",
 "- - **docfreq** ({int, float}) – Document frequency.\n",
 "\n",
 "- - **totaldocs** (int) – Total number of documents.\n",
 "\n",
 "- - **g_base** float, optional) – Base of logarithm.\n",
 "\n",
 "- - **add** (float, optional) – Offset.\n",
 "\n",
 "\n",
 "- Returns:\n",
 "- - Inverse document frequency.\n",
 "\n",
 "\n",
 "- Return type:\n",
 "- - float"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "#### 4. [pprint](https://docs.python.org/3/library/pprint.html)\n",
 "对字典和数列进行优美地打印\n",
 "\n",
 "- import pprint\n",
 "- pprint.pprint(object)\n",
 "\n",
 "- pprint 模块提供了“美化打印”任意 Python 数据结构的功能，这种美化形式可用作对解释器的输入。 如果经格式化的结构包含非基本 Python 类型的对象，则其美化形式可能无法被加载。 包含文件、套接字或类对象，以及许多其他不能用 Python 字面值来表示的对象都有可能导致这样的结果。\n"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 "## 六、将原始语料库转化为tfidf向量的过程总结\n",
 "- 1. **获得分词语料库**。对原始语料库*`original_corpus`*进行分词并去除停用词得到分词语料库*`processed_corpus`*。\n",
 "- 2. **获得字典**。使用`gensim.corpora.Dictionary(processed_corpus)`对分词语料库建立字典*`dictionary`*。（`dictionary.token2id`方法查看“词”与“ID”的一一对应）\n",
 "- 3. **获得bow向量**。使用字典的`dictionary.doc2bow(prrocessed_corpus)`将分词语料库转化为bow向量*`bow_corpus`*。（processed_corpus一层列表）\n",
 "- 4. **训练tfidf模型**。使用`gensim.models.TfidfModel(bow_corpus)`通过bow向量训练tfidf模型*`tfidf model`*。（bow_corpus二层向量列表）\n",
 "- 5. **获得tfidf向量**。使用训练好的模型`tfidf[bow_corpus]`将bow向量转化为tfidf向量*`tfidf_corpus`*。（bow_corpus一层向量列表）\n",
 "- 6. **对向量后续运算**。对tfidf向量进行其他算法运算，如文章相似度计算，调用sklearn库算法等等。 "
 ]
 },
 {
 "cell_type": "code",
 "execution_count": 1,
 "metadata": {},
 "outputs": [
 {
 "name": "stdout",
 "output_type": "stream",
 "text": [
 "[(0, 0.1457894663281049), (1, 0.39501820517610303), (2, 0.39501820517610303), (3, 0.7900364103522061), (4, 0.1457894663281049), (5, 0.1457894663281049)]\n"
 ]
 }
 ],
 "source": [
 "# 通过加载已经建立好的tfidf模块进行tfidf向量转化\n",
 "\n",
 "from gensim import corpora,models\n",
 "\n",
 "corpus = [['睡觉觉', '好评', '王者', '本地', '最强', '王者', '风衣'], ['大哥', '风衣', '喝酒', '茶', '睡觉觉'], ['大哥', '大嫂', '过年', '茶', '好评']]\n",
 "dictionary = corpora.Dictionary(corpus) # 使用gensim.corpora.Dictionary(corpus)类对分词语料库建立（词：ID）一一对应的词典\n",
 "\n",
 "content = ['睡觉觉', '好评', '王者', '本地', '最强', '王者', '风衣']\n",
 "tfidf = models.TfidfModel.load(\"my_model.tfidf\") #加载已训练好的模块\n",
 "print(tfidf[dictionary.doc2bow(content)])"
 ]
 },
 {
 "cell_type": "markdown",
 "metadata": {},
 "source": [
 ">Jupyter Notebook官方手册：https://jupyter-notebook.readthedocs.io/en/latest/index.html#"
 ]
 },
 {
 "cell_type": "code",
 "execution_count": null,
 "metadata": {},
 "outputs": [],
 "source": []
 }
 ],
 "metadata": {
 "kernelspec": {
 "display_name": "Python 3",
 "language": "python",
 "name": "python3"
 },
 "language_info": {
 "codemirror_mode": {
 "name": "ipython",
 "version": 3
 },
 "file_extension": ".py",
 "mimetype": "text/x-python",
 "name": "python",
 "nbconvert_exporter": "python",
 "pygments_lexer": "ipython3",
 "version": "3.7.6"
 },
 "toc": {
 "base_numbering": "0",
 "nav_menu": {},
 "number_sections": false,
 "sideBar": true,
 "skip_h1_title": true,
 "title_cell": "Contents/目录",
 "title_sidebar": "中文文本预处理",
 "toc_cell": true,
 "toc_position": {
 "height": "calc(100% - 180px)",
 "left": "10px",
 "top": "150px",
 "width": "288.2px"
 },
 "toc_section_display": true,
 "toc_window_display": false
 }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}

一键复制编辑原始数据按行查看历史

提交于 2020-12-23 08:23 . Jupyter Notebook

马建仓 AI 助手

尝试更多

代码解读

代码找茬

代码优化

Python

https://gitee.com/d-benhua/Preprocessing_Chinese_Text.git

git@gitee.com:d-benhua/Preprocessing_Chinese_Text.git

d-benhua

Preprocessing_Chinese_Text

main

d_benhua / Preprocessing_Chinese_Text

简介

发行版

贡献者

近期动态

d_benhua / Preprocessing_Chinese_Text .gitee-modal { width: 500px !important; }

简介

发行版

贡献者

近期动态

搜索帮助

d_benhua / Preprocessing_Chinese_Text