Numpy_Task001.ipynb · 孟庆国/DataWhale学习记录

加入 Gitee

与超过 1200万开发者一起发现、参与优秀开源项目，私有仓库也完全免费：）

该仓库未声明开源许可证文件（LICENSE），使用请关注具体项目描述及其代码上游依赖。

克隆/下载

Numpy_Task001.ipynb 13.86 KB

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 二进制文件\n",
    "\n",
    "`save()`、`savez()`和`load()`函数以 numpy 专用的二进制类型（npy、npz）保存和读取数据，这三个函数会自动处理ndim、dtype、shape等信息，使用它们读写数组非常方便，但是`save()`输出的文件很难与其它语言编写的程序兼容。           \n",
    "             \n",
    "npy格式：以二进制的方式存储文件，在二进制文件第一行以文本形式保存了数据的元信息（ndim，dtype，shape等），可以用二进制工具查看内容。\n",
    "\n",
    "npz格式：以压缩打包的方式存储文件，可以用压缩软件解压。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `numpy.save(file, arr, allow_pickle=True, fix_imports=True)` Save an array to a binary file in NumPy `.npy` format.\n",
    "- `numpy.load(file, mmap_mode=None, allow_pickle=False, fix_imports=True, encoding='ASCII')` Load arrays or pickled objects from ``.npy``, ``.npz`` or pickled files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1 2 3]\n",
      " [4 5 6]]\n",
      "[[1 2 3]\n",
      " [4 5 6]]\n"
     ]
    }
   ],
   "source": [
    "outfile = r\"./data/numpy_task001_1.npy\"\n",
    "x = np.array([[1,2,3],[4,5,6]])\n",
    "print(x)\n",
    "np.save(outfile, x) # 参数为：保存路径 保存的数组\n",
    "y = np.load(outfile)\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['z_d', 'arr_0', 'arr_1']\n"
     ]
    }
   ],
   "source": [
    "outfile = r\"./data/numpy_task001_2.npz\"\n",
    "x = np.linspace(0, np.pi, 5)\n",
    "y = np.sin(x)\n",
    "z = np.cos(x)\n",
    "np.savez(outfile, x, y, z_d=z)\n",
    "data = np.load(outfile)\n",
    "np.set_printoptions(suppress=True) # 设置小数不需要以科学计数法形式输出\n",
    "print(data.files)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`savez()`第一个参数是文件名，其后的参数都是需要保存的数组，也可以使用关键字参数为数组起一个名字，非关键字参数传递的数组会自动起名为arr_0, arr_1, …。\n",
    "\n",
    "`savez()`输出的是一个压缩文件（扩展名为npz），其中每个文件都是一个`save()`保存的npy文件，文件名对应于数组名。`load()`自动识别`npz`文件，并且返回一个类似于字典的对象，可以通过数组名作为关键字获取数组的内容。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.         0.78539816 1.57079633 2.35619449 3.14159265]\n",
      "[0.         0.70710678 1.         0.70710678 0.        ]\n",
      "[ 1.          0.70710678  0.         -0.70710678 -1.        ]\n"
     ]
    }
   ],
   "source": [
    "print(data['arr_0'], data['arr_1'], data['z_d'], sep='\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "解压之后文件内容为：![](./data/images/image_23.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 文本文件"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`savetxt()`，`loadtxt()`和`genfromtxt()`函数用来存储和读取文本文件（如TXT，CSV等）。`genfromtxt()`比`loadtxt()`更加强大，可对缺失数据进行处理。\n",
    "\n",
    "* `numpy.savetxt(fname, X, fmt='%.18e', delimiter=' ', newline='\\n', header='', footer='', comments='# ', encoding=None)` Save an array to a text file.\n",
    " * fname：文件路径\n",
    " * X：存入文件的数组。\n",
    " * fmt：写入文件中每个元素的字符串格式，默认'%.18e'（保留18位小数的浮点数形式）。\n",
    " * delimiter：分割字符串，默认以空格分隔。\n",
    " \n",
    " \n",
    " \n",
    "* `numpy.loadtxt(fname, dtype=float, comments='#', delimiter=None, converters=None, skiprows=0, usecols=None, unpack=False, ndmin=0, encoding='bytes', max_rows=None) `Load data from a text file.\n",
    " * fname：文件路径。\n",
    " * dtype：数据类型，默认为float。\n",
    " * comments: 字符串或字符串组成的列表，默认为# ， 表示注释字符集开始的标志。\n",
    " * skiprows：跳过多少行，一般跳过第一行表头。\n",
    " * usecols：元组（元组内数据为列的数值索引）， 用来指定要读取数据的列（第一列为0）。\n",
    " * unpack：当加载多列数据时是否需要将数据列进行解耦赋值给不同的变量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1 2 3]\n",
      " [4 5 6]]\n",
      "int32\n",
      "\n",
      "[[1. 2. 3.]\n",
      " [4. 5. 6.]]\n",
      "float64\n"
     ]
    }
   ],
   "source": [
    "outfile = r\"./data/numpy_task001_1.txt\"\n",
    "x = np.array([[1,2,3],[4,5,6]])\n",
    "np.savetxt(outfile, x)\n",
    "y = np.loadtxt(outfile)\n",
    "print(x, x.dtype, sep='\\n', end='\\n\\n')\n",
    "print(y, y.dtype, sep='\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`numpy_task001_1.txt`文件内容：\n",
    "```\n",
    "1.000000000000000000e+00 2.000000000000000000e+00 3.000000000000000000e+00\n",
    "4.000000000000000000e+00 5.000000000000000000e+00 6.000000000000000000e+00\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1.1 2.2 3.3]\n",
      " [4.4 5.5 6.6]]\n",
      "float32\n",
      "\n",
      "[[1 2 3]\n",
      " [4 5 6]]\n",
      "int32\n"
     ]
    }
   ],
   "source": [
    "outfile = r\"./data/numpy_task001_2.csv\"\n",
    "x = np.array([[1.1,2.2,3.3],[4.4,5.5,6.6]], dtype=np.float32)\n",
    "np.savetxt(outfile, x, fmt='%d', delimiter=',') # 保存成整数，以逗号分隔每个数字\n",
    "y = np.loadtxt(outfile, delimiter=',', dtype=np.int32)\n",
    "print(x, x.dtype, sep='\\n', end='\\n\\n')\n",
    "print(y, y.dtype, sep='\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`numpy_task001_2.csv`文件内容为：\n",
    "```\n",
    "1,2,3\n",
    "4,5,6\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "创建一个csv文件，内容为：\n",
    "```\n",
    "id,value1,value2,value3\n",
    "1,123,1.0,23\n",
    "2,110,0.5,18\n",
    "3,164,2.1,19\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[  1.  123.    1.   23. ]\n",
      " [  2.  110.    0.5  18. ]\n",
      " [  3.  164.    2.1  19. ]]\n"
     ]
    }
   ],
   "source": [
    "outfile = r'./data/numpy_task001_data.csv'\n",
    "x = np.loadtxt(outfile, delimiter=',', skiprows=1) # 跳过第一行表头哦\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[123.    1. ]\n",
      " [110.    0.5]\n",
      " [164.    2.1]]\n"
     ]
    }
   ],
   "source": [
    "x = np.loadtxt(outfile, delimiter=',', skiprows=1, usecols=(1,2))\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[123. 110. 164.] [1.  0.5 2.1]\n"
     ]
    }
   ],
   "source": [
    "x1, x2 = np.loadtxt(outfile, delimiter=',', skiprows=1, usecols=(1,2), unpack=True)\n",
    "print(x1, x2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`genfromtxt()`是面向结构数组和缺失数据处理的。\n",
    "\n",
    "* `numpy.genfromtxt(fname, dtype=float, comments='#', delimiter=None, skip_header=0, skip_footer=0, converters=None, missing_values=None, filling_values=None, usecols=None, names=None, excludelist=None, deletechars=''.join(sorted(NameValidator.defaultdeletechars)), replace_space='_', autostrip=False, case_sensitive=True, defaultfmt=\"f%i\", unpack=None, usemask=False, loose=True, invalid_raise=True, max_rows=None, encoding='bytes')` Load data from a text file, with missing values handled as specified.\n",
    " * names：设置为True时，程序将把第一行作为列名称。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(1., 123., 1. , 23.) (2., 110., 0.5, 18.) (3., 164., 2.1, 19.)]\n",
      "[('id', '<f8'), ('value1', '<f8'), ('value2', '<f8'), ('value3', '<f8')]\n"
     ]
    }
   ],
   "source": [
    "outfile = r'./data/numpy_task001_data.csv'\n",
    "x = np.genfromtxt(outfile, delimiter=',', names=True)\n",
    "print(x) # x的类型为 <class 'numpy.ndarray'>\n",
    "print(x.dtype)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1. 2. 3.]\n",
      "[123. 110. 164.]\n",
      "[1.  0.5 2.1]\n",
      "[23. 18. 19.]\n"
     ]
    }
   ],
   "source": [
    "print(x['id'])\n",
    "print(x['value1'])\n",
    "print(x['value2']) \n",
    "print(x['value3'])  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "修改csv文件，使其中存在空值，内容为：\n",
    "```\n",
    "id,value1,value2,value3\n",
    "1,,1.0,23\n",
    "2,110,,18\n",
    "3,164,2.1,\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1. 2. 3.]\n",
      "[ nan 110. 164.]\n",
      "[1.  nan 2.1]\n",
      "[23. 18. nan]\n"
     ]
    }
   ],
   "source": [
    "x = np.genfromtxt(outfile, delimiter=',', names=True)\n",
    "print(x['id'])\n",
    "print(x['value1'])\n",
    "print(x['value2']) \n",
    "print(x['value3'])  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 文本格式选项\n",
    "\n",
    "- `numpy.set_printoptions(precision=None,threshold=None, edgeitems=None,linewidth=None, suppress=None, nanstr=None, infstr=None,formatter=None, sign=None, floatmode=None, **kwarg)` Set printing options.\n",
    "    - `precision`：设置浮点精度，控制输出的小数点个数，默认是8。\n",
    "    - `threshold`：概略显示，超过该值则以“…”的形式来表示，默认是1000。\n",
    "    - `linewidth`：用于确定每行多少字符数后插入换行符，默认为75。\n",
    "    - `suppress`：当`suppress=True`，表示小数不需要以科学计数法的形式输出，默认是False。\n",
    "    - `nanstr`：浮点非数字的字符串表示形式，默认`nan`。\n",
    "    - `infstr`：浮点无穷大的字符串表示形式，默认`inf`。\n",
    "\n",
    "These options determine the way floating point numbers, arrays and other NumPy objects are displayed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1.1122]\n"
     ]
    }
   ],
   "source": [
    "np.set_printoptions(precision=4)\n",
    "x = np.array([1.112233445566])\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " ...\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]]\n"
     ]
    }
   ],
   "source": [
    "np.set_printoptions(threshold=2000)\n",
    "x = np.ones((50, 50))\n",
    "print(x) # x 中元素个数为2500，大于2000，所以省略显示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'edgeitems': 3, 'threshold': 2000, 'floatmode': 'maxprec', 'precision': 4, 'suppress': True, 'linewidth': 75, 'nanstr': 'nan', 'infstr': 'inf', 'sign': '-', 'formatter': None, 'legacy': False}\n"
     ]
    }
   ],
   "source": [
    "print(np.get_printoptions()) # 返回当前的打印格式"
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": true,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}

一键复制编辑原始数据按行查看历史

提交于 2020-11-23 22:11 . 初始化仓库以及Task001

https://gitee.com/meng_qing_guo/data-whale-learning-record.git

git@gitee.com:meng_qing_guo/data-whale-learning-record.git

meng_qing_guo

data-whale-learning-record

DataWhale学习记录

master

孟庆国 / DataWhale学习记录

简介

发行版

贡献者

近期动态

孟庆国 / DataWhale学习记录 .gitee-modal { width: 500px !important; }

简介

发行版

贡献者

近期动态

搜索帮助

孟庆国 / DataWhale学习记录