6_数据文件的读取.ipynb · 王澎/数据分析实践

加入 Gitee

与超过 1200万开发者一起发现、参与优秀开源项目，私有仓库也完全免费：）

克隆/下载

6_数据文件的读取.ipynb 47.58 KB

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 数据文件的读取详解\n",
    "\n",
    "* read_csv的常用参数\n",
    "* names,sep,dtype,header,parse_dates,encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#help(pd.read_csv)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* sep 用来定义分隔符\n",
    "\n",
    "* names用来自定义列表\n",
    "\n",
    "* dtype来定义列的类型，特别是想改变默认类型的时候"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>brand</th>\n",
       "      <th>behavr</th>\n",
       "      <th>date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>10944750</td>\n",
       "      <td>13451</td>\n",
       "      <td>0</td>\n",
       "      <td>06/04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10944750</td>\n",
       "      <td>13451</td>\n",
       "      <td>2</td>\n",
       "      <td>06/04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>10944750</td>\n",
       "      <td>13451</td>\n",
       "      <td>2</td>\n",
       "      <td>06/04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>10944750</td>\n",
       "      <td>13451</td>\n",
       "      <td>0</td>\n",
       "      <td>06/04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10944750</td>\n",
       "      <td>13451</td>\n",
       "      <td>0</td>\n",
       "      <td>06/04</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       user  brand  behavr   date\n",
       "0  10944750  13451       0  06/04\n",
       "1  10944750  13451       2  06/04\n",
       "2  10944750  13451       2  06/04\n",
       "3  10944750  13451       0  06/04\n",
       "4  10944750  13451       0  06/04"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r1=pd.read_csv(r\"D:\\t_alibaba_data3.txt\",names=[\"user\",\"brand\",\"behavr\",\"date\"],sep=\"\\t\",dtype={\"behavr\":int})\n",
    "r1.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "#r1.behavr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>brand</th>\n",
       "      <th>behavr</th>\n",
       "      <th>date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>10944750</td>\n",
       "      <td>13451</td>\n",
       "      <td>0</td>\n",
       "      <td>06/04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10944750</td>\n",
       "      <td>13451</td>\n",
       "      <td>2</td>\n",
       "      <td>06/04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>10944750</td>\n",
       "      <td>13451</td>\n",
       "      <td>2</td>\n",
       "      <td>06/04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>10944750</td>\n",
       "      <td>13451</td>\n",
       "      <td>0</td>\n",
       "      <td>06/04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10944750</td>\n",
       "      <td>13451</td>\n",
       "      <td>0</td>\n",
       "      <td>06/04</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       user  brand  behavr   date\n",
       "0  10944750  13451       0  06/04\n",
       "1  10944750  13451       2  06/04\n",
       "2  10944750  13451       2  06/04\n",
       "3  10944750  13451       0  06/04\n",
       "4  10944750  13451       0  06/04"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r1=pd.read_csv(r\"D:\\try\\t_alibaba_data31.txt\",sep=\"\\t\")\n",
    "r1.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* header用来设置使用数据第几行作为列标，和names联合使用可以替代数据列标"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>10944750</td>\n",
       "      <td>13451</td>\n",
       "      <td>0</td>\n",
       "      <td>06/04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10944750</td>\n",
       "      <td>13451</td>\n",
       "      <td>2</td>\n",
       "      <td>06/04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>10944750</td>\n",
       "      <td>13451</td>\n",
       "      <td>2</td>\n",
       "      <td>06/04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>10944750</td>\n",
       "      <td>13451</td>\n",
       "      <td>0</td>\n",
       "      <td>06/04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10944750</td>\n",
       "      <td>13451</td>\n",
       "      <td>0</td>\n",
       "      <td>06/04</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          a      b  c      d\n",
       "0  10944750  13451  0  06/04\n",
       "1  10944750  13451  2  06/04\n",
       "2  10944750  13451  2  06/04\n",
       "3  10944750  13451  0  06/04\n",
       "4  10944750  13451  0  06/04"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r1=pd.read_csv(r\"D:\\try\\t_alibaba_data31.txt\",names=[\"a\",\"b\",\"c\",\"d\"],header=0,sep=\"\\t\")\n",
    "r1.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* parse_dates用来自动解析时间数据，注意前提是格式要符合要求"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>10944750</td>\n",
       "      <td>13451</td>\n",
       "      <td>0</td>\n",
       "      <td>2011-06-04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10944750</td>\n",
       "      <td>13451</td>\n",
       "      <td>2</td>\n",
       "      <td>2011-06-04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>10944750</td>\n",
       "      <td>13451</td>\n",
       "      <td>2</td>\n",
       "      <td>2011-06-04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>10944750</td>\n",
       "      <td>13451</td>\n",
       "      <td>0</td>\n",
       "      <td>2011-06-04</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          a      b  c          d\n",
       "0  10944750  13451  0 2011-06-04\n",
       "1  10944750  13451  2 2011-06-04\n",
       "2  10944750  13451  2 2011-06-04\n",
       "3  10944750  13451  0 2011-06-04"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r1=pd.read_csv(r\"D:\\try\\t_alibaba_data311.txt\",names=[\"a\",\"b\",\"c\",\"d\"], header=0, sep=\"\\t\",parse_dates=[3])\n",
    "r1.head()\n",
    "#通过parse_dates自动解析时间数据，[3]代表解析3号位也就是第四列数据cv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0   2011-06-04\n",
       "1   2011-06-04\n",
       "2   2011-06-04\n",
       "3   2011-06-04\n",
       "Name: d, dtype: datetime64[ns]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r1.d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* xls数据用read_xls，xls转化csv\n",
    "* encoding用来定义解析数据使用的编码格式，默认使用utf-8来进行解析，当你发现数据无法解析的时候可以尝试更换编码格式，再读取带中文的数据，特别是xls转成的csv数据时很常用\n",
    "\n",
    "除utf-8外的主要中文编码：gb2312 gbk gb18030"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>姓名</th>\n",
       "      <th>企业</th>\n",
       "      <th>部门</th>\n",
       "      <th>职位</th>\n",
       "      <th>工号</th>\n",
       "      <th>响应状态</th>\n",
       "      <th>签到时间</th>\n",
       "      <th>是否迟到</th>\n",
       "      <th>非受邀</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>项欣瑶</td>\n",
       "      <td>杭州师范大学阿里巴巴商学院</td>\n",
       "      <td>杭州师范大学阿里巴巴商学院-在校学生-在校本科生-电子商务-19级-电商191</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>未响应</td>\n",
       "      <td>2019-10-15 11:59:52</td>\n",
       "      <td>迟到</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>斯煜佳</td>\n",
       "      <td>杭州师范大学阿里巴巴商学院</td>\n",
       "      <td>杭州师范大学阿里巴巴商学院-在校学生-在校本科生-国际商务-19级-国商191</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>未响应</td>\n",
       "      <td>2019-10-15 11:59:52</td>\n",
       "      <td>迟到</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>王佳旋</td>\n",
       "      <td>杭州师范大学阿里巴巴商学院</td>\n",
       "      <td>杭州师范大学阿里巴巴商学院-在校学生-在校本科生-电子商务-19级-电商191</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>未响应</td>\n",
       "      <td>2019-10-15 11:59:54</td>\n",
       "      <td>迟到</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>罗佳怡</td>\n",
       "      <td>杭州师范大学阿里巴巴商学院</td>\n",
       "      <td>杭州师范大学阿里巴巴商学院-在校学生</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>未响应</td>\n",
       "      <td>2019-10-15 11:59:54</td>\n",
       "      <td>迟到</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>张滨滨</td>\n",
       "      <td>杭州师范大学阿里巴巴商学院</td>\n",
       "      <td>杭州师范大学阿里巴巴商学院-在校学生</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>未响应</td>\n",
       "      <td>2019-10-15 11:59:54</td>\n",
       "      <td>迟到</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    姓名             企业                                       部门  职位  工号 响应状态  \\\n",
       "0  项欣瑶  杭州师范大学阿里巴巴商学院  杭州师范大学阿里巴巴商学院-在校学生-在校本科生-电子商务-19级-电商191 NaN NaN  未响应   \n",
       "1  斯煜佳  杭州师范大学阿里巴巴商学院  杭州师范大学阿里巴巴商学院-在校学生-在校本科生-国际商务-19级-国商191 NaN NaN  未响应   \n",
       "2  王佳旋  杭州师范大学阿里巴巴商学院  杭州师范大学阿里巴巴商学院-在校学生-在校本科生-电子商务-19级-电商191 NaN NaN  未响应   \n",
       "3  罗佳怡  杭州师范大学阿里巴巴商学院                       杭州师范大学阿里巴巴商学院-在校学生 NaN NaN  未响应   \n",
       "4  张滨滨  杭州师范大学阿里巴巴商学院                       杭州师范大学阿里巴巴商学院-在校学生 NaN NaN  未响应   \n",
       "\n",
       "                  签到时间 是否迟到  非受邀  \n",
       "0  2019-10-15 11:59:52   迟到  NaN  \n",
       "1  2019-10-15 11:59:52   迟到  NaN  \n",
       "2  2019-10-15 11:59:54   迟到  NaN  \n",
       "3  2019-10-15 11:59:54   迟到  NaN  \n",
       "4  2019-10-15 11:59:54   迟到  NaN  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r1=pd.read_csv(r\"D:\\try\\xxx.csv\",encoding=\"gb18030\")\n",
    "r1.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* chunksize 当数据超出内存总量时通过设置chunksize来分段读取其中部分，chunksize用来定义一次读取的行数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pandas.io.parsers.TextFileReader at 0x2573fba4278>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv(r\"D:\\t_alibaba_data3.txt\",names=[\"user\",\"brand\",\"behavr\",\"date\"],sep=\"\\t\",chunksize=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       user  brand  behavr   date\n",
      "0  10944750  13451       0  06/04\n",
      "1  10944750  13451       2  06/04\n",
      "2  10944750  13451       2  06/04\n",
      "3  10944750  13451       0  06/04\n",
      "4  10944750  13451       0  06/04\n",
      "5  10944750  13451       0  06/04\n",
      "6  10944750  13451       0  06/04\n",
      "7  10944750  13451       0  06/04\n",
      "8  10944750  21110       0  06/07\n",
      "9  10944750   8689       0  05/02\n",
      "        user  brand  behavr   date\n",
      "10  10944750   8689       2  05/02\n",
      "11  10944750   8689       2  05/02\n",
      "12  10944750   8689       0  05/02\n",
      "13  10944750   8689       0  05/02\n",
      "14  10944750  26619       0  06/28\n",
      "15  10944750   5185       0  07/10\n",
      "16  10944750  18575       0  05/02\n",
      "17  10944750  23662       0  06/19\n",
      "18  10944750  23662       0  06/19\n",
      "19  10944750  15761       0  04/24\n",
      "        user  brand  behavr   date\n",
      "20  10944750  15761       0  04/24\n",
      "21  10944750  15761       0  04/24\n",
      "22  10944750  15761       0  04/24\n",
      "23  10944750  19673       0  07/05\n",
      "24  10944750  19673       0  07/05\n",
      "25  10944750  19673       0  07/05\n",
      "26  10944750  19673       0  07/05\n",
      "27  10944750  19673       0  07/04\n",
      "28  10944750  19673       0  07/04\n",
      "29  10944750  19673       1  07/05\n"
     ]
    }
   ],
   "source": [
    "k=0\n",
    "for i in pd.read_csv(r\"D:\\t_alibaba_data3.txt\",names=[\"user\",\"brand\",\"behavr\",\"date\"],sep=\"\\t\",chunksize=10):\n",
    "    print(i)\n",
    "    k=k+1\n",
    "    if k==3:break\n",
    "#加了chunksize的read_csv变成了一个迭代子,每次读入十行数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "brand\n",
      "1996      4.0\n",
      "4078      1.0\n",
      "5185      1.0\n",
      "6290      3.0\n",
      "7373      4.0\n",
      "8628      5.0\n",
      "8689      5.0\n",
      "9817      1.0\n",
      "11465     1.0\n",
      "13451     8.0\n",
      "13779     1.0\n",
      "14580     1.0\n",
      "15315     2.0\n",
      "15761     4.0\n",
      "15980     1.0\n",
      "18575     1.0\n",
      "19673     7.0\n",
      "21110     1.0\n",
      "21501     4.0\n",
      "23251     3.0\n",
      "23662     2.0\n",
      "25687    29.0\n",
      "26619     1.0\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "k=0\n",
    "for i in pd.read_csv(r\"D:\\t_alibaba_data3.txt\",names=[\"user\",\"brand\",\"behavr\",\"date\"],sep=\"\\t\",chunksize=30):\n",
    "    z=i.groupby(\"brand\").size()\n",
    "    #print(z)\n",
    "    if k==0:\n",
    "        s=z\n",
    "    else:\n",
    "        s=s.add(z,fill_value=0)\n",
    "    k=k+1\n",
    "    if k==3:break\n",
    "print(s)\n",
    "\n",
    "#统计后的结果往往比原始记录小\n",
    "#结果合并的方法根据统计结果而定，如果结果是数据框，则可能需要concat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 课间题\n",
    "\n",
    "* 请尝试读入t_alibaba_data31，列标就使用其数据第一行，注意brand id要设置为字符串\n",
    "\n",
    "* 请尝试读入t_alibaba_data31，列标用a,b,c,d替代\n",
    "\n",
    "* 请读入t_alibaba_data311，注意其时间列要读为时间戳\n",
    "\n",
    "* 请自行把“2019-10-15日程的签到详情(1)”转为csv格式，文件名随意，然后读入，注意编码设置\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### json数据的读取问题\n",
    "\n",
    "* pandas对json数据的读取本质上都是要把json数据转换为数据框"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAABaCAIAAAD/8Fy5AAAVT0lEQVR4Ae1dX2gcR9KvvbtwB5cjkq1oDX5REq9Big2HT585JBTjtcFIhEMYI0wg5EWsBHqwXvKU4EXBb4aPFXyCXaGHmANjFiOEybdLPmIld0ZLEnzGYPkEJ/8RJA+WZVsKdw95uGO+6pnZmdmZnr89MzuzqkFoe6q7q6t+3VNTXd0zk5Ek6ccff3z8+DHQkVoEtra2+vr6Tp06RV3Zrj6kLmgL8r9oS6vUKCFACBAC4giQ/RLHkDgQAoRAexDIfPPNN+1pmVolBAgBQkAMgcwPP/wgxoFqEwKEACHQHgR+dfv27fa0TK0SAoQAISCGwK+w+nfffSfGhGoTAoQAIdAGBJj9wuOTTz5REsn/f/jw4efPn/f29nr5n3x1SEJCgBAIjED61h+9mC2lTGBQqCIhQAikAoH02S+PnheasFR0AAlJCBACgRFIn/0i/ytwZ1NFQqDDEEif/SL/q8OGIKlDCARGIA77dX95aWlp+f5eYCFbKqqxrUymN5sFh/+9vS3V8KQOc5nm35Q5Uzl/MS8XsORuTGXmMpm54XksdmdYTk/V+SxsqOIcbBhbyGVUchheWOhhEBy0MGY5NGUsFgxJB+Z61rLW0xnY0MlhpcS1EOfgSRcjDnceWavcGWYDvszGdSqPKO3Xnmy4lpaevD34TnjgKP4X8vvXP/9p94e5QeJfdViY5Qv65sAQyzh2FP/1HGPJ7ECO/Xg+xDm4NKWa3gy8W3IpKZDtoEV/RSrWCq68HTi41vVaAC041KAosb+JAlTDN2HiWohzcENDvl1rOBRrsJoDnglDPtuzdjlujbQ7P0r71fX785PsOP/7rhDVVP0vN45o5rhFsiV5YFe4mYzIClhye47KRkuupIy83qNHbFnwMsQ58LgaaD2X1Ct2hBnZiA5xLcQ5uKi2gc5zCc6PqsX6K5Afgg1/zrJLE3gPS/54gFE2HjQc8HSmBA//16TayBoj47GzacpJx2mU9isaBDT/y5l9EP/LmaOjz6VOhXCOyf6Gbe5z7l4bXn1skmuZwLqJFmu+m+9ZX1ZxQCimuLM3Nw4QHAc0WNOXWuBQHOYWUjgnzlq8mB+WB4MyJKY26lNzlpiDMweUMjgO4aiYdC7q/tWki2mQT9D/MnDyk8wNZAEUn0u5977ZOn0cWZNGNH6P5su5zE5N0m9+mOXGQaud6IS7FtVyptFbk4qKA4QXbWY4v7k2onmr7hzCBuAv63DB4lELNuKmBYa3quulorSmtoM4jC1CYVxv1o2DXjKclBwcyafTy7JHIPP555/j80MR77/f+nrpXveFcOaR6v77bBaDX6//7ndW1RT68+1t8xQSAwJjbHpouj1rHDCChPGv4zWD063l+UzgvfcmXJu+pF21PusLFUc9r8DMGvQIcQlSmV2l6y3WCrkg8cpAca3VJwrCPVAdjIW9uwm6+QzExG8lVHllvFhpzmGV6lyiX84BymMI/4FcDY0XDwdl2DtcFwHajK1Kuv0vNFV2SJmNl125Jl3t5SF4r3XUNfNdftFgLcw2jIVwQOzHY2jCfI2Mjh8fW8FZZH/ccMj3K5uLNlJZNlYWswMfm5sYrag+qTkj4vPzGAXDJhCNHOxwbs4YOM1XYXUW5qrtueuJ6E/xLxU97GUWyGzAX/2HemXjdWxCkorNv5mSvF4p0jNUVwQBtho7BhMS1+MQYZzaunI4//kV68YahGq1IS9btcNlF8QzffYruvhXj7xw9/wfviHdqEJ+sxK7f+FbzjgqNKrmtYv6yoOhgVjBwaD3AvoSUvwun4Jw/3hhu2pe6YsDfPc2GrDDL9Qb4ao1v8VQqOmzX21cf7RDvOdYY/Wq7rbhWqRpLmlX0UpX15ui2n+qNqjsalzWRbYKEpQyBA9zGZ2zEhG75jv4FRwH3JFZBSgafAl0MGw2aEaFw2glD7Om1UZ5wyp/NdYB6+A4sJqtw4htiysFs+lRAeWgubesSONfGLb/Sv+u0c2lu7JM75ydPN3nTTpeKc3/4gbvtRp+419axQAJ3LqZZ5vy1arZ0ubMxEcLs7m5ainGuLUc7tGkX2hKg3Mok/NTl+O5hRCWKbTWMMFW3BYBhkrT0qds/4SaV5iQ1kztG2uFn97BKGQD5lAUw8ENRkaDg9Iqrkf3sGc2DDIUakVp1HAecRL3kczggpRBgsArU1ECJYhCPOuPgkK2VOe+/wsfJAJJUkyb8X9LTeMFXuBsUmUhzjG5Bje3hVeKT/DGjIbGatZSrFIg0QkHhA29VQx+4eGwzpFkoNI3fzSaJy2NHaCllQlmEP9L3qLMOrOjj41FFqyN1SdKJJ6Eg9YtOB7Mq8ZaHnrWCR4wkc4fDRiEl9TMk9FgIXs7ut6yvAKjn/JS7CEc37EaHqPE0h4BvhftVGfr6AV8wkFGCZ8f0vddc3FLNlDps19Gs6WnQ/G/uP3XYcQjMC11mEqB1CEcPMKWbKDSN3+087OsdI8dRMUIAUIgpQh0uP+1vZ3SfiGxCQFCwB0B8r/cMaIShAAhkEwE0me/9JiX4RNqCK6VnkzESSpCgBAIC4H02S9rnAspCIeVHhZGxIcQIASSiUD67JfVz0IKgmulJxNxkooQIATCQiBy+4XPEGnH11shiG31s7z7X3uLUD4EZcurTZ5+nCkfypTfZ58xuPe+nP7Y38OB4hw8QvN6NZMtD7++57G4v2IOWhizHJgaiwVD0oG5lvWb2wiC+nfwHuezFFrJYAlxLcQ5eJTceTzce58NePZcV4ce0dov/IDHV3BWfgn+5OTZdx5/tSRuwqx+llf/axVuXOZ3Y3duiGX0s2fwu+Wd6Qdyre9X5dfTqeIcdF7c1N78QfmK/fWR0s/cAmEQHbR466o0fb3g2ogDB9e63grU3yhn3oDa9rQk/9X+/X0udBMmroU4Bxc0/IyHV5fh3pYLv5RmR2m/tr6+C4OTp/tUaPpOXxg88Pjplnoa9EfE/8I2D3wG01fNbXcd0T/PoYy8g0eOmAs5notzcGQP0HXppXzFvjzBjGxEh7gW4hzcVBv9CXE4M9osNvrTxRI8CvllNeJaiHNoKmjz6208nPgCLn7GOOw+seGTcnKU+7/6Tk/2tcDT1dUNuy2UACdc/wsD+Fa6X+YOPhdOhb5X3p7BmA6d/HbtRB+HvQMHpfTTj+HLPwN8yLGhHHZtIrlpUb99aKz5HvXCuWeVtyxyunGADsBhb3H4xuVGU/XCuevwZX18+qpmWFlOZ+DQ1DGJv1HaL4u+9+/tDuZPW8j+CN79LyUu5on72wMHABSfS7lzdr/dUu/EF9IJjbA1X/1jZve6dCavkQDcOBiKJjjprkW1eqhx8Lo0rei+OlU+NNxizd05hKt+/Y0bs3CyaU7D4u2mBYa3vtwoTT9rfp4DcfhgET40fJ7DjUNYku5zPvG9PwdjYU/eDuEbHq+99hrHhGWzygc7jF4Ydq1x/z0G7zH+lbsOLXYnUP/jvff/4NpEwd8cM1BT1koYALryn4tr/+qyZkVMYVfpeou1wgaR+N8D01/E/UQ4hvDfkK3Wv09uvjwRb0egyhZXi+FgJUbcITJ7l/GgDHsMm+CXfDvviMf/Yi8y3B0MwXhhB3CMl83+L6P/dfsQsNE+CH8wOk2e+7N1ssCq4YDYj8fghHninB/PfbDyFMA6i4wUn5/PSD+fwRbw6s1ld2uGiFikzTLmT+uLB3KWNex8RfVJI2/fXwNdBTh5C76/DOVbcPELiP+u509cn6Wjt19795dv3u0+O3m+z6doNsWNHpaedot/nXkGf5D9r7+t+va/ZON17NyzNe0Slf0vG/mIHCsCLJz/nypuKBltg0Maq6YBG0P/C0O3nep//SIgKh6roeN1E2eN+iKkx3oOxbz7XyYmXfIk46X/3UJPbsHJbzlRahP/fXF6t2peiV9d2Rwc0Cx7m0Bo/FJ4Xci75G+NFl7dCnnF03vrwUoejHeGHUzIALUitF/oeMm7vwyfrUXS8v0AUhqr6D5XXM8/dvc3vl/Qt7PiWqRh4ckomnsa193YBtr3Yc+9bPASOFnGVm6vBudgW3MQHv0xo3NWImL/4zv4FRyHp1Om7bu4gfO3UPqJZ0GjwiFfOQmz5dYdzvKG1SmcR/s6guPgqxm3wlEB5daueH6E88e93VcAr75a0r/gwcQ9MCgotHf/yxj/EmkUt26eZJvyVR4HPtu8+KePblzOlW+VYoxbY6Bn7DeaGjfwomXHz+ck86W7Kkf6PvQ9TdZ4cxNsxQ13fgyWJp59ehufVVAL4f4JfVrNrRgy8a3K9sX5g031GfOcTfArGhwUdXA9ups9s2FQ7sPa9LOWzROGvCiSnseDa+NRAuXauGCB+NYfBQXVqgdef4RVKH8gs0n29itN02AJZWvVuWdxB9SDSRtdLcIBscXnh5R9iye/BfPCSxP6VAMV4fyxiU/Iv979L3PDeXUvspneWedP/syCtbzpVGfp6aYN4aAhhOPBznhhmVQDFeH8UYMv3AQ3/uVx/z2uJU934i4YHeEteAnwX52to66tfYpwkLHB54f0fddctFIOVPrsl3f/K6z4F7ffE0rsg4lnCRUtVrEIB49wpxyo9M0fuf4XdpaV7rEHqRghQAikFIEO97+y2ZT2C4lNCBAC7giQ/+WOEZUgBAiBZCKQPvvlPf6VTMRJKkKAEAgLgfTZL2ucCykIh5UeFkbEhxAgBJKJQPrsF/lfyRxJJBUhED8Ckdsv4/c7lu+H8Nif1c/y7n+9mIe5DMxNmXHemMrMZTJzw+z7HXeG5fSU/sCjuTTvXJwDj6sN7c6wrAZqkoFlf3LacFTJDloYsxyYGIsFQ9KBuZ61LOvO+jIDGzo5rJS4FuIcPOlixOEO580Eykgps3HNOTam9HFkV4ZTLUmkSO0Xs1369zsmz3bfvSluwoL7X3VYmOVj/+aA/P2OY0cxu0d+FX52wN/3O8Q58CUzU+tsxD2cgKKk/p0P85k7By36K1Kx5r4v1oGDWZXA5+UMQE1VH1/KVw3fhIlrIc7BDR55JGg4FGuwmgOeCUM+27N2OW6NJD4/UvuFL8CfnDzd1wSBfcADnmw1TwP+ivhf2GS2BMWKuemeo/r3O5SR13v0iLmQ47k4B0f2zczyGExIMO37fQ/N+i6/4lqIc3AREX0GKIFmtfsrkB+CjTCdUBRAXAtxDi44wCiz4BoOeDpTgofmt/qMrDEyHju8N2wjeOw+WHNrKsH5kdqvSPQO7n+5iePgc6lTIZxjsr9hm/scOHBQGlc9dssE1k00OR8r99ag31NZkUJuWtSXVRwQiinu7M2NAwTHAa85k/lWHGYRhW3qOmvxYn5YHgzKkJjaqE/NWWIOzhyw2eA42Mi838hx7l/d+vomflDtgiDEXP/L4/OPtk3nBrL4ZmrZ51LunG+2Th9H1qQRrfKj+XIus1OT9JsfZrlx0GoHT2wsAl69GNJYbchMhmBmDXqC8+PUdNeiWs40emtSUZm24kWbGc5vro1o3qo7B06zQqS/rMMFi0ctxNG9NzG8VV0vFaXm9zsQh7FFKIzrzcaNgxwcyfO8LF2mDkzF8f4c9vZ7+SVgB8J4Bf7hw4etJgwyGBNpOZTPeRhJGLzH+NfxmsHpNmb7SeO99yZcm76kXbV+Kgcui8FaKMCDddVsMX2q4ZswB/HYVbreYq2wMBKvDBTXoprSOojDsjAW9u4m6ObTpXg42ajyynix0hp55BLDac+RC46KB3IBNF48HJRhj2ETk9uqM8VI2hiLq9gW0IsmLhWH/8XCYKdRc/Yy6aXds4aIWBA4rMaLUSTJSjdyV3t5CN5rHXXGMg5pNFgLs4rXo5bC/o77wF1u203jhW33XIIZgL/WQ7DH3jUZmjBfI6Pjx8dWcBYZ/bzWJKV82dlctKai4Z5urCxmByzf7xitqD5puI25cjuPUTAshGjkYIdzc8Zhkq/C6izMxXuzcxU8lAJxxr+YHRvcvSe4iYIb/7IaL6QYAcJeZoHMBrve/R6y8To2IUnF5t9MSV6v9MtIsDxOkrMTLRPGnb8LskxrdbYRRl7KMFvTtCokLLcczn9+BV6YOSFUGG9gy1ZhBxvMLbXjPE77pej3andPSFGuqeIaNVMzPUcZ4fk/TGT3040q5DcrsfsXFsHeK8F21TxATYE6S6WQCY2qee2ivvJgaCBWcDDozSbOUvwunwJm/3hhu2pe6QsZ54DsGrDDr9krD35+XpqpUdovtv2rZb8XftADA/in+4QA45oqrlETasZQuedYY/Wq7rbhWqRpLmko65JU15uGzVbIpZqSjTOB4w242dyMiDfW6jr0c2Jwyq7GUHe2NgUcgoe5jM5ZiYhd8x38Co4DLl9UocWXQBxsNl9GhcNoJQ+zptVGecMqfzW2iR3nNzgOrGbrMGLb4krtsukc3WIhRRn/wvniBfz4I9qs5vGOaPALGXFNFdeoNVsV/cWtm3m2KV/lky1tzkx8tDCbm6uW4o5b4zSYXZSzsig26491OZ5bCDksxlbcFgGGStPSp2z/hApGYUJai9X52sEoZAPmUBTDwQ1GRoOD0iquR/ewZzYMMhRqRWnUcB5xEleiZ3ABxyBBKCtTEUsdOvs41h/DFZq7/mhn1FqalgO+jFLgbGFtKZnmE7wxo6HBXa6xmpXkIUY4YJ9om2246xwKRErXpXT9Mcr5YzRj2s5UWenm9uUtymZix53jLjEci/vceGGvEg7a0Mbx0KnrHFHOHzX8Qk1wp4pW46VQTC1jBKnoO1Zj4pHs00eAy66nOltHLz1AOMgo4fND+r5rC27s+aGwd/5aGomWkD77ZWeqrPRokUsm9yMwLSVTsnilIhzixbtdraVv/ujd/2oXptQuIUAIxINA+uyX1c+yo8SDILVCCBAC7UIgffaL/K92jRVqlxBIGgLps1923paVnjSsSR5CgBAIF4H02S/yv8IdAcSNEEgvAumzX1Y/y46S3l4hyQkBQsALAumzX+R/eelXKkMI7AcE0me/7LwtK30/9B/pSAjsZwTSZ7/I/9rP45V0JwSMCKTPfln9LDuKUU9KEwKEQOchkD77Rf5X541C0ogQCIZA+uyXnbdlpQdDhGoRAoRAWhBIn/0i/ystY4vkJASiRiB99svqZ9lRosaO+BMChEB7EUif/SL/q70jhlonBJKDQPrsl523ZaUnB2WShBAgBKJAIH32i/yvKMYB8SQE0ohA+uyX1c+yo6SxP0hmQoAQ8I5A+uwX+V/ee5dKEgKdjUD67Jedt2Wld3bPkXaEACGQPvtF/heNWkKAEFAQ+H8b9EeDwA7pdgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import Image\n",
    "Image(\"4.png\")\n",
    "#json数据的标准格式"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   a  b  c\n",
      "0  1  2  3\n",
      "1  1  3  1\n",
      "2  6  2  3\n"
     ]
    }
   ],
   "source": [
    "d = pd.read_json(r'D:\\try\\js1.json')\n",
    "print(d)\n",
    "#注意pandas要求的json数据要符合上面的标准格式"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "shoes1.json中的一行数据（数据来源为2017年从淘宝爬取，例子只选取了其中五行）\n",
    "\n",
    "{\"_id\":{\"$oid\":\"5aa77042be9b0338dc1faab5\"},\"title\":\"意尔康旗舰店官方店男鞋春秋真皮爸爸鞋中老年人商务休闲皮鞋子男\",\"url\":\"http://detail.tmall.com/item.htm?id=545941839337\\u0026ns=1\\u0026abbucket=9\",\"price\":\"199.00\",\"location\":\"浙江 丽水\",\"sales\":\"1607人付款\",\"nick\":\"意尔康皮鞋旗舰店\",\"itemid\":\"545941839337\",\n",
    " \"info\":{\"上市年份季节\":\"2017年秋季\",\"货号\":\"7341ZL97655W\",\"销售渠道类型\":\"商场同款(线上线下都销售)\",\"鞋垫材质\":\"二层猪皮\",\"鞋跟高\":\"低跟(1-3cm)\",\"品牌\":\"YEARCON/意尔康\",\"鞋头款式\":\"圆头\",\"闭合方式\":\"套脚\",\"鞋底材质\":\"塑胶\",\"鞋面材质\":\"头层牛皮（除牛反绒）\",\"真皮材质工艺\":\"修面皮\",\"鞋面内里材质\":\"二层猪皮\",\"鞋制作工艺\":\"注压鞋\",\"跟底款式\":\"平跟\",\"图案\":\"纯色\",\"流行元素\":\"金属\",\"风格\":\"商务\",\"细分风格\":\"商务休闲\",\"场合\":\"办公室\",\"季节\":\"夏季\",\"颜色分类\":\"黑色棕色黑色加绒版棕色加绒版黑色镂空棕色镂空\",\"尺码\":\"38394041424344\",\"款式\":\"商务休闲鞋\",\"功能\":\"轻质\",\"适用对象\":\"青年（18-40周岁）中年（40-60周岁）老年（60周岁以上）\",\"低帮鞋品名\":\"商务休闲鞋\"}},"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>_id</th>\n",
       "      <th>info</th>\n",
       "      <th>itemid</th>\n",
       "      <th>location</th>\n",
       "      <th>nick</th>\n",
       "      <th>price</th>\n",
       "      <th>sales</th>\n",
       "      <th>title</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>{'$oid': '5aa77041be9b0338dc1faab4'}</td>\n",
       "      <td>{'上市年份季节': '2018年春季', '货号': '8142ZR90254W', '销...</td>\n",
       "      <td>562121114807</td>\n",
       "      <td>浙江 丽水</td>\n",
       "      <td>意尔康皮鞋旗舰店</td>\n",
       "      <td>269</td>\n",
       "      <td>1583人付款</td>\n",
       "      <td>意尔康男鞋2018春季新款英伦真皮商务休闲皮鞋青年男士皮鞋子男潮</td>\n",
       "      <td>http://detail.tmall.com/item.htm?id=5621211148...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>{'$oid': '5aa77042be9b0338dc1faab5'}</td>\n",
       "      <td>{'上市年份季节': '2017年秋季', '货号': '7341ZL97655W', '销...</td>\n",
       "      <td>545941839337</td>\n",
       "      <td>浙江 丽水</td>\n",
       "      <td>意尔康皮鞋旗舰店</td>\n",
       "      <td>199</td>\n",
       "      <td>1607人付款</td>\n",
       "      <td>意尔康旗舰店官方店男鞋春秋真皮爸爸鞋中老年人商务休闲皮鞋子男</td>\n",
       "      <td>http://detail.tmall.com/item.htm?id=5459418393...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>{'$oid': '5aa77042be9b0338dc1faab6'}</td>\n",
       "      <td>{'鞋垫材质': '二层猪皮', '鞋跟高': '平跟(小于等于1cm)', '品牌': '...</td>\n",
       "      <td>563340844728</td>\n",
       "      <td>浙江 温州</td>\n",
       "      <td>吸引力xl</td>\n",
       "      <td>148</td>\n",
       "      <td>248人付款</td>\n",
       "      <td>2017新款正品意尔康男鞋男士商务正装皮鞋透气真皮系带尖头办公室</td>\n",
       "      <td>http://item.taobao.com/item.htm?id=56334084472...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>{'$oid': '5aa77042be9b0338dc1faab7'}</td>\n",
       "      <td>{'货号': '883053', '鞋垫材质': '头层猪皮', '鞋跟高': '平跟(小于...</td>\n",
       "      <td>561684297836</td>\n",
       "      <td>浙江 温州</td>\n",
       "      <td>乙方乙方88888</td>\n",
       "      <td>158</td>\n",
       "      <td>195人付款</td>\n",
       "      <td>皮鞋男正品意尔康休闲商务真皮一脚蹬爸爸鞋春季新款圆头透气男鞋</td>\n",
       "      <td>http://item.taobao.com/item.htm?id=56168429783...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>{'$oid': '5aa77042be9b0338dc1faab8'}</td>\n",
       "      <td>{'上市年份季节': '2017年秋季', '货号': '7511ZE97187W', '销...</td>\n",
       "      <td>555004335522</td>\n",
       "      <td>浙江 丽水</td>\n",
       "      <td>意尔康男鞋旗舰店</td>\n",
       "      <td>279</td>\n",
       "      <td>37人付款</td>\n",
       "      <td>意尔康男鞋2018春季真皮套脚休闲鞋潮流新款舒适一脚蹬单鞋皮鞋</td>\n",
       "      <td>http://detail.tmall.com/item.htm?id=5550043355...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    _id  \\\n",
       "0  {'$oid': '5aa77041be9b0338dc1faab4'}   \n",
       "1  {'$oid': '5aa77042be9b0338dc1faab5'}   \n",
       "2  {'$oid': '5aa77042be9b0338dc1faab6'}   \n",
       "3  {'$oid': '5aa77042be9b0338dc1faab7'}   \n",
       "4  {'$oid': '5aa77042be9b0338dc1faab8'}   \n",
       "\n",
       "                                                info        itemid location  \\\n",
       "0  {'上市年份季节': '2018年春季', '货号': '8142ZR90254W', '销...  562121114807    浙江 丽水   \n",
       "1  {'上市年份季节': '2017年秋季', '货号': '7341ZL97655W', '销...  545941839337    浙江 丽水   \n",
       "2  {'鞋垫材质': '二层猪皮', '鞋跟高': '平跟(小于等于1cm)', '品牌': '...  563340844728    浙江 温州   \n",
       "3  {'货号': '883053', '鞋垫材质': '头层猪皮', '鞋跟高': '平跟(小于...  561684297836    浙江 温州   \n",
       "4  {'上市年份季节': '2017年秋季', '货号': '7511ZE97187W', '销...  555004335522    浙江 丽水   \n",
       "\n",
       "        nick  price    sales                             title  \\\n",
       "0   意尔康皮鞋旗舰店    269  1583人付款  意尔康男鞋2018春季新款英伦真皮商务休闲皮鞋青年男士皮鞋子男潮   \n",
       "1   意尔康皮鞋旗舰店    199  1607人付款    意尔康旗舰店官方店男鞋春秋真皮爸爸鞋中老年人商务休闲皮鞋子男   \n",
       "2      吸引力xl    148   248人付款  2017新款正品意尔康男鞋男士商务正装皮鞋透气真皮系带尖头办公室   \n",
       "3  乙方乙方88888    158   195人付款    皮鞋男正品意尔康休闲商务真皮一脚蹬爸爸鞋春季新款圆头透气男鞋   \n",
       "4   意尔康男鞋旗舰店    279    37人付款   意尔康男鞋2018春季真皮套脚休闲鞋潮流新款舒适一脚蹬单鞋皮鞋   \n",
       "\n",
       "                                                 url  \n",
       "0  http://detail.tmall.com/item.htm?id=5621211148...  \n",
       "1  http://detail.tmall.com/item.htm?id=5459418393...  \n",
       "2  http://item.taobao.com/item.htm?id=56334084472...  \n",
       "3  http://item.taobao.com/item.htm?id=56168429783...  \n",
       "4  http://detail.tmall.com/item.htm?id=5550043355...  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d1 = pd.read_json(r'D:\\try\\shoes1.json')\n",
    "d1\n",
    "#注意info中的双层结构并没有被解析,read_json通常只能解析一层"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "#d1.info"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    " from pandas.io.json import json_normalize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>上市年份季节</th>\n",
       "      <th>低帮鞋品名</th>\n",
       "      <th>功能</th>\n",
       "      <th>品牌</th>\n",
       "      <th>图案</th>\n",
       "      <th>场合</th>\n",
       "      <th>季节</th>\n",
       "      <th>尺码</th>\n",
       "      <th>款式</th>\n",
       "      <th>流行元素</th>\n",
       "      <th>...</th>\n",
       "      <th>闭合方式</th>\n",
       "      <th>鞋制作工艺</th>\n",
       "      <th>鞋垫材质</th>\n",
       "      <th>鞋头款式</th>\n",
       "      <th>鞋底材质</th>\n",
       "      <th>鞋跟高</th>\n",
       "      <th>鞋面内里材质</th>\n",
       "      <th>鞋面材质</th>\n",
       "      <th>颜色分类</th>\n",
       "      <th>风格</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2018年春季</td>\n",
       "      <td>商务休闲鞋</td>\n",
       "      <td>轻质</td>\n",
       "      <td>YEARCON/意尔康</td>\n",
       "      <td>纯色</td>\n",
       "      <td>日常</td>\n",
       "      <td>春秋</td>\n",
       "      <td>383940414243</td>\n",
       "      <td>商务休闲鞋</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>松紧带</td>\n",
       "      <td>注压鞋</td>\n",
       "      <td>PU</td>\n",
       "      <td>圆头</td>\n",
       "      <td>塑胶</td>\n",
       "      <td>NaN</td>\n",
       "      <td>PU</td>\n",
       "      <td>头层牛皮（除牛反绒）</td>\n",
       "      <td>黑色土黄</td>\n",
       "      <td>商务</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2017年秋季</td>\n",
       "      <td>商务休闲鞋</td>\n",
       "      <td>轻质</td>\n",
       "      <td>YEARCON/意尔康</td>\n",
       "      <td>纯色</td>\n",
       "      <td>办公室</td>\n",
       "      <td>夏季</td>\n",
       "      <td>38394041424344</td>\n",
       "      <td>商务休闲鞋</td>\n",
       "      <td>金属</td>\n",
       "      <td>...</td>\n",
       "      <td>套脚</td>\n",
       "      <td>注压鞋</td>\n",
       "      <td>二层猪皮</td>\n",
       "      <td>圆头</td>\n",
       "      <td>塑胶</td>\n",
       "      <td>低跟(1-3cm)</td>\n",
       "      <td>二层猪皮</td>\n",
       "      <td>头层牛皮（除牛反绒）</td>\n",
       "      <td>黑色棕色黑色加绒版棕色加绒版黑色镂空棕色镂空</td>\n",
       "      <td>商务</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>透气</td>\n",
       "      <td>YEARCON/意尔康</td>\n",
       "      <td>纯色</td>\n",
       "      <td>办公室</td>\n",
       "      <td>春秋</td>\n",
       "      <td>38 39 40 41 42 43 44</td>\n",
       "      <td>德比鞋（正装皮鞋）</td>\n",
       "      <td>车缝线</td>\n",
       "      <td>...</td>\n",
       "      <td>系带</td>\n",
       "      <td>胶粘鞋</td>\n",
       "      <td>二层猪皮</td>\n",
       "      <td>尖头</td>\n",
       "      <td>橡胶</td>\n",
       "      <td>平跟(小于等于1cm)</td>\n",
       "      <td>二层猪皮</td>\n",
       "      <td>头层牛皮（除牛反绒）</td>\n",
       "      <td>黑色</td>\n",
       "      <td>商务</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>透气</td>\n",
       "      <td>YEARCON/意尔康</td>\n",
       "      <td>纯色</td>\n",
       "      <td>日常</td>\n",
       "      <td>冬季</td>\n",
       "      <td>37 38 39 40 41 42 43 44</td>\n",
       "      <td>商务休闲鞋</td>\n",
       "      <td>素面</td>\n",
       "      <td>...</td>\n",
       "      <td>套脚</td>\n",
       "      <td>注压鞋</td>\n",
       "      <td>头层猪皮</td>\n",
       "      <td>圆头</td>\n",
       "      <td>橡胶</td>\n",
       "      <td>平跟(小于等于1cm)</td>\n",
       "      <td>头层猪皮</td>\n",
       "      <td>头层牛皮（除牛反绒）</td>\n",
       "      <td>黑色真皮 棕色真皮 黑色加绒真皮 棕色加绒真皮</td>\n",
       "      <td>商务</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2017年秋季</td>\n",
       "      <td>休闲皮鞋</td>\n",
       "      <td>耐磨</td>\n",
       "      <td>YEARCON/意尔康</td>\n",
       "      <td>纯色</td>\n",
       "      <td>日常</td>\n",
       "      <td>春秋</td>\n",
       "      <td>383940414243</td>\n",
       "      <td>休闲皮鞋</td>\n",
       "      <td>车缝线</td>\n",
       "      <td>...</td>\n",
       "      <td>套脚</td>\n",
       "      <td>胶粘鞋</td>\n",
       "      <td>二层猪皮</td>\n",
       "      <td>圆头</td>\n",
       "      <td>橡胶</td>\n",
       "      <td>平跟(小于等于1cm)</td>\n",
       "      <td>布</td>\n",
       "      <td>头层牛皮（除牛反绒）</td>\n",
       "      <td>黑色单鞋土黄单鞋</td>\n",
       "      <td>休闲</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 26 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    上市年份季节  低帮鞋品名  功能           品牌  图案   场合  季节                       尺码  \\\n",
       "0  2018年春季  商务休闲鞋  轻质  YEARCON/意尔康  纯色   日常  春秋             383940414243   \n",
       "1  2017年秋季  商务休闲鞋  轻质  YEARCON/意尔康  纯色  办公室  夏季           38394041424344   \n",
       "2      NaN    NaN  透气  YEARCON/意尔康  纯色  办公室  春秋     38 39 40 41 42 43 44   \n",
       "3      NaN    NaN  透气  YEARCON/意尔康  纯色   日常  冬季  37 38 39 40 41 42 43 44   \n",
       "4  2017年秋季   休闲皮鞋  耐磨  YEARCON/意尔康  纯色   日常  春秋             383940414243   \n",
       "\n",
       "          款式 流行元素 ... 闭合方式 鞋制作工艺  鞋垫材质 鞋头款式 鞋底材质          鞋跟高 鞋面内里材质  \\\n",
       "0      商务休闲鞋  NaN ...  松紧带   注压鞋    PU   圆头   塑胶          NaN     PU   \n",
       "1      商务休闲鞋   金属 ...   套脚   注压鞋  二层猪皮   圆头   塑胶    低跟(1-3cm)   二层猪皮   \n",
       "2  德比鞋（正装皮鞋）  车缝线 ...   系带   胶粘鞋  二层猪皮   尖头   橡胶  平跟(小于等于1cm)   二层猪皮   \n",
       "3      商务休闲鞋   素面 ...   套脚   注压鞋  头层猪皮   圆头   橡胶  平跟(小于等于1cm)   头层猪皮   \n",
       "4       休闲皮鞋  车缝线 ...   套脚   胶粘鞋  二层猪皮   圆头   橡胶  平跟(小于等于1cm)      布   \n",
       "\n",
       "         鞋面材质                     颜色分类  风格  \n",
       "0  头层牛皮（除牛反绒）                     黑色土黄  商务  \n",
       "1  头层牛皮（除牛反绒）   黑色棕色黑色加绒版棕色加绒版黑色镂空棕色镂空  商务  \n",
       "2  头层牛皮（除牛反绒）                       黑色  商务  \n",
       "3  头层牛皮（除牛反绒）  黑色真皮 棕色真皮 黑色加绒真皮 棕色加绒真皮  商务  \n",
       "4  头层牛皮（除牛反绒）                 黑色单鞋土黄单鞋  休闲  \n",
       "\n",
       "[5 rows x 26 columns]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c1=json_normalize(d1[\"info\"])\n",
    "c1\n",
    "#多层结构可以使用json_normalize进行进一步的解析，对于无法对应的列默认置空"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 课间题\n",
    "\n",
    "* 请自行读入js1与shoes1两份数据\n",
    "\n",
    "* 请解析下shoes1的info列"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

一键复制编辑原始数据按行查看历史

提交于 2019-12-18 12:56 . rename pandas-数据文件的读取-19.ipynb to 6_数据文件的读取.ipynb.

Python

https://gitee.com/wing2004/data_analysis.git

git@gitee.com:wing2004/data_analysis.git

wing2004

data_analysis

数据分析实践

master

王澎 / 数据分析实践

简介

发行版

贡献者

近期动态

王澎 / 数据分析实践 .gitee-modal { width: 500px !important; }

简介

发行版

贡献者

近期动态

搜索帮助

王澎 / 数据分析实践