# wajue_final **Repository Path**: bakasui/wajue_final ## Basic Information - **Project Name**: wajue_final - **Description**: No description available - **Primary Language**: Unknown - **License**: Not specified - **Default Branch**: master - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 0 - **Forks**: 0 - **Created**: 2020-07-19 - **Last Updated**: 2024-10-26 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README 数据挖掘期末

拉勾网爬取流程代码及数据加值阐述

  • ## 由于gitee限制，图表无法加载，内容以右边的链接为准，请点击观看完整内容，感谢！网页链接

搭建基础框架，爬取广州市包含文案关键词所有职位信息

In [ ]:
import requests
import time# 加载所需模块

def headers_to_dict(headers):
    headers = headers.split("\n")
    d_headers = dict()
    for h in headers:
        if h:
            k, v = h.split(":", 1)
            if k == 'cookie' and d_headers.get(k, None) is not None:
                d_headers[k] = d_headers.get(k) + "; " + v.strip()
            else:
                d_headers[k] = v.strip()
    return d_headers
In [200]:
import urllib3 # 此模块用于解决警告弹出问题

urllib3.disable_warnings()
In [95]:
home_url = 'https://www.lagou.com/jobs/list_%E6%96%87%E6%A1%88?city=%E5%B9%BF%E5%B7%9E'# 需要爬取的url
url = 'https://www.lagou.com/jobs/positionAjax.json?px=new&needAddtionalResult=false'
headers = """
accept: application/json, text/javascript, */*; q=0.01
origin: https://www.lagou.com
referer: https://www.lagou.com/jobs/list_%E6%96%87%E6%A1%88?city=%E5%B9%BF%E5%B7%9E
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36
""" # 使用网站的headers信息 伪装成客户端来爬取数据
In [96]:
headers_dict = headers_to_dict(headers)


def get_data_from_cloud(page):# 设置爬取参数
    params = {
        'first': 'false',
        'pn': page,
        'kd': '文案'
    }
    s = requests.Session()  # 创建一个session对象
    s.get(home_url, headers=headers_dict, timeout=5)  # 用session对象发出get请求，请求首页获取cookies
    cookie = s.cookies  # 为此次获取的cookies
    response = requests.post(url, data=params, headers=headers_dict, cookies=cookie, timeout=3,verify= False)
    result = response.text
    write_file(result)
In [97]:
def write_file(content):
    filename = '文案.txt'# 设置输出文件的名称和格式
    with open(filename, 'a',encoding='utf-8') as f:
        f.write(content + '\n')
In [99]:
def get_data():
    for i in range(76):# 所爬取的数据页数，分行业部分根据各行业筛选结果实际页面数调整
        page = i + 1
        get_data_from_cloud(page)
        time.sleep(10)# 设置爬取间隔时间，根据爬取量的不同调整


get_data()

分行业爬取数据

  • 图上红圈部分为我划分的传统文案相关行业 传统行业

  • 图上蓝圈部分为我划分的网新相关行业 网新行业

爬取移动互联网分类下的数据

In [143]:
home_url = 'https://www.lagou.com/jobs/list_%E6%96%87%E6%A1%88/p-city_213?hy=%E7%A7%BB%E5%8A%A8%E4%BA%92%E8%81%94%E7%BD%91'
url = 'https://www.lagou.com/jobs/positionAjax.json?px=new&needAddtionalResult=false'
headers = """
accept: application/json, text/javascript, */*; q=0.01
origin: https://www.lagou.com
referer: https://www.lagou.com/jobs/list_%E6%96%87%E6%A1%88/p-city_213?hy=%E7%A7%BB%E5%8A%A8%E4%BA%92%E8%81%94%E7%BD%91
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36
"""
In [144]:
headers_dict = headers_to_dict(headers)


def get_data_from_cloud(page):
    params = {
        'first': 'false',
        'pn': page,
        'kd': '文案'
    }
    s = requests.Session()  # 创建一个session对象
    s.get(home_url, headers=headers_dict, timeout=5)  # 用session对象发出get请求，请求首页获取cookies
    cookie = s.cookies  # 为此次获取的cookies
    response = requests.post(url, data=params, headers=headers_dict, cookies=cookie, timeout=3,verify= False)
    result = response.text
    write_file(result)
In [145]:
def write_file(content):
    filename = '移动互联网分类.txt'
    with open(filename, 'a',encoding='utf-8') as f:
        f.write(content + '\n')
In [146]:
def get_data():
    for i in range(11):
        page = i + 1
        get_data_from_cloud(page)
        time.sleep(8)


get_data()

爬取文娱内容分类下的数据

In [149]:
home_url = 'https://www.lagou.com/jobs/list_%E6%96%87%E6%A1%88/p-city_213?hy=%E6%96%87%E5%A8%B1%E4%B8%A8%E5%86%85%E5%AE%B9'
url = 'https://www.lagou.com/jobs/positionAjax.json?px=new&needAddtionalResult=false'
headers = """
accept: application/json, text/javascript, */*; q=0.01
origin: https://www.lagou.com
referer: https://www.lagou.com/jobs/list_%E6%96%87%E6%A1%88/p-city_213?hy=%E6%96%87%E5%A8%B1%E4%B8%A8%E5%86%85%E5%AE%B9
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36
"""
In [150]:
headers_dict = headers_to_dict(headers)


def get_data_from_cloud(page):
    params = {
        'first': 'false',
        'pn': page,
        'kd': '文案'
    }
    s = requests.Session()  # 创建一个session对象
    s.get(home_url, headers=headers_dict, timeout=5)  # 用session对象发出get请求，请求首页获取cookies
    cookie = s.cookies  # 获取的cookies
    response = requests.post(url, data=params, headers=headers_dict, cookies=cookie, timeout=3,verify= False)
    result = response.text
    write_file(result)
In [151]:
def write_file(content):
    filename = '文娱内容分类.txt'
    with open(filename, 'a',encoding='utf-8') as f:
        f.write(content + '\n')
In [152]:
def get_data():
    for i in range(17):
        page = i + 1
        get_data_from_cloud(page)
        time.sleep(8)


get_data()

爬取广告营销分类下的数据

In [183]:
home_url = 'https://www.lagou.com/jobs/list_%E6%96%87%E6%A1%88/p-city_213?hy=%E5%B9%BF%E5%91%8A%E8%90%A5%E9%94%80'
url = 'https://www.lagou.com/jobs/positionAjax.json?px=new&needAddtionalResult=false'
headers = """
accept: application/json, text/javascript, */*; q=0.01
origin: https://www.lagou.com
referer: https://www.lagou.com/jobs/list_%E6%96%87%E6%A1%88/p-city_213?hy=%E5%B9%BF%E5%91%8A%E8%90%A5%E9%94%80
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36
"""
In [160]:
headers_dict = headers_to_dict(headers)


def get_data_from_cloud(page):
    params = {
        'first': 'false',
        'pn': page,
        'kd': '文案'
    }
    s = requests.Session()  # 创建一个session对象
    s.get(home_url, headers=headers_dict, timeout=5)  # 用session对象发出get请求，请求首页获取cookies
    cookie = s.cookies  # 为此次获取的cookies
    response = requests.post(url, data=params, headers=headers_dict, cookies=cookie, timeout=3,verify= False)
    result = response.text
    write_file(result)
In [185]:
def write_file(content):
    filename = '广告营销分类.txt'
    with open(filename, 'a',encoding='utf-8') as f:
        f.write(content + '\n')
In [186]:
def get_data():
    for i in range(7):
        page = i + 1
        get_data_from_cloud(page)
        time.sleep(8)


get_data()

爬取电商分类下的数据

In [201]:
home_url = 'https://www.lagou.com/jobs/list_%E6%96%87%E6%A1%88/p-city_213?hy=%E7%94%B5%E5%95%86'
url = 'https://www.lagou.com/jobs/positionAjax.json?px=new&needAddtionalResult=false'
headers = """
accept: application/json, text/javascript, */*; q=0.01
origin: https://www.lagou.com
referer: https://www.lagou.com/jobs/list_%E6%96%87%E6%A1%88/p-city_213?hy=%E7%94%B5%E5%95%86
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36
"""
In [172]:
headers_dict = headers_to_dict(headers)


def get_data_from_cloud(page):
    params = {
        'first': 'false',
        'pn': page,
        'kd': '文案'
    }
    s = requests.Session()  # 创建一个session对象
    s.get(home_url, headers=headers_dict, timeout=5)  # 用session对象发出get请求，请求首页获取cookies
    cookie = s.cookies  # 为此次获取的cookies
    response = requests.post(url, data=params, headers=headers_dict, cookies=cookie, timeout=3,verify= False)
    result = response.text
    write_file(result)
In [203]:
def write_file(content):
    filename = '电商分类.txt'
    with open(filename, 'a',encoding='utf-8') as f:
        f.write(content + '\n')
In [204]:
def get_data():
    for i in range(10):
        page = i + 1
        get_data_from_cloud(page)
        time.sleep(8)


get_data()

爬取数据服务分类下的数据

In [171]:
home_url = 'https://www.lagou.com/jobs/list_%E6%96%87%E6%A1%88/p-city_213?hy=%E6%95%B0%E6%8D%AE%E6%9C%8D%E5%8A%A1'
url = 'https://www.lagou.com/jobs/positionAjax.json?px=new&needAddtionalResult=false'
headers = """
accept: application/json, text/javascript, */*; q=0.01
origin: https://www.lagou.com
referer: https://www.lagou.com/jobs/list_%E6%96%87%E6%A1%88/p-city_213?hy=%E6%95%B0%E6%8D%AE%E6%9C%8D%E5%8A%A1
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36
"""
In [184]:
headers_dict = headers_to_dict(headers)


def get_data_from_cloud(page):
    params = {
        'first': 'false',
        'pn': page,
        'kd': '文案'
    }
    s = requests.Session()  # 创建一个session对象
    s.get(home_url, headers=headers_dict, timeout=5)  # 用session对象发出get请求，请求首页获取cookies
    cookie = s.cookies  # 为此次获取的cookies
    response = requests.post(url, data=params, headers=headers_dict, cookies=cookie, timeout=3,verify= False)
    result = response.text
    write_file(result)
In [173]:
def write_file(content):
    filename = '数据服务分类.txt'
    with open(filename, 'a',encoding='utf-8') as f:
        f.write(content + '\n')
In [174]:
def get_data():
    for i in range(6):
        page = i + 1
        get_data_from_cloud(page)
        time.sleep(8)


get_data()

爬取人工智能分类下的数据

In [189]:
home_url = 'https://www.lagou.com/jobs/list_%E6%96%87%E6%A1%88/p-city_213?hy=%E4%BA%BA%E5%B7%A5%E6%99%BA%E8%83%BD'
url = 'https://www.lagou.com/jobs/positionAjax.json?px=new&needAddtionalResult=false'
headers = """
accept: application/json, text/javascript, */*; q=0.01
origin: https://www.lagou.com
referer: https://www.lagou.com/jobs/list_%E6%96%87%E6%A1%88/p-city_213?hy=%E4%BA%BA%E5%B7%A5%E6%99%BA%E8%83%BD
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36
"""
In [190]:
headers_dict = headers_to_dict(headers)


def get_data_from_cloud(page):
    params = {
        'first': 'false',
        'pn': page,
        'kd': '文案'
    }
    s = requests.Session()  # 创建一个session对象
    s.get(home_url, headers=headers_dict, timeout=5)  # 用session对象发出get请求，请求首页获取cookies
    cookie = s.cookies  # 为此次获取的cookies
    response = requests.post(url, data=params, headers=headers_dict, cookies=cookie, timeout=3,verify= False)
    result = response.text
    write_file(result)
In [191]:
def write_file(content):
    filename = '人工智能分类.txt'
    with open(filename, 'a',encoding='utf-8') as f:
        f.write(content + '\n')
In [192]:
def get_data():
    for i in range(1):
        page = i + 1
        get_data_from_cloud(page)
        time.sleep(8)


get_data()

爬取软件开发分类下的数据

In [195]:
home_url = 'https://www.lagou.com/jobs/list_%E6%96%87%E6%A1%88/p-city_213?hy=%E8%BD%AF%E4%BB%B6%E5%BC%80%E5%8F%91'
url = 'https://www.lagou.com/jobs/positionAjax.json?px=new&needAddtionalResult=false'
headers = """
accept: application/json, text/javascript, */*; q=0.01
origin: https://www.lagou.com
referer: https://www.lagou.com/jobs/list_%E6%96%87%E6%A1%88/p-city_213?hy=%E8%BD%AF%E4%BB%B6%E5%BC%80%E5%8F%91
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36
"""
In [196]:
headers_dict = headers_to_dict(headers)


def get_data_from_cloud(page):
    params = {
        'first': 'false',
        'pn': page,
        'kd': '文案'
    }
    s = requests.Session()  # 创建一个session对象
    s.get(home_url, headers=headers_dict, timeout=5)  # 用session对象发出get请求，请求首页获取cookies
    cookie = s.cookies  # 为此次获取的cookies
    response = requests.post(url, data=params, headers=headers_dict, cookies=cookie, timeout=3,verify= False)
    result = response.text
    write_file(result)
In [197]:
def write_file(content):
    filename = '软件开发分类.txt'
    with open(filename, 'a',encoding='utf-8') as f:
        f.write(content + '\n')
In [198]:
def get_data():
    for i in range(4):
        page = i + 1
        get_data_from_cloud(page)
        time.sleep(8)


get_data()

爬取企业服务分类下的数据

In [159]:
home_url = 'https://www.lagou.com/jobs/list_%E6%96%87%E6%A1%88/p-city_213?hy=%E4%BC%81%E4%B8%9A%E6%9C%8D%E5%8A%A1'
url = 'https://www.lagou.com/jobs/positionAjax.json?px=new&needAddtionalResult=false'
headers = """
accept: application/json, text/javascript, */*; q=0.01
origin: https://www.lagou.com
referer: https://www.lagou.com/jobs/list_%E6%96%87%E6%A1%88/p-city_213?hy=%E4%BC%81%E4%B8%9A%E6%9C%8D%E5%8A%A1
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36
"""
In [202]:
headers_dict = headers_to_dict(headers)


def get_data_from_cloud(page):
    params = {
        'first': 'false',
        'pn': page,
        'kd': '文案'
    }
    s = requests.Session()  # 创建一个session对象
    s.get(home_url, headers=headers_dict, timeout=5)  # 用session对象发出get请求，请求首页获取cookies
    cookie = s.cookies  # 为此次获取的cookies
    response = requests.post(url, data=params, headers=headers_dict, cookies=cookie, timeout=3,verify= False)
    result = response.text
    write_file(result)
In [161]:
def write_file(content):
    filename = '企业服务分类.txt'
    with open(filename, 'a',encoding='utf-8') as f:
        f.write(content + '\n')
In [162]:
def get_data():
    for i in range(14):
        page = i + 1
        get_data_from_cloud(page)
        time.sleep(8)


get_data()

用图表深入分析爬取的数据

1.对整体数据进行分析

In [2]:
%matplotlib inline
import numpy as np # 载入所需模块
from pyecharts import options as opts
from pyecharts.charts import Bar
from pyecharts.charts import Pie
from wordcloud import WordCloud
import matplotlib.pyplot as plt
import json
import pandas as pd
In [3]:
def get_data():
    with open('文案.txt',encoding='utf-8') as f:# 打开之前导出的数据文件
        data = []
        for line in f.readlines():
            result = json.loads(line)
            result_list = result['content']['positionResult']['result']
            for item in result_list:# 构建字典
                dict = {
                   'district': item['district'],
                    'industryField': item['industryField'],
                    'education': item['education'],
                    'workYear': item['workYear'],
                    'salary': item['salary'],
                    'firstType': item['firstType'],
                    'secondType': item['secondType'],
                    'thirdType': item['thirdType'],
                    'skillLables': item['skillLables']
                }
                data.append(dict)
        return data
In [5]:
data = get_data()# 调用数据
data = pd.DataFrame(data)
data.head(500)# 显现其中的500条数据以展示分类效果
Out[5]:
district industryField education workYear salary firstType secondType thirdType skillLables
0 海淀区 移动互联网 本科 1-3年 8k-15k 运营|编辑|客服类 编辑 文案策划 [文案]
1 浦东新区 广告营销 本科 3-5年 10k-16k 运营|编辑|客服类 编辑 文案策划 [文案, 策划]
2 锦江区 企业服务 本科 不限 8k-15k 运营|编辑|客服类 运营 其他运营 [新媒体运营, 网络运营]
3 南山区 移动互联网 本科 1-3年 8k-16k 运营|编辑|客服类 编辑 文案策划 [策划, 文案]
4 朝阳区 文娱丨内容 大专 1-3年 6k-12k 运营|编辑|客服类 编辑 文案策划 [文案, 策划, 软文]
... ... ... ... ... ... ... ... ... ...
495 南沙区 汽车丨出行 本科 3-5年 15k-20k 运营|编辑|客服类 运营 内容运营 [运营, PGC, 策略运营, 用户增长]
496 鼓楼区 软件开发、企业服务 不限 1-3年 4k-8k 运营|编辑|客服类 编辑 文案策划 [策划, 文案, 编辑, 软文]
497 朝阳区 移动互联网,文娱丨内容 大专 3-5年 10k-15k 运营|编辑|客服类 编辑 文案策划 []
498 拱墅区 广告营销,文娱丨内容 不限 不限 6k-10k 运营|编辑|客服类 编辑 文案策划 [内容, 软文, 视频编辑, 广告]
499 朝阳区 企业服务 本科 1-3年 4k-7k 市场|商务类 品牌|广告 广告创意 [策划, 公关（PR）, 文案]

500 rows × 9 columns

In [250]:
citys_value_counts = data['district'].value_counts() # 选取区域标签进行整合
top = 20 # 选取前20的地区
citys = list(citys_value_counts.head(top).index)
city_counts = list(citys_value_counts.head(top))

bar = (
    Bar()
        .add_xaxis(citys)
        .add_yaxis("", city_counts)
)
bar.render_notebook()
Out[250]:

从图表可以看出 广州天河区对文案相关职业的需求最高

In [210]:
industrys = list(data['industryField']) # 选取行业标签进行整合
industry_list = [i for item in industrys for i in item.split(',')]

industry_series = pd.Series(data=industry_list)
industry_value_counts = industry_series.value_counts()

industrys = list(industry_value_counts.head(top).index)
industry_counts = list(industry_value_counts.head(top))

pie = (  # 调用pyechart的饼图模块
    Pie()
        .add("", [list(z) for z in zip(industrys, industry_counts)])
        .set_global_opts(title_opts=opts.TitleOpts(title=""))
        .set_global_opts(legend_opts=opts.LegendOpts(is_show=False))
)
pie.render_notebook()
Out[210]:

从饼图中可以看出 绝大多数的文案相关职位需求集中在移动互联网、电商、广告营销等传统行业

In [211]:
eduction_value_counts = data['education'].value_counts() # 选取学历标签进行整合

eduction = list(eduction_value_counts.index)
eduction_counts = list(eduction_value_counts)

pie = (
    Pie()
        .add("", [list(z) for z in zip(eduction, eduction_counts)])
        .set_global_opts(title_opts=opts.TitleOpts(title=""))
        .set_global_opts(legend_opts=opts.LegendOpts(is_show=False))
)
pie.render_notebook()
Out[211]:

从饼图中可以看出 绝大多数的文案相关职位对学历的要求都是达到本科学历

In [212]:
work_year_value_counts = data['workYear'].value_counts() # 工作经验作为标签进行整合
work_year = list(work_year_value_counts.index)
work_year_counts = list(work_year_value_counts)

bar = (
    Bar()
        .add_xaxis(work_year)
        .add_yaxis("", work_year_counts)
)
bar.render_notebook()
Out[212]:

从图表出可以看出用人单位对工作经验1-3年的员工的需求量最大 令人惊奇的是应届毕业生居然也占据了一席之地

In [213]:
salary_value_counts = data['salary'].value_counts() # 选取工资标签进行整合
salary = list(salary_value_counts.head(top).index)
salary_counts = list(salary_value_counts.head(top))

bar = (
    Bar()
        .add_xaxis(salary)
        .add_yaxis("", salary_counts)
        .set_global_opts(xaxis_opts=opts.AxisOpts(name_rotate=0, name="薪资", axislabel_opts={"rotate": 45}))
)
bar.render_notebook()
Out[213]:

图表显示 各薪资阶段的职位分布都较为平均 结合行业进行分析会更有参考价值

2.将传统行业与网新相关行业文案相关工作薪资进行对比

In [243]:
import numpy as np # 载入所需模块
from pyecharts import options as opts
from pyecharts.charts import Bar
from pyecharts.charts import Pie
from wordcloud import WordCloud
import matplotlib.pyplot as plt
import json
import pandas as pd

移动互联网分类薪资分布图

In [216]:
def get_data():
    with open('移动互联网分类.txt',encoding='utf-8') as f: # 选取要分析的行业数据
        data = []
        for line in f.readlines():
            result = json.loads(line)
            result_list = result['content']['positionResult']['result']
            for item in result_list:
                dict = {
                    'city': item['city'],
                   'district': item['district'],
                    'industryField': item['industryField'],
                    'education': item['education'],
                    'workYear': item['workYear'],
                    'salary': item['salary'],
                    'firstType': item['firstType'],
                    'secondType': item['secondType'],
                    'thirdType': item['thirdType'],
                    'skillLables': item['skillLables']
                }
                data.append(dict)
        return data
In [217]:
data = get_data()
data = pd.DataFrame(data)
In [218]:
salary_value_counts = data['salary'].value_counts()
salary = list(salary_value_counts.head(top).index)
salary_counts = list(salary_value_counts.head(top))

bar = (
    Bar()
        .add_xaxis(salary)
        .add_yaxis("", salary_counts)
        .set_global_opts(xaxis_opts=opts.AxisOpts(name_rotate=0, name="薪资", axislabel_opts={"rotate": 45}))
)
bar.render_notebook()
Out[218]:

文娱内容分类薪资分布图

In [220]:
def get_data():
    with open('文娱内容分类.txt',encoding='utf-8') as f:
        data = []
        for line in f.readlines():
            result = json.loads(line)
            result_list = result['content']['positionResult']['result']
            for item in result_list:
                dict = {
                    'city': item['city'],
                   'district': item['district'],
                    'industryField': item['industryField'],
                    'education': item['education'],
                    'workYear': item['workYear'],
                    'salary': item['salary'],
                    'firstType': item['firstType'],
                    'secondType': item['secondType'],
                    'thirdType': item['thirdType'],
                    'skillLables': item['skillLables']
                }
                data.append(dict)
        return data
In [221]:
data = get_data()
data = pd.DataFrame(data)
In [222]:
salary_value_counts = data['salary'].value_counts()
salary = list(salary_value_counts.head(top).index)
salary_counts = list(salary_value_counts.head(top))

bar = (
    Bar()
        .add_xaxis(salary)
        .add_yaxis("", salary_counts)
        .set_global_opts(xaxis_opts=opts.AxisOpts(name_rotate=0, name="薪资", axislabel_opts={"rotate": 45}))
)
bar.render_notebook()
Out[222]:

广告营销分类薪资分布图

In [224]:
def get_data():
    with open('广告营销分类.txt',encoding='utf-8') as f:
        data = []
        for line in f.readlines():
            result = json.loads(line)
            result_list = result['content']['positionResult']['result']
            for item in result_list:
                dict = {
                    'city': item['city'],
                   'district': item['district'],
                    'industryField': item['industryField'],
                    'education': item['education'],
                    'workYear': item['workYear'],
                    'salary': item['salary'],
                    'firstType': item['firstType'],
                    'secondType': item['secondType'],
                    'thirdType': item['thirdType'],
                    'skillLables': item['skillLables']
                }
                data.append(dict)
        return data
In [225]:
data = get_data()
data = pd.DataFrame(data)
In [226]:
salary_value_counts = data['salary'].value_counts()
salary = list(salary_value_counts.head(top).index)
salary_counts = list(salary_value_counts.head(top))

bar = (
    Bar()
        .add_xaxis(salary)
        .add_yaxis("", salary_counts)
        .set_global_opts(xaxis_opts=opts.AxisOpts(name_rotate=0, name="薪资", axislabel_opts={"rotate": 45}))
)
bar.render_notebook()
Out[226]:

电商分类薪资分布图

In [228]:
def get_data():
    with open('电商分类.txt',encoding='utf-8') as f:
        data = []
        for line in f.readlines():
            result = json.loads(line)
            result_list = result['content']['positionResult']['result']
            for item in result_list:
                dict = {
                    'city': item['city'],
                   'district': item['district'],
                    'industryField': item['industryField'],
                    'education': item['education'],
                    'workYear': item['workYear'],
                    'salary': item['salary'],
                    'firstType': item['firstType'],
                    'secondType': item['secondType'],
                    'thirdType': item['thirdType'],
                    'skillLables': item['skillLables']
                }
                data.append(dict)
        return data
In [229]:
data = get_data()
data = pd.DataFrame(data)
In [230]:
salary_value_counts = data['salary'].value_counts()
salary = list(salary_value_counts.head(top).index)
salary_counts = list(salary_value_counts.head(top))

bar = (
    Bar()
        .add_xaxis(salary)
        .add_yaxis("", salary_counts)
        .set_global_opts(xaxis_opts=opts.AxisOpts(name_rotate=0, name="薪资", axislabel_opts={"rotate": 45}))
)
bar.render_notebook()
Out[230]:

企业服务分类薪资分布图

In [232]:
def get_data():
    with open('企业服务分类.txt',encoding='utf-8') as f:
        data = []
        for line in f.readlines():
            result = json.loads(line)
            result_list = result['content']['positionResult']['result']
            for item in result_list:
                dict = {
                    'city': item['city'],
                   'district': item['district'],
                    'industryField': item['industryField'],
                    'education': item['education'],
                    'workYear': item['workYear'],
                    'salary': item['salary'],
                    'firstType': item['firstType'],
                    'secondType': item['secondType'],
                    'thirdType': item['thirdType'],
                    'skillLables': item['skillLables']
                }
                data.append(dict)
        return data
In [233]:
data = get_data()
data = pd.DataFrame(data)
In [234]:
salary_value_counts = data['salary'].value_counts()
salary = list(salary_value_counts.head(top).index)
salary_counts = list(salary_value_counts.head(top))

bar = (
    Bar()
        .add_xaxis(salary)
        .add_yaxis("", salary_counts)
        .set_global_opts(xaxis_opts=opts.AxisOpts(name_rotate=0, name="薪资", axislabel_opts={"rotate": 45}))
)
bar.render_notebook()
Out[234]:

数据服务分类薪资分布图

In [236]:
def get_data():
    with open('数据服务分类.txt',encoding='utf-8') as f:
        data = []
        for line in f.readlines():
            result = json.loads(line)
            result_list = result['content']['positionResult']['result']
            for item in result_list:
                dict = {
                    'city': item['city'],
                   'district': item['district'],
                    'industryField': item['industryField'],
                    'education': item['education'],
                    'workYear': item['workYear'],
                    'salary': item['salary'],
                    'firstType': item['firstType'],
                    'secondType': item['secondType'],
                    'thirdType': item['thirdType'],
                    'skillLables': item['skillLables']
                }
                data.append(dict)
        return data
In [237]:
data = get_data()
data = pd.DataFrame(data)
In [238]:
salary_value_counts = data['salary'].value_counts()
salary = list(salary_value_counts.head(top).index)
salary_counts = list(salary_value_counts.head(top))

bar = (
    Bar()
        .add_xaxis(salary)
        .add_yaxis("", salary_counts)
        .set_global_opts(xaxis_opts=opts.AxisOpts(name_rotate=0, name="薪资", axislabel_opts={"rotate": 45}))
)
bar.render_notebook()
Out[238]:

人工智能分类薪资分布图

In [240]:
def get_data():
    with open('人工智能分类.txt',encoding='utf-8') as f:
        data = []
        for line in f.readlines():
            result = json.loads(line)
            result_list = result['content']['positionResult']['result']
            for item in result_list:
                dict = {
                    'city': item['city'],
                   'district': item['district'],
                    'industryField': item['industryField'],
                    'education': item['education'],
                    'workYear': item['workYear'],
                    'salary': item['salary'],
                    'firstType': item['firstType'],
                    'secondType': item['secondType'],
                    'thirdType': item['thirdType'],
                    'skillLables': item['skillLables']
                }
                data.append(dict)
        return data
In [241]:
data = get_data()
data = pd.DataFrame(data)
In [242]:
salary_value_counts = data['salary'].value_counts()
salary = list(salary_value_counts.head(top).index)
salary_counts = list(salary_value_counts.head(top))

bar = (
    Bar()
        .add_xaxis(salary)
        .add_yaxis("", salary_counts)
        .set_global_opts(xaxis_opts=opts.AxisOpts(name_rotate=0, name="薪资", axislabel_opts={"rotate": 45}))
)
bar.render_notebook()
Out[242]:

软件开发分类薪资分布图

In [244]:
def get_data():
    with open('软件开发分类.txt',encoding='utf-8') as f:
        data = []
        for line in f.readlines():
            result = json.loads(line)
            result_list = result['content']['positionResult']['result']
            for item in result_list:
                dict = {
                    'city': item['city'],
                   'district': item['district'],
                    'industryField': item['industryField'],
                    'education': item['education'],
                    'workYear': item['workYear'],
                    'salary': item['salary'],
                    'firstType': item['firstType'],
                    'secondType': item['secondType'],
                    'thirdType': item['thirdType'],
                    'skillLables': item['skillLables']
                }
                data.append(dict)
        return data
In [245]:
data = get_data()
data = pd.DataFrame(data)
In [246]:
salary_value_counts = data['salary'].value_counts()
salary = list(salary_value_counts.head(top).index)
salary_counts = list(salary_value_counts.head(top))

bar = (
    Bar()
        .add_xaxis(salary)
        .add_yaxis("", salary_counts)
        .set_global_opts(xaxis_opts=opts.AxisOpts(name_rotate=0, name="薪资", axislabel_opts={"rotate": 45}))
)
bar.render_notebook()
Out[246]:

分析上列图表可以看出：

  • 传统行业的文案工作薪资范围大概稳定在6k-10k 网新相关行业则没有稳定范围 值得关注的是人工智能与企业服务这两个行业 薪资水平均属于比较高的范围 撰写产品文档也许是文案工作发展的新方向

相关链接及尝试过程中参考文章分享

1.拉勾网首页

2.本文档基础框架参考github仓库

3.python请求https出现443错误的解决办法

4.编码方式gbk出现错误解决方法

5.阻止安全警告弹出方法

6.requests语法参考文档

7.尝试过的scrapy框架爬取文章

8.无法成功导出xlsx的原因解释

9.如何将xlsx及csv文件导入jupyter

10.尝试过的将txt文件转换为xlsx格式文章

11.尝试过的scrapy爬取智联招聘网方法

12.尝试过的scrapy导出数据为空解决方法

13.尝试过的values中的list长度不一致报错原因

14.尝试过的scrapy爬取直聘网方法

15.尝试过的No connection adapters报错解决方法

16.尝试过的MySQL远程连接的时候报错1130解决方法

17.尝试过的scrapy爬取58同城招聘信息文章

18.尝试过的scrapy爬取三个招聘网站的gitee仓库

19.尝试过的scrapy爬取腾讯招聘信息的文章（腾讯招聘已限制爬取）

20.scrapy框架内item文件的参数介绍

感谢你的浏览 辛苦了(๑╹◡╹)ﾉ"""

In [ ]: