Watch 1 Star 0 Fork 0

WeFamily / python-noaa-data-use-seleniumPython

Join us
Explore and code with more than 2 million developers,Free private repositories !:)
Sign up
This repository doesn't specify license. Without author's permission, this code is only for learning and cannot be used for other purposes.
使用Python的selenium爬取noaa上面的指定数据 spread retract

Clone or download
Cancel
Notice: Creating folder will generate an empty file .keep, because not support in Git
Loading...
README.md

数据爬取

变量介绍

# 网络较慢时,设置大一些,中间可以有些停顿等待
lower_network = 3
# type 是对应的检测类型
type = 'MSL'
# units 单位(0表示 Feet, 1表示 Meters)
units = 0
# epoch 时间(0表示 Present, 1表示 Superseded)
epoch = 0
# main_url 是主站点,后面会用到 url 的拼接
main_url = 'https://tidesandcurrents.noaa.gov/datums.html?datum=' + type + '&units=' + str(units) + '&epoch=' + str(epoch)
# path 是要本地配置的chromedriver的路径,网络搜索chromedriver
# 如果非Windows系统,注意用户权限
path = '/usr/bin/chromedriver'

安装chromedriver,参考:https://npm.taobao.org/mirrors/chromedriver

安装环境依赖

# 安装Python和pip,建议3.x版本(使用的3.6版本)
# 安装依赖 selenium
# 如果非Windows系统,注意用户权限;安装了多个版本的Python,注意正确使用命令行
pip install selenium

安装多版本Python,参考:https://www.jianshu.com/p/810f3be23431

工程运行

# cmd到工程目录下,执行
python noaa.py
# 或者 - 根据系统安装配置执行
python3 noaa.py
# 如果不报错,会打开chrome浏览器,不要关闭这个弹窗,数据会开始爬取,控制台中可以看到爬取的简要内容
# 会在工程目录下生成MSL.txt文件,每行为一条记录,由id;name;state;value;url组成,使用excel打开,选择分号';'分列

运行结果示例

# 前15行的数据
 1611347;PORT ALLEN, HANAPEPE BAY, KAUAI ISLAND;HI;0.00;https://tidesandcurrents.noaa.gov/datums.html?datum=MSL&units=0&epoch=0&id=1611347&name=PORT%20ALLEN,%20HANAPEPE%20BAY,%20KAUAI%20ISLAND&state=HI
 1611400;Nawiliwili;HI;0.00;https://tidesandcurrents.noaa.gov/datums.html?datum=MSL&units=0&epoch=0&id=1611400&name=Nawiliwili&state=HI
 1612340;Honolulu;HI;0.00;https://tidesandcurrents.noaa.gov/datums.html?datum=MSL&units=0&epoch=0&id=1612340&name=Honolulu&state=HI
 1612366;FORT KAMEHAMEHA, BISHOP POINT, PEARL HBR;HI;0.00;https://tidesandcurrents.noaa.gov/datums.html?datum=MSL&units=0&epoch=0&id=1612366&name=FORT%20KAMEHAMEHA,%20BISHOP%20POINT,%20PEARL%20HBR&state=HI
 1612404;FORD ISLAND, FERRY DOCK, PEARL HARBOR;HI;0.00;https://tidesandcurrents.noaa.gov/datums.html?datum=MSL&units=0&epoch=0&id=1612404&name=FORD%20ISLAND,%20FERRY%20DOCK,%20PEARL%20HARBOR&state=HI
 1612480;Mokuoloe;HI;0.00;https://tidesandcurrents.noaa.gov/datums.html?datum=MSL&units=0&epoch=0&id=1612480&name=Mokuoloe&state=HI
 1612668;HALEIWA, WAIALUA BAY, OAHU IS;HI;0.00;https://tidesandcurrents.noaa.gov/datums.html?datum=MSL&units=0&epoch=0&id=1612668&name=HALEIWA,%20WAIALUA%20BAY,%20OAHU%20IS&state=HI
 1612702;LAIEMALOO, OAHU ISLAND;HI;0.00;https://tidesandcurrents.noaa.gov/datums.html?datum=MSL&units=0&epoch=0&id=1612702&name=LAIEMALOO,%20OAHU%20ISLAND&state=HI
 1613198;Kaunakakai Harbor;HI;0.00;https://tidesandcurrents.noaa.gov/datums.html?datum=MSL&units=0&epoch=0&id=1613198&name=Kaunakakai%20Harbor&state=HI
 1614465;KAUMALAPAU HARBOR, LANAI ISLAND;HI;0.00;https://tidesandcurrents.noaa.gov/datums.html?datum=MSL&units=0&epoch=0&id=1614465&name=KAUMALAPAU%20HARBOR,%20LANAI%20ISLAND&state=HI
 1615680;Kahului, Kahului Harbor;HI;0.00;https://tidesandcurrents.noaa.gov/datums.html?datum=MSL&units=0&epoch=0&id=1615680&name=Kahului,%20Kahului%20Harbor&state=HI
 1617433;Kawaihae;HI;0.00;https://tidesandcurrents.noaa.gov/datums.html?datum=MSL&units=0&epoch=0&id=1617433&name=Kawaihae&state=HI
 1617760;Hilo, Hilo Bay, Kuhio Bay;HI;0.00;https://tidesandcurrents.noaa.gov/datums.html?datum=MSL&units=0&epoch=0&id=1617760&name=Hilo,%20Hilo%20Bay,%20Kuhio%20Bay&state=HI
 1619000;Johnston Atoll;United States of America;0.00;https://tidesandcurrents.noaa.gov/datums.html?datum=MSL&units=0&epoch=0&id=1619000&name=Johnston%20Atoll&state=United%20States%20of%20America
 1619910;Sand Island, Midway Islands;;0.00;https://tidesandcurrents.noaa.gov/datums.html?datum=MSL&units=0&epoch=0&id=1619910&name=Sand%20Island,%20Midway%20Islands&state=

Comments ( 0 )

Sign in for post a comment

Python
1
https://gitee.com/wefamily/python-noaa-data-use-selenium.git
git@gitee.com:wefamily/python-noaa-data-use-selenium.git
wefamily
python-noaa-data-use-selenium
python-noaa-data-use-selenium
master

Help Search