1 Star 0 Fork 1

WeFamily / python-noaa-data-use-selenium

Create your Gitee Account
Explore and code with more than 12 million developers,Free private repositories !:)
Sign up
This repository doesn't specify license. Please pay attention to the specific project description and its upstream code dependency when using it.
Clone or Download
README.md 4.22 KB
Copy Edit Raw Blame History
B.K. authored 2019-11-19 13:34 . 更新说明

数据爬取

变量介绍

# 网络较慢时,设置大一些,中间可以有些停顿等待
lower_network = 3
# type 是对应的检测类型
type = 'MSL'
# units 单位(0表示 Feet, 1表示 Meters)
units = 0
# epoch 时间(0表示 Present, 1表示 Superseded)
epoch = 0
# main_url 是主站点,后面会用到 url 的拼接
main_url = 'https://tidesandcurrents.noaa.gov/datums.html?datum=' + type + '&units=' + str(units) + '&epoch=' + str(epoch)
# path 是要本地配置的chromedriver的路径,网络搜索chromedriver
# 如果非Windows系统,注意用户权限
path = '/usr/bin/chromedriver'

安装chromedriver,参考:https://npm.taobao.org/mirrors/chromedriver

安装环境依赖

# 安装Python和pip,建议3.x版本(使用的3.6版本)
# 安装依赖 selenium
# 如果非Windows系统,注意用户权限;安装了多个版本的Python,注意正确使用命令行
pip install selenium

安装多版本Python,参考:https://www.jianshu.com/p/810f3be23431

工程运行

# cmd到工程目录下,执行
python noaa.py
# 或者 - 根据系统安装配置执行
python3 noaa.py
# 如果不报错,会打开chrome浏览器,不要关闭这个弹窗,数据会开始爬取,控制台中可以看到爬取的简要内容
# 会在工程目录下生成MSL.txt文件,每行为一条记录,由id;name;state;value;url组成,使用excel打开,选择分号';'分列

运行结果示例

# 前15行的数据
 1611347;PORT ALLEN, HANAPEPE BAY, KAUAI ISLAND;HI;0.00;https://tidesandcurrents.noaa.gov/datums.html?datum=MSL&units=0&epoch=0&id=1611347&name=PORT%20ALLEN,%20HANAPEPE%20BAY,%20KAUAI%20ISLAND&state=HI
 1611400;Nawiliwili;HI;0.00;https://tidesandcurrents.noaa.gov/datums.html?datum=MSL&units=0&epoch=0&id=1611400&name=Nawiliwili&state=HI
 1612340;Honolulu;HI;0.00;https://tidesandcurrents.noaa.gov/datums.html?datum=MSL&units=0&epoch=0&id=1612340&name=Honolulu&state=HI
 1612366;FORT KAMEHAMEHA, BISHOP POINT, PEARL HBR;HI;0.00;https://tidesandcurrents.noaa.gov/datums.html?datum=MSL&units=0&epoch=0&id=1612366&name=FORT%20KAMEHAMEHA,%20BISHOP%20POINT,%20PEARL%20HBR&state=HI
 1612404;FORD ISLAND, FERRY DOCK, PEARL HARBOR;HI;0.00;https://tidesandcurrents.noaa.gov/datums.html?datum=MSL&units=0&epoch=0&id=1612404&name=FORD%20ISLAND,%20FERRY%20DOCK,%20PEARL%20HARBOR&state=HI
 1612480;Mokuoloe;HI;0.00;https://tidesandcurrents.noaa.gov/datums.html?datum=MSL&units=0&epoch=0&id=1612480&name=Mokuoloe&state=HI
 1612668;HALEIWA, WAIALUA BAY, OAHU IS;HI;0.00;https://tidesandcurrents.noaa.gov/datums.html?datum=MSL&units=0&epoch=0&id=1612668&name=HALEIWA,%20WAIALUA%20BAY,%20OAHU%20IS&state=HI
 1612702;LAIEMALOO, OAHU ISLAND;HI;0.00;https://tidesandcurrents.noaa.gov/datums.html?datum=MSL&units=0&epoch=0&id=1612702&name=LAIEMALOO,%20OAHU%20ISLAND&state=HI
 1613198;Kaunakakai Harbor;HI;0.00;https://tidesandcurrents.noaa.gov/datums.html?datum=MSL&units=0&epoch=0&id=1613198&name=Kaunakakai%20Harbor&state=HI
 1614465;KAUMALAPAU HARBOR, LANAI ISLAND;HI;0.00;https://tidesandcurrents.noaa.gov/datums.html?datum=MSL&units=0&epoch=0&id=1614465&name=KAUMALAPAU%20HARBOR,%20LANAI%20ISLAND&state=HI
 1615680;Kahului, Kahului Harbor;HI;0.00;https://tidesandcurrents.noaa.gov/datums.html?datum=MSL&units=0&epoch=0&id=1615680&name=Kahului,%20Kahului%20Harbor&state=HI
 1617433;Kawaihae;HI;0.00;https://tidesandcurrents.noaa.gov/datums.html?datum=MSL&units=0&epoch=0&id=1617433&name=Kawaihae&state=HI
 1617760;Hilo, Hilo Bay, Kuhio Bay;HI;0.00;https://tidesandcurrents.noaa.gov/datums.html?datum=MSL&units=0&epoch=0&id=1617760&name=Hilo,%20Hilo%20Bay,%20Kuhio%20Bay&state=HI
 1619000;Johnston Atoll;United States of America;0.00;https://tidesandcurrents.noaa.gov/datums.html?datum=MSL&units=0&epoch=0&id=1619000&name=Johnston%20Atoll&state=United%20States%20of%20America
 1619910;Sand Island, Midway Islands;;0.00;https://tidesandcurrents.noaa.gov/datums.html?datum=MSL&units=0&epoch=0&id=1619910&name=Sand%20Island,%20Midway%20Islands&state=

Python
1
https://gitee.com/wefamily/python-noaa-data-use-selenium.git
git@gitee.com:wefamily/python-noaa-data-use-selenium.git
wefamily
python-noaa-data-use-selenium
python-noaa-data-use-selenium
master

Search