Sign in
Sign up
Explore
Enterprise
Education
Search
Help
Terms of use
About Us
Explore
Enterprise
Education
Gitee Premium
Gitee AI
AI teammates
Sign in
Sign up
Fetch the repository succeeded.
description of repo status
Donate
Please sign in before you donate.
Cancel
Sign in
Scan WeChat QR to Pay
Cancel
Complete
Prompt
Switch to Alipay.
OK
Cancel
Watch
Unwatch
Watching
Releases Only
Ignoring
10
Star
2
Fork
66
src-openEuler
/
curl
Closed
Code
Issues
2
Pull Requests
1
Wiki
Insights
Pipelines
Service
JavaDoc
PHPDoc
Quality Analysis
Jenkins for Gitee
Tencent CloudBase
Tencent Cloud Serverless
悬镜安全
Aliyun SAE
Codeblitz
SBOM
DevLens
Don’t show this again
Update failed. Please try again later!
Remove this flag
Content Risk Flag
This task is identified by
as the content contains sensitive information such as code security bugs, privacy leaks, etc., so it is only accessible to contributors of this repository.
curl命令向hadoop3.2.1 webhdfs put文件失败
Top
Backlog
#I6TQ8W
Task
lovelijunyi
Opened this issue
2023-04-07 18:02
【问题现象】: 1、通过openEuler 20.03系统主机,我们的大数据管理平台使用curl命令向hadoop3.2.1 webhdfs put文件失败: curl -sS -L -w '%{http_code}' -X PUT --data-binary @/usr/lib/hivemall/hivemall-all-0.6.0-incubating.jar -H 'Content-Type: application/octet-stream' --negotiate -u : 'http://neoap031:50070/webhdfs/v1/apps/hivemall/hivemall-all-0.6.0-incubating.jar?op=CREATE&overwrite=True&permission=444' curl http返回码为 307 报错: curl: (55) Send failure: Connection reset by peer 2、针对同样的webhdfs服务地址,通过centos 7.x put文件成功。HTTP 返回码为201 。 【问题分析】:现象上看,我们初步分析,怀疑是openEuler 20.03系统中的curl不支持 http Temporary Redirect ,curl 直接返回了307,并没有从http 307返回的location重定向地址进行下一步操作。 通过curl命令向webhdfs put文件,分为以下几个步骤: 1、通过客户端通过curl向nameNode:50075端口提交写请求,形如: curl -sS -L -w '%{http_code}' -X PUT --data-binary @/usr/lib/tez/tez.tar.gz -H 'Content-Type: application/octet-stream' --negotiate -u : 'http://neoap032:50070/webhdfs/v1/ntk/apps/3.0/tez/tez.tar.gz?op=CREATE&overwritrue&permission=444&noredirect=false' 2、NameNode通过http 307 Temporary Redirect 在response head的location中返回实际需要写入的dataNode url地址,形如: {"Location":"http://neoap033:1022/webhdfs/v1/ntk/apps/3.0/tez/tez.tar.gz?op=CREATE&delegation=HQAEaGRmcwRoZGZzAIoBh0aBQE6KAYdqjcROj4oXFHguAEscAZRLJUN0kZPXeJpJXZSPEldFQkhERlMgZGVsZWdhdGlvbhMxNzIuMzIuMTUzLjE1MDo4MDIw&namenoderpcaddress=newland&createflag=&createparent=true&overwrite=true&permission=444"} 3、curl 继续向重定向的Location地址发起写请求,在这个步骤 centos7.x是oK的,会根据location返回的重定向地址继续进行写操作,最后返回Http 201;而openEuler 20.03的curl直接返回了307,没有进行这步操作。 4、如果在centos7.4curl请求中设置不允许redirect,noredirect=true,可以模拟出来同样的报错和返回值
【问题现象】: 1、通过openEuler 20.03系统主机,我们的大数据管理平台使用curl命令向hadoop3.2.1 webhdfs put文件失败: curl -sS -L -w '%{http_code}' -X PUT --data-binary @/usr/lib/hivemall/hivemall-all-0.6.0-incubating.jar -H 'Content-Type: application/octet-stream' --negotiate -u : 'http://neoap031:50070/webhdfs/v1/apps/hivemall/hivemall-all-0.6.0-incubating.jar?op=CREATE&overwrite=True&permission=444' curl http返回码为 307 报错: curl: (55) Send failure: Connection reset by peer 2、针对同样的webhdfs服务地址,通过centos 7.x put文件成功。HTTP 返回码为201 。 【问题分析】:现象上看,我们初步分析,怀疑是openEuler 20.03系统中的curl不支持 http Temporary Redirect ,curl 直接返回了307,并没有从http 307返回的location重定向地址进行下一步操作。 通过curl命令向webhdfs put文件,分为以下几个步骤: 1、通过客户端通过curl向nameNode:50075端口提交写请求,形如: curl -sS -L -w '%{http_code}' -X PUT --data-binary @/usr/lib/tez/tez.tar.gz -H 'Content-Type: application/octet-stream' --negotiate -u : 'http://neoap032:50070/webhdfs/v1/ntk/apps/3.0/tez/tez.tar.gz?op=CREATE&overwritrue&permission=444&noredirect=false' 2、NameNode通过http 307 Temporary Redirect 在response head的location中返回实际需要写入的dataNode url地址,形如: {"Location":"http://neoap033:1022/webhdfs/v1/ntk/apps/3.0/tez/tez.tar.gz?op=CREATE&delegation=HQAEaGRmcwRoZGZzAIoBh0aBQE6KAYdqjcROj4oXFHguAEscAZRLJUN0kZPXeJpJXZSPEldFQkhERlMgZGVsZWdhdGlvbhMxNzIuMzIuMTUzLjE1MDo4MDIw&namenoderpcaddress=newland&createflag=&createparent=true&overwrite=true&permission=444"} 3、curl 继续向重定向的Location地址发起写请求,在这个步骤 centos7.x是oK的,会根据location返回的重定向地址继续进行写操作,最后返回Http 201;而openEuler 20.03的curl直接返回了307,没有进行这步操作。 4、如果在centos7.4curl请求中设置不允许redirect,noredirect=true,可以模拟出来同样的报错和返回值
Comments (
4
)
Sign in
to comment
Status
Backlog
Backlog
Doing
Done
Declined
Assignees
Not set
sherlock2010
sherlock2010
Assignee
Collaborator
+Assign
+Mention
Labels
sig/Networking
Not set
Projects
Unprojected
Unprojected
Milestones
No related milestones
No related milestones
Pull Requests
None yet
None yet
Successfully merging a pull request will close this issue.
Branches
No related branch
Branches (
-
)
Tags (
-
)
Planed to start   -   Planed to end
-
Top level
Not Top
Top Level: High
Top Level: Medium
Top Level: Low
Priority
Not specified
Serious
Main
Secondary
Unimportant
Duration
(hours)
参与者(5)
1
https://gitee.com/src-openeuler/curl.git
git@gitee.com:src-openeuler/curl.git
src-openeuler
curl
curl
Going to Help Center
Search
Git 命令在线学习
如何在 Gitee 导入 GitHub 仓库
Git 仓库基础操作
企业版和社区版功能对比
SSH 公钥设置
如何处理代码冲突
仓库体积过大,如何减小?
如何找回被删除的仓库数据
Gitee 产品配额说明
GitHub仓库快速导入Gitee及同步更新
什么是 Release(发行版)
将 PHP 项目自动发布到 packagist.org
Comment
Repository Report
Back to the top
Login prompt
This operation requires login to the code cloud account. Please log in before operating.
Go to login
No account. Register