6 Star 0 Fork 0

Apache Big Data / cloudera-hue

 / 详情

Hue - 提示支持Nginx负载均衡、高可用

Done
member
Opened this issue  
2021-12-01 10:56

修改Hue源码提示使用负载均衡URL链接

修改:/usr/bin/vi /opt/hue/desktop/core/src/desktop/templates/hue.mako

搜索main-page,在以下代码

% if banner_message or conf.CUSTOM.BANNER_TOP_HTML.get():
<div class="banner">
    ${ banner_message or conf.CUSTOM.BANNER_TOP_HTML.get() | n,unicode }
</div>
% endif

前添加:

    <!-- Add by FengZhou, support Nginx loadbalance. -->
    % if request is not None and request.META['HTTP_USER_AGENT'] != 'nginx':
          <div class="banner" style="background-color:#317298;color:#ffffff;text-align:center;padding:5px 0;">
            您正在访问一个未经优化的Hue,这可能会导致您的查询效率变慢。请访问: <a style="color:#ffffff;text-decoration:underline;" href="http://hadoop1:8889">http://hadoop1:8889</a> 来使用Hue.
          </div>
    % endif

实现如果用户直接访问Hue,而不使用Nginx反向代理,显示如下:

image-20211201103914160

部署多套Hue

准备工作

安装操作系统依赖项。

yum install -y asciidoc cyrus-sasl-devel cyrus-sasl-gssapi cyrus-sasl-plain krb5-devel libffi-devel libxml2-devel libxslt-devel

部署多套Hue

1、拷贝其他几套Hue到hadoop2、hadoop3、hadoop4、hadoop5节点。

copy-files.sh /opt/hue-4.9.0-bin /opt masters

2、执行以下启动命令,启动多个hue。

# 创建快捷方式
ssh -i /opt/key/id_rsa hadoop2 "ln -snf /opt/hue-4.9.0-bin/hue /opt/hue"
ssh -i /opt/key/id_rsa hadoop3 "ln -snf /opt/hue-4.9.0-bin/hue /opt/hue"
ssh -i /opt/key/id_rsa hadoop4 "ln -snf /opt/hue-4.9.0-bin/hue /opt/hue"
ssh -i /opt/key/id_rsa hadoop5 "ln -snf /opt/hue-4.9.0-bin/hue /opt/hue"

# 修改权限
ssh -i /opt/key/id_rsa hadoop2 "chown -R hue:hadoop /opt/hue-4.9.0-bin"
ssh -i /opt/key/id_rsa hadoop3 "chown -R hue:hadoop /opt/hue-4.9.0-bin"
ssh -i /opt/key/id_rsa hadoop4 "chown -R hue:hadoop /opt/hue-4.9.0-bin"
ssh -i /opt/key/id_rsa hadoop5 "chown -R hue:hadoop /opt/hue-4.9.0-bin"

# 清理之前创建的日志文件
ssh -i /opt/key/id_rsa hadoop2 "rm -f /opt/hue-4.9.0-bin/hue/logs/*"
ssh -i /opt/key/id_rsa hadoop3 "rm -f /opt/hue-4.9.0-bin/hue/logs/*"
ssh -i /opt/key/id_rsa hadoop4 "rm -f /opt/hue-4.9.0-bin/hue/logs/*"
ssh -i /opt/key/id_rsa hadoop5 "rm -f /opt/hue-4.9.0-bin/hue/logs/*"

3、分别启动hue,然后浏览器打开测试访问。

ssh -i /opt/key/id_rsa hadoop1 "su - hue -c '/opt/hue/build/env/bin/supervisor --daemon'"
ssh -i /opt/key/id_rsa hadoop2 "su - hue -c '/opt/hue/build/env/bin/supervisor --daemon'"
ssh -i /opt/key/id_rsa hadoop3 "su - hue -c '/opt/hue/build/env/bin/supervisor --daemon'"
ssh -i /opt/key/id_rsa hadoop4 "su - hue -c '/opt/hue/build/env/bin/supervisor --daemon'"
ssh -i /opt/key/id_rsa hadoop5 "su - hue -c '/opt/hue/build/env/bin/supervisor --daemon'"

# 关闭Hue
ps aux | grep hue | grep python2.7 | awk 'BEGIN {pids = ""} $0 ~ /supervisor|runcherrypyserver/ { pids = $2 " " pids } END { print pids }' | xargs kill -9

部署Nginx

安装依赖

yum -y install make zlib zlib-devel gcc-c++ libtool  openssl openssl-devel

安装pcre

下载地址:http://downloads.sourceforge.net/project/pcre/pcre/8.35/pcre-8.35.tar.gz

tar zxvf pcre-8.35.tar.gz
cd pcre-8.35
./configure
make && make install
pcre-config --version

安装nginx

下载地址:http://nginx.org/download/nginx-1.6.2.tar.gz

useradd nginx
tar zxvf nginx-1.16.1.tar.gz
cd nginx-1.16.1
./configure --user=nginx --prefix=/usr/local/nginx --with-pcre=/root/nginx/pcre-8.35
make
make install

启动nginx

/usr/local/nginx/sbin/nginx						 # 启动 Nginx
/usr/local/nginx/sbin/nginx -s reload            # 重新载入配置文件
/usr/local/nginx/sbin/nginx -s reopen            # 重启 Nginx
/usr/local/nginx/sbin/nginx -s stop              # 停止 Nginx

测试访问nginx

http://hadoop1/

反向代理 + 负载均衡

修改配置文件为以下:/usr/bin/vi /usr/local/nginx/conf/nginx.conf


#user  nobody;
worker_processes  1;

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
error_log  logs/error.log  info;

#pid        logs/nginx.pid;


events {
    worker_connections  1024;
}


http {

	# 配置Hue服务器地址,注意hadoop1配置为机器名称
	upstream hadoop1{
		ip_hash;
		server hadoop1:8888;
		server hadoop2:8888;
		server hadoop3:8888;
		server hadoop4:8888;
		server hadoop5:8888;
	}

    include       mime.types;
    default_type  application/octet-stream;

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  logs/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    #gzip  on;

    server {
        listen       8889;
        # 配置服务器名称
        server_name  10.94.158.51;
        
        location / {
        	set $agent "nginx";
        	proxy_set_header User-Agent $agent;
        	proxy_pass	http://hadoop1;
        }
        
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }

}

重新加载配置文件

/usr/local/nginx/sbin/nginx -s reload

测试

Hue访问优化提示测试

1、直接访问Hue。

image-20211201105204973

2、通过Nginx反向代理访问Hue。

image-20211201105224682

高可用停机测试

1、启动5台Hue服务器。

ssh -i /opt/key/id_rsa hadoop1 "su - hue -c '/opt/hue/build/env/bin/supervisor --daemon'"
ssh -i /opt/key/id_rsa hadoop2 "su - hue -c '/opt/hue/build/env/bin/supervisor --daemon'"
ssh -i /opt/key/id_rsa hadoop3 "su - hue -c '/opt/hue/build/env/bin/supervisor --daemon'"
ssh -i /opt/key/id_rsa hadoop4 "su - hue -c '/opt/hue/build/env/bin/supervisor --daemon'"
ssh -i /opt/key/id_rsa hadoop5 "su - hue -c '/opt/hue/build/env/bin/supervisor --daemon'"

访问Nginx:hadoop1:8889。

2、关闭第一台Hue服务器。

ps aux | grep hue | grep python2.7 | awk 'BEGIN {pids = ""} $0 ~ /supervisor|runcherrypyserver/ { pids = $2 " " pids } END { print pids }' | xargs kill -9

刷新Nginx,依然能够访问。

3、关闭第二台服务器。刷新Nginx,依然能够访问。

4、关闭第三台服务器。刷新Nginx,依然能够访问。

5、关闭第四台服务器。刷新Nginx,依然能够访问。

5、关闭第五台服务器。到目前为止,所有Hue服务器全部都已经关闭。刷新Nginx,Nginx报错。

image-20211201104604317

符合预期。

测试Livy session共享

1、启动5台Hue服务器。

ssh -i /opt/key/id_rsa hadoop1 "su - hue -c '/opt/hue/build/env/bin/supervisor --daemon'"
ssh -i /opt/key/id_rsa hadoop2 "su - hue -c '/opt/hue/build/env/bin/supervisor --daemon'"
ssh -i /opt/key/id_rsa hadoop3 "su - hue -c '/opt/hue/build/env/bin/supervisor --daemon'"
ssh -i /opt/key/id_rsa hadoop4 "su - hue -c '/opt/hue/build/env/bin/supervisor --daemon'"
ssh -i /opt/key/id_rsa hadoop5 "su - hue -c '/opt/hue/build/env/bin/supervisor --daemon'"

访问Nginx:hadoop1:8889。

2、使用sparksql执行一条SQL。

select category_id, count(1) from benchmark_sparksql.youtube_bigtable_3000w group by 1 order by 2;

3、退出Hue,重新再执行SQL。测试无需重新创建Session、且执行历史记录存在。

image-20211201105046309

测试下载

image-20211201105437420

数据能够正常下载。

Comments (0)

周锋 created任务
周锋 set assignee to 周锋
周锋 changed issue state from 待办的 to 进行中
周锋 added
 
enhancement
label
周锋 set branch to 4.9
周锋 changed issue state from 进行中 to 已完成
Expand operation logs

Sign in to comment

Status
Assignees
Milestones
Pull Requests
Successfully merging a pull request will close this issue.
Branches
Planed to start   -   Planed to end
-
Top level
Priority
参与者(1)
Python
1
https://gitee.com/bigdatago/cloudera-hue.git
git@gitee.com:bigdatago/cloudera-hue.git
bigdatago
cloudera-hue
cloudera-hue

Search

53164aa7 5694891 3bd8fe86 5694891