[1.4.0RC2][on-yarn] hetu-cli fails with untrusted certificate exception when connecting to a secure OLK instance.

Software Environment:

openLooKeng version (source or binary):
tar package of September 15
OS platform & distribution (eg., Linux Ubuntu 16.04):
centos7.6
Java version:
jdk1.8.0

Describe the current behavior

Failed to deploy the security cluster

Describe the expected behavior

Steps to reproduce the issue

1.Enable SSL communication.
config.properties.template add ssl Related Configurations
One question:
输入图片说明
This file is local. How do I upload it to Yarn?
2.I simply used the command, and the error was reported.

输入图片说明

Related log/screenshots

Special notes for this issue

@dengran You have not selected a milestone,please select a milestone.After setting the milestone, you can use the /check-milestone command to remove the needs-milestone label.

Issue reproduced.
Also need to update advanced configuration under documents.

This has been addressed in the wiki: https://gitee.com/openlookeng/openlookeng-on-yarn/wikis

I didn't find any information about security clusters on the wiki

I investigated this and was able to start an SSL cluster in my development environment, but the steps were a bit awkward.

Here is how I set up the keystore.jks file:

I had to put keystore.jks on the machine manually, I couldn't have the script automatically upload/download it from HDFS. It's okay to manually upload the file to HDFS (hdfs dfs -put <the file> <the destination in HDFS>), and manually download it from HDFS on the destination machine (hdfs dfs -get <file location in HDFS> <destination on the machine>). In my example, I put the keystore file at /tmp/keystore.jks on my machine.
The named used when creating the keystore file has to be the same as the machine's fully qualified domain name.

Here is the config.properties.template that I used on coordinator nodes:

coordinator=true
node-scheduler.include-coordinator=false
http-server.http.port=_OLK_PROPERTIES_TEMPLATE_HTTP_SERVER_PORT
query.max-memory=10GB
query.max-total-memory=10GB
query.max-memory-per-node=2GB
query.max-total-memory-per-node=2GB
discovery-server.enabled=true
discovery.uri=https://_OLK_PROPERTIES_TEMPLATE_CN_DISCOVERY_IP:_OLK_PROPERTIES_TEMPLATE_HTTP_SERVER_PORT
hetu.multiple-coordinator.enabled=_OLK_PROPERTIES_TEMPLATE_HA_ENABLED
hetu.embedded-state-store.enabled=_OLK_PROPERTIES_TEMPLATE_HA_ENABLED
hetu.queryeditor-ui.allow-insecure-over-http=true

http-server.http.enabled=false
node.internal-address-source=FQDN
http-server.https.enabled=true
http-server.https.port=_OLK_PROPERTIES_TEMPLATE_HTTP_SERVER_PORT
http-server.https.keystore.path=/tmp/keystore.jks
http-server.https.keystore.key=keystorepassword
internal-communication.https.required=true
internal-communication.https.keystore.path=/tmp/keystore.jks
internal-communication.https.keystore.key=keystorepassword

Here is the config.properties.template that I used on worker nodes:

coordinator=false
node-scheduler.include-coordinator=false
http-server.http.port=_OLK_PROPERTIES_TEMPLATE_HTTP_SERVER_PORT
query.max-memory=10GB
query.max-total-memory=10GB
query.max-memory-per-node=2GB
query.max-total-memory-per-node=2GB
discovery.uri=https://_OLK_PROPERTIES_TEMPLATE_CN_DISCOVERY_IP:_OLK_PROPERTIES_TEMPLATE_HTTP_SERVER_PORT
hetu.multiple-coordinator.enabled=_OLK_PROPERTIES_TEMPLATE_HA_ENABLED

http-server.http.enabled=false
node.internal-address-source=FQDN
http-server.https.enabled=true
http-server.https.port=_OLK_PROPERTIES_TEMPLATE_HTTP_SERVER_PORT
http-server.https.keystore.path=/tmp/keystore.jks
http-server.https.keystore.key=keystorepassword
internal-communication.https.required=true
internal-communication.https.keystore.path=/tmp/keystore.jks
internal-communication.https.keystore.key=keystorepassword

Could you try the steps from the above comment and see if that successfully starts the SSL cluster in your environment?

I think there's something wrong with it.
The startup is successful.
输入图片说明
but

Haven't returned for a long time so I manually stopped.
then
I set http-server.http.enabled=true at the same time

no worker

We think this is related to an untrusted SSL certificate. (Are there also errors when starting a cluster with SSL manually without yarn?)
Please ensure that the certificate used by olk is trusted by Java.
“Java Truststore File for TLS” from this page might be related? https://openlookeng.io/docs/docs/security/tls.html

This issue is not specific to olk-on-yarn. We need to update our OLK docs for the CLI to trust the server side certificate. Lower the priority, and remove the on-yarn lable.

The -Djavax.net.ssl.trustStore=client_truststore.jks Java option needs to be passed when calling the hetu-cli:

Here are the steps I used to generate the keystore, truststore, and certificate:

I generate the keystore, truststore, and certificate from your steps
输入图片说明

Here is the config.properties.template that I used on coordinator nodes:

coordinator=true
node-scheduler.include-coordinator=false
http-server.http.port=_OLK_PROPERTIES_TEMPLATE_HTTP_SERVER_PORT
query.max-memory=10GB
query.max-total-memory=10GB
query.max-memory-per-node=2GB
query.max-total-memory-per-node=2GB
discovery-server.enabled=true
discovery.uri=_OLK_PROPERTIES_TEMPLATE_DISCOVERY_URI
hetu.multiple-coordinator.enabled=_OLK_PROPERTIES_TEMPLATE_HA_ENABLED
hetu.embedded-state-store.enabled=_OLK_PROPERTIES_TEMPLATE_HA_ENABLED
hetu.queryeditor-ui.allow-insecure-over-http=true

node.internal-address=51-38-77-19.huawei.com
discovery.uri=https://51-38-77-19.huawei.com:9090
#node.internal-adress-source=FQDN
http-server.http.enabled=false
http-server.https.enabled=true
http-server.https.port=9090
#http-server.https.port=_OLK_PROPERTIES_TEMPLATE_HTTP_SERVER_PORT
http-server.https.keystore.path=/opt/keystore.jks
http-server.https.keystore.key=Huawei@123
internal-communication.https.required=true
internal-communication.https.keystore.path=/opt/keystore.jks
internal-communication.https.keystore.key=Huawei@123

Here is the config.properties.template that I used on worker nodes:

coordinator=false
node-scheduler.include-coordinator=false
http-server.http.port=_OLK_PROPERTIES_TEMPLATE_HTTP_SERVER_PORT
query.max-memory=10GB
query.max-total-memory=10GB
query.max-memory-per-node=2GB
query.max-total-memory-per-node=2GB
discovery.uri=_OLK_PROPERTIES_TEMPLATE_DISCOVERY_URI
hetu.multiple-coordinator.enabled=_OLK_PROPERTIES_TEMPLATE_HA_ENABLED

node.internal-address=51-38-77-19.huawei.com
discovery.uri=https://51-38-77-19.huawei.com:9090
#node.internal-address-source=FQDN
http-server.http.enabled=false
http-server.https.enabled=true
http-server.https.port=9090
#http-server.https.port=_OLK_PROPERTIES_TEMPLATE_HTTP_SERVER_PORT
http-server.https.keystore.path=/opt/keystore.jks
http-server.https.keystore.key=Huawei@123
internal-communication.https.required=true
internal-communication.https.keystore.path=/opt/keystore.jks
internal-communication.https.keystore.key=Huawei@123

python3 olk_on_yarn.py start -ha true -cn 2 -w 2 --use-nginx true --nginx-port 8070
输入图片说明

输入图片说明

If I set 'http-server.http.enabled=true',I can access the WebUI,but there is no worker.

When creating the keystore file, the section for "What is your first and last name?" needs to be the same as the machine's fully qualified domain name or unqualified hostname. (I'm not entirely sure which one it's supposed to be, for my environment they're the same. I recommend trying both?)

https://openlookeng.io/docs/docs/security/tls.html "You'll be prompted for the first and last name. Use the Common Name that will be used in the certificate. In this case, it should be the unqualified hostname of the openLooKeng coordinator"
https://docs.oracle.com/cd/E19509-01/820-3503/ggfen/index.html "You must specify a fully qualified domain for the “first and last name” question"

Also please refer to the olk-on-yarn documentation for SSL: https://gitee.com/openlookeng/openlookeng-on-yarn/wikis/openLooKeng%20Examples#ssl

Here are keystore, truststore, and certificate:
输入图片说明
keytool -v -list -keystore keystore.jks

keytool -v -list -keystore olk_trust.jks

输入图片说明
There are no http_uri like 'https://51-38-76-121.huawei.com:xxx' both cn and worker

openLooKeng / hetu-core

内容风险标识

Software Environment:

Describe the current behavior

Describe the expected behavior

Steps to reproduce the issue

Related log/screenshots

Special notes for this issue

评论 (15)

openLooKeng / hetu-core .gitee-modal { width: 500px !important; }

内容风险标识