Enterprise Columnstore Docker Project

Summary

MariaDB ColumnStore is a columnar storage engine that utilizes a massively parallel distributed data architecture. It was built by porting InfiniDB to MariaDB and has been released under the GPL license.

MariaDB ColumnStore is designed for big data scaling to process petabytes of data, linear scalability and exceptional performance with real-time response to analytical queries. It leverages the I/O benefits of columnar storage, compression, just-in-time projection, and horizontal and vertical partitioning to deliver tremendous performance when analyzing large data sets.

Requirements

Please install the following software packages before you begin.

• Git
• Docker

Also make sure to grab your download credentials from our website:

• MariaDB Enterprise Token

Docker-Compose Cluster Instructions

This project will produce a 3 node MariaDB Columnstore cluster fronted by 2 MaxScale load balancers.

1. $ git clone https://github.com/mariadb-corporation/mariadb-enterprise-columnstore-docker.git
2. $ cd mariadb-enterprise-columnstore-docker
3. $ cp .env_example .env
4. Customize the new .env file
5. $ docker-compose up -d && docker exec -it mcs1 provision

If you have configured your .env file properly, you should now see an output like this:

Waiting for PM1 to be initialized ................... done
Adding PM1 to CMAPI ... done
Adding PM2 to CMAPI ... done
Adding PM3 to CMAPI ... done
Adding PM1 to MaxScale 1 ... done
Adding PM2 to MaxScale 1 ... done
Adding PM3 to MaxScale 1 ... done
Adding PM1 to MaxScale 2 ... done
Adding PM2 to MaxScale 2 ... done
Adding PM3 to MaxScale 2 ... done
Adding SERVICE to MaxScale 1 ... done
Adding SERVICE to MaxScale 2 ... done
Adding LISTENER to MaxScale 1 ... done
Adding LISTENER to MaxScale 2 ... done
Adding MONITOR to MaxScale 1 ... done
Adding MONITOR to MaxScale 2 ... done
Sleeping 5 seconds ... done
Validating ColumnStore engine ... done

MariaDB Access

• PM1: $ docker exec -it mcs1 bash
• PM2: $ docker exec -it mcs2 bash
• PM3: $ docker exec -it mcs3 bash

MaxScale 1 GUI Access

• url: http://127.0.0.1:8989
• username: admin
• password: mariadb

MaxScale 2 GUI Access

• url: http://127.0.0.1:8990
• username: admin
• password: mariadb

Cluster Manipulation Tools

The following commands are a collection of aliases provided as shortcuts to common Columnstore management tools.

• core Change directory to /var/log/mariadb/columnstore/corefiles
• dbrm Change directory to /var/lib/columnstore/data1/systemFiles/dbrm
• extentSave Backup extent map
• mcsModule View current module name
• mcsStart Start cluster via CMAPI
• mcsStatus Get cluster status via CMAPI
• mcsShutdown Shutdown cluster via CMAPI
• mcsReadOnly Set Columnstore to read only
• mcsReadWrite Set Columnstore to read/write
• tcrit Tail crit.log
• tdebug Tail debug.log
• terror Tail error.log
• tinfo Tail info.log
• twarning Tail warning.log

Other Glossary Items

• PM: Performance Module
• PM1: Primary Database Node
• PM2: Secondary Database Node
• PM3: Tertiary Database Node
• MX1: Primary MaxScale Node
• MX2: Secondary MaxScale Node

REST-API Instructions

Format of url endpoints for REST API:

https://{server}:{port}/cmapi/{version}/{route}/{command}

Examples urls for available endpoints:

• https://127.0.0.1:8640/cmapi/0.4.0/cluster/status
• https://127.0.0.1:8640/cmapi/0.4.0/cluster/start
• https://127.0.0.1:8640/cmapi/0.4.0/cluster/shutdown
• https://127.0.0.1:8640/cmapi/0.4.0/cluster/node
• https://127.0.0.1:8640/cmapi/0.4.0/cluster/remove-node
• https://127.0.0.1:8640/cmapi/0.4.0/cluster/mode-set

Request Headers Needed:

• 'x-api-key': 'somekey123'
• 'Content-Type': 'application/json'

Note: x-api-key can be set to any value of your choice during the first call to the server. Subsequent connections will require this same key

Examples using curl:

Get Status:

curl -s https://127.0.0.1:8640/cmapi/0.4.0/cluster/status --header 'Content-Type:application/json' --header 'x-api-key:somekey123' -k | jq .

Start Cluster:

curl -s -X PUT https://127.0.0.1:8640/cmapi/0.4.0/cluster/start --header 'Content-Type:application/json' --header 'x-api-key:somekey123' --data '{"timeout":20}' -k | jq .

Stop Cluster:

curl -s -X PUT https://127.0.0.1:8640/cmapi/0.4.0/cluster/shutdown --header 'Content-Type:application/json' --header 'x-api-key:somekey123' --data '{"timeout":20}' -k | jq .

Add Node:

curl -s -X PUT https://127.0.0.1:8640/cmapi/0.4.0/cluster/node --header 'Content-Type:application/json' --header 'x-api-key:somekey123' --data '{"timeout":20, "node": "<replace_with_desired_hostname>"}' -k | jq .

Remove Node:

curl -s -X DELETE https://127.0.0.1:8640/cmapi/0.4.0/cluster/node --header 'Content-Type:application/json' --header 'x-api-key:somekey123' --data '{"timeout":20, "node": "<replace_with_desired_hostname>"}' -k | jq .

Set Read Only Mode:

curl -s -X PUT https://127.0.0.1:8640/cmapi/0.4.0/cluster/mode-set --header 'Content-Type:application/json' --header 'x-api-key:somekey123' --data '{"timeout":20, "mode": "readonly"}' -k | jq .

Set Read/Write Mode:

curl -s -X PUT https://127.0.0.1:8640/cmapi/0.4.0/cluster/mode-set --header 'Content-Type:application/json' --header 'x-api-key:somekey123' --data '{"timeout":20, "mode": "readwrite"}' -k | jq .

Single Container Instructions

1. $ git clone https://github.com/mariadb-corporation/mariadb-enterprise-columnstore-docker.git
2. $ cd mariadb-enterprise-columnstore-docker
3. $ docker run -d --shm-size=512m -p 3306:3306 --hostname mcs1 --env PM1=mcs1 --name mcs1 mariadb/enterprise-columnstore && docker exec -it mcs1 provision

You should now see an output like this:

Waiting for PM1 to be initialized ................... done
Adding PM1 to CMAPI ... done
Sleeping 5 seconds ... done
Validating ... done

Run Variables

Example Use

docker run -d -p 3306:3306 \
--shm-size=512m \
--env USE_S3_STORAGE=true \
--env S3_BUCKET=my-bucket \
--env S3_ENDPOINT=s3.myendpoint.com \
--env S3_ACCESS_KEY_ID=myAccessKeyId \
--env S3_SECRET_ACCESS_KEY=mySuperSecr3tAcceS$key \
--name mcs1 mariadb/enterprise-columnstore \
&& docker exec -it mcs1 provision

Custom Build Instructions

$ docker build . --tag <your_image_name> --build-arg VERSION=10.6 --build-arg TOKEN=your_mariadb_enterprise_token

Build Variables