title

René Jochum

Blog of René Jochum - A FOSS enthusiast who's programming since 2002.

Howto Install Rancher k3s with MariaDB Galera

Installing Rancher k3s with MariaDB Galera


In this blog Post we gonna install a HA Rancher Kubernetes Cluster with a MariaDB Galera Cluster as Datastore.

Outline

Prepare Ubuntu Bionic Server

You need 3 Nodes, 4 CPU, >8GiB RAM, 100GiB Disk, I have 3 Nodes 4 CPU, 24 GiB RAM, 250GiB Disk.

Install Ubuntu on one Server, remove snapd, ufw, cloud-init. Then clone it and edit /etc/hosts /etc/hostname /etc/netplan/50-cloud-init.yaml and rm -f /etc/ssh/ssh_host_* - reboot.

Install MariaDB Galera

Install the MariaDB repo on each server.

sudo apt-get install software-properties-common
sudo apt-key adv --fetch-keys 'https://mariadb.org/mariadb_release_signing_key.asc'
sudo add-apt-repository 'deb [arch=amd64] http://ftp.hosteurope.de/mirror/mariadb.org/repo/10.4/ubuntu bionic main'

sudo apt update
sudo apt install mariadb-server mariadb-client mariadb-backup
Secure MariaDB on each Node

Set Password with mysql_secure_installation:

pcdummy@rancher01:~$ sudo mysql_secure_installation

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
      SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we'll need the current
password for the root user. If you've just installed MariaDB, and
haven't set the root password yet, you should just press enter here.

Enter current password for root (enter for none): 
OK, successfully used password, moving on...

Setting the root password or using the unix_socket ensures that nobody
can log into the MariaDB root user without the proper authorisation.

You already have your root account protected, so you can safely answer 'n'.

Switch to unix_socket authentication [Y/n] n
 ... skipping.

You already have your root account protected, so you can safely answer 'n'.

Change the root password? [Y/n]  
New password: 
Re-enter new password: 
Password updated successfully!
Reloading privilege tables..
 ... Success!


By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them.  This is intended only for testing, and to make the installation
go a bit smoother.  You should remove them before moving into a
production environment.

Remove anonymous users? [Y/n] Y
 ... Success!

Normally, root should only be allowed to connect from 'localhost'.  This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] n
 ... skipping.

By default, MariaDB comes with a database named 'test' that anyone can
access.  This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n] Y
 - Dropping test database...
 ... Success!
 - Removing privileges on test database...
 ... Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n] Y
 ... Success!

Cleaning up...

All done!  If you've completed all of the above steps, your MariaDB
installation should now be secure.

Thanks for using MariaDB!
Configure Galera on each Node

Stop Mariadb

sudo systemctl stop mariadb

Liste on all Interfaces (if you want configure it to listen only on a specific address):

sudo sed -i 's/max_connections\t\t= 100/max_connections\t\t= 1000/g' /etc/mysql/my.cnf
sudo sed -i 's/bind-address\t\t= 127.0.0.1/#bind-address\t\t= 127.0.0.1/g' /etc/mysql/my.cnf

Enable Galera, Paste the following into /etc/mysql/mariadb.conf.d/99-cluster.cnf

[galera]

wsrep_on = on
wsrep_provider = /usr/lib/galera/libgalera_smm.so
wsrep_cluster_address = gcomm://10.128.1.17,10.128.1.18,10.128.1.19 
wsrep_cluster_name = k3s_cluster_0
wsrep_provider_options="gcache.size=512M"
wsrep_sst_method = mariabackup
wsrep_sst_auth = root:SuperSecretRootPassword

default_storage_engine = InnoDB
innodb_autoinc_lock_mode = 2
innodb_doublewrite = 1

binlog_format = ROW

And change the ip addresse for wsrep_cluster_address

Some tuning if you use this Galera cluster for other purposes

sudo nano /etc/mysql/mariadb.conf.d/98-tuning.cnf
[mysqld]
key_buffer_size=256M
thread_stack=192K
thread_cache_size=8
max_connections=1000
innodb_buffer_pool_size=2G
query_cache_limit=2M
query_cache_size=0
query_cache_type=0
table_open_cache=128
join_buffer_size=512k
table_definition_cache=-1
performance_schema=ON
innodb_log_file_size=256M
innodb_buffer_pool_instances=2
tmp_table_size=32M
max_heap_table_size=32M
Bootstrap the cluster

One one node run sudo galera_new_cluster

One the other 2 nodes run: sudo systemctl start mariadb.service

Check the MariaDB Galera Cluster
mysql -u root -p -e "SELECT * FROM information_schema.global_status WHERE variable_name IN ('WSREP_CLUSTER_STATUS','WSREP_LOCAL_STATE_COMMENT','WSREP_CLUSTER_SIZE','WSREP_EVS_REPL_LATENCY','WSREP_EVS_DELAYED','WSREP_READY');"
Install and configure MaxScale
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 135659E928C12247
sudo add-apt-repository 'deb [arch=amd64] http://downloads.mariadb.com/MaxScale/2.2/ubuntu bionic main'
sudo apt install maxscale
mysql -u root -p
CREATE USER 'maxscale'@'%' IDENTIFIED BY 'SuperSecretPassword';
GRANT SELECT ON mysql.user TO 'maxscale'@'%';
GRANT SELECT ON mysql.db TO 'maxscale'@'%';
GRANT SELECT ON mysql.tables_priv TO 'maxscale'@'%';
GRANT SELECT ON mysql.roles_mapping TO 'maxscale'@'%';
GRANT SHOW DATABASES ON *.* TO 'maxscale'@'%';
GRANT REPLICATION CLIENT ON *.* TO 'maxscale'@'%';
GRANT SUPER ON *.* TO maxscale@'%';
FLUSH PRIVILEGES;
exit

Generate MaxScale Keys note the encrypted password and write replace it in /etc/maxscale.cnf

sudo systemctl stop maxscale
sudo maxkeys /var/lib/maxscale/
sudo maxpasswd 'SuperSecretPassword'
sudo chown maxscale: /var/lib/maxscale/.secrets

/etc/maxscale.cnf

# MaxScale documentation:
# https://mariadb.com/kb/en/mariadb-enterprise/mariadb-maxscale-22/

# Global parameters
#
# Complete list of configuration options:
# https://mariadb.com/kb/en/mariadb-enterprise/mariadb-maxscale-22-mariadb-maxscale-configuration-usage-scenarios/

[maxscale]
threads=auto

# Server definitions
#
# Set the address of the server to the network
# address of a MariaDB server.
#

[server1]
type=server
address=10.248.8.1
port=3306
protocol=MariaDBBackend

[server2]
type=server
address=10.248.8.2
port=3306
protocol=MariaDBBackend

[server3]
type=server
address=10.248.8.3
port=3306
protocol=MariaDBBackend

# Monitor for the servers
#
# This will keep MaxScale aware of the state of the servers.
# MariaDB Monitor documentation:
# https://mariadb.com/kb/en/mariadb-enterprise/mariadb-maxscale-22-mariadb-monitor/

[MariaDB-Monitor]
type=monitor
module=galeramon
servers=server1,server2,server3
user=maxscale
passwd=D83ED4E84351BD822950FDE5C2991889
monitor_interval=2000

# Service definitions
#
# Service Definition for a read-only service and
# a read/write splitting service.
#

# ReadWriteSplit documentation:
# https://mariadb.com/kb/en/mariadb-enterprise/mariadb-maxscale-22-readwritesplit/

[Read-Write-Service]
type=service
router=readwritesplit
servers=server1,server2,server3
user=maxscale
passwd=D83ED4E84351BD822950FDE5C2991889

# This service enables the use of the MaxAdmin interface
# MaxScale administration guide:
# https://mariadb.com/kb/en/mariadb-enterprise/mariadb-maxscale-22-maxadmin-admin-interface/

[MaxAdmin-Service]
type=service
router=cli

# Listener definitions for the services
#
# These listeners represent the ports the
# services will listen on.
#

[Read-Write-Listener]
type=listener
service=Read-Write-Service
protocol=MariaDBClient
port=4006

[MaxAdmin-Listener]
type=listener
service=MaxAdmin-Service
protocol=maxscaled
socket=default
sudo systemctl start maxscale
sudo systemctl enable maxscale
Create the k3s Database

One one node run:

mysql -u root -p
CREATE DATABASE `k3s`;
GRANT ALL PRIVILEGES ON `k3s`.* TO 'k3s'@'%' IDENTIFIED BY '<superSecret>';

Deploy k3s

Install k3s one each nodes, one after another:

curl -sfL https://get.k3s.io | sh -s - server --datastore-endpoint="mysql://k3s:SuperSecretPassword@tcp(localhost:4006)/k3s" --no-deploy servicelb --no-deploy traefik

Check the nodes after.

sudo k3s kubectl get nodes

One one node copy the config (I choose node1 for that):

mkdir ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown -R $(whoami): ~/.kube

Install kubectl and helm

Install kubectl

sudo apt-get update && sudo apt-get install -y apt-transport-https
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubectl

Install helm to ~/bin

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

Install MetalLB

Install MetalLB (change the address range!)

See: metallb install

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml
# On first install only
kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"

metallb.yaml -> kubectl apply -f metallb.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 10.248.11.1-10.248.11.253    

Check the deployment

kubectl get pods -n metallb-system -l app=metallb -o wide

Install cert-manager for Let’s Encrypt

kubectl create namespace cert-manager
kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.15.0/cert-manager.crds.yaml
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --version v0.15.0
$ kubectl get pods --namespace cert-manager
NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-6bcdf8c5cc-5bcrg              1/1     Running   0          54s
cert-manager-cainjector-6659d6844d-zrr5h   1/1     Running   0          54s
cert-manager-webhook-547567b88f-ptrlg      1/1     Running   0          54s

Install Nginx

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm install ingress-nginx ingress-nginx/ingress-nginx \
    --namespace kube-system \
    --set controller.image.runAsUser=101 \
    --set defaultBackend.enabled=false

Install Rancher

helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
helm repo update
kubectl create namespace cattle-system
helm install rancher rancher-latest/rancher \
  --namespace cattle-system \
  --set hostname=rancher.example.org \
  --set ingress.tls.source=letsEncrypt \
  --set letsEncrypt.email=support@example.org
$ kubectl -n cattle-system rollout status deploy/rancher
Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available...

Waiting for deployment "rancher" rollout to finish: 1 of 3 updated replicas are available...
Waiting for deployment "rancher" rollout to finish: 2 of 3 updated replicas are available...
deployment "rancher" successfully rolled out

Forward HTTP/HTTPS to the Rancher Load Balancer IP

$ kubectl -n kube-system describe service/traefik
Name:                     traefik
Namespace:                kube-system
Labels:                   app=traefik
                          chart=traefik-1.81.0
                          heritage=Helm
                          release=traefik
Annotations:              field.cattle.io/publicEndpoints:
                            [{"addresses":["10.128.3.1"],"port":80,"protocol":"TCP","serviceName":"kube-system:traefik","allNodes":false},{"addresses":["10.128.3.1"],...
Selector:                 app=traefik,release=traefik
Type:                     LoadBalancer
IP:                       10.43.47.63
LoadBalancer Ingress:     10.128.3.1
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  32316/TCP
Endpoints:                10.42.0.6:80
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  30752/TCP
Endpoints:                10.42.0.6:443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type    Reason        Age                  From                Message
  ----    ------        ----                 ----                -------
  Normal  IPAllocated   16m                  metallb-controller  Assigned IP "10.128.3.1"
  Normal  nodeAssigned  3m10s (x5 over 16m)  metallb-speaker     announcing from node "rancher01"

Here the IP is 10.128.3.1 i forward HTTP (80) and HTTPS (443) to it.

Wait for the Let’s Encrypt Cert

$ kubectl -n cattle-system describe certificate
Name:         tls-rancher-ingress
Namespace:    cattle-system
Labels:       app=rancher
              chart=rancher-2.4.3
              heritage=Tiller
              release=rancher
Annotations:  <none>
API Version:  cert-manager.io/v1alpha2
Kind:         Certificate
Metadata:
  Creation Timestamp:  2020-05-07T23:07:41Z
  Generation:          1
  Owner References:
    API Version:           extensions/v1beta1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Ingress
    Name:                  rancher
    UID:                   625bd78c-819a-4ba5-8ed0-4e8cf0497860
  Resource Version:        23050
  Self Link:               /apis/cert-manager.io/v1alpha2/namespaces/cattle-system/certificates/tls-rancher-ingress
  UID:                     6e0886ec-8b2d-4459-8c60-f994f269a146
Spec:
  Dns Names:
    rancher.example.org
  Issuer Ref:
    Group:      cert-manager.io
    Kind:       Issuer
    Name:       rancher
  Secret Name:  tls-rancher-ingress
Status:
  Conditions:
    Last Transition Time:  2020-05-07T23:07:41Z
    Message:               Waiting for CertificateRequest "tls-rancher-ingress-2753661366" to complete
    Reason:                InProgress
    Status:                False
    Type:                  Ready
Events:
  Type    Reason        Age   From          Message
  ----    ------        ----  ----          -------
  Normal  GeneratedKey  18m   cert-manager  Generated a new private key
  Normal  Requested     18m   cert-manager  Created new CertificateRequest resource "tls-rancher-ingress-2753661366"

I had some troubles with cert-manager where it wasn’t able to access http:// without that it is not able to generate the certificate.

kauri.io - Some Informations from there howtoforge Galera - Install Galera