K3s Cluster in Docker-Compose Running PHP Nginx

Notes/Web-hosting , , ,

Run a K3s cluster in docker-compose with PHP + Nginx on 1 Gb, 1 vCPU server.

If anything goes wrong: read documents; check if the versions match (things are changing every day)

At the end of this page includes a list of links used as reference when writing.

Table of Content

Environment

  • 1 GB Memory
  • 1 vCPU
  • 25 GB SSD
  • Debian 10.2 on Digital Ocean Droplet

Prerequisites

  • docker
  • docker-compose
  • a domain if wish to use https (and certificates)

Try this to set up but it’s more stable to use iptables at this moment (01/04/2020):

Set Up Debian 10 Server on Digital Ocean

Set Up Directory

Feel free to choose any work directory, but for the purpose of simplicity, a directory owned by a non-root user in the root directory (/) will be used for this task.

1
2
3
4
sudo mkdir -v /docker/
# replace with your non-root user name and group
# (default group name is the same as the user name)
sudo chown __NAME__:__GROUP__ /docker/

To make it easier to manage and run scripts, create a bin directory

  • here it is under /docker/
  • alternatively, put it under ~/, some distribution will automatically append it to $PATH
    • ($PATH is a list of directories that the shell will use to look for commands/executables)
  • editing the $PATH variable is optional
    • it makes calling the script the same as calling normal commands
1
2
3
4
5
mkdir -v /docker/bin/

# pre-pend it to PATH, making the search end earlier
echo "PATH=/docker/bin:$PATH" >> ~/.bashrc
source ~/.bashrc

It doesn’t hurt to create several sub-directories for:

  • Kubernetes resource yaml files
  • volumes to mount
1
2
3
4
5
6
mkdir -v /docker/k3s
mkdir -v /docker/kube

# this directory holds the PHP code, i.e. the index.php
# explained later
mkdir -v /docker/www

Launch the Cluster using Docker-Compose

As easy as one simple docker-compile file from k3s official repo

Modifications:

  • rename services
  • disable traefik by --no-deploy traefik
  • mount directory with kubeconfig to host’s ./k3s (created above)
  • mount php-code directory to container’s /var/www
  • open 80 and 443 (http/https) port for agents
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
# to run define K3S_TOKEN, K3S_VERSION is optional, eg:
#   K3S_TOKEN=${RANDOM}${RANDOM}${RANDOM} docker-compose up

version: '3'
services:

  k3s-server:
    image: "rancher/k3s:${K3S_VERSION:-latest}"
    command: server --no-deploy traefik
    tmpfs:
    - /run
    - /var/run
    privileged: true
    environment:
    - K3S_TOKEN=${K3S_TOKEN:?err}
    - K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml
    - K3S_KUBECONFIG_MODE=666
    volumes:
    - k3s-server:/var/lib/rancher/k3s
    # This is just so that we get the kubeconfig file out
    - ./k3s:/output
    - /docker/www:/var/www
    ports:
    - 6443:6443

  k3s-agent:
    image: "rancher/k3s:${K3S_VERSION:-latest}"
    tmpfs:
    - /run
    - /var/run
    privileged: true
    environment:
    - K3S_URL=https://k3s-server:6443
    - K3S_TOKEN=${K3S_TOKEN:?err}
    volumes:
    - /docker/www:/var/www
    ports:
    - 80:80
    - 443:443

volumes:
  k3s-server: {}

Let’s create a script to run it:

1
2
touch /docker/bin/k3s-up
chmod u+x /docker/bin/k3s-up

Inside the up script:

1
2
3
4
5
6
7
8
9
10
#!/bin/bash

# feel free to change this
export K3S_TOKEN=${RANDOM}${RANDOM}${RANDOM}${RANDOM}${RANDOM}

echo "K3S_TOKEN: ${K3S_TOKEN}"
# might want to store token somewhere, e.g.
#echo ${K3S_TOKEN} > ~/.token/K3S_TOKEN

docker-compose -f /docker/docker-compose.yml up -d --scale k3s-agent="${1:-1}"

Now we just need to run this script

  • since the directory is in $PATH, we can call it directly
  • it creates 1 agent by default, can change it by providing command line argument
1
2
k3s-up    # spawn 1 agents
#k3s-up 3 # spawn 3 agents

Due to extreme memory constrain, let’s begin with 1 agent

Talk to the Cluster

Now the k3s cluster is up and running.

The system might go through a short thrashing period but it will settle down soon (tested on real machine).

It’s time to kubectl. Let’s use the $KUBECONFIG variable to simplify things:

1
export KUBECONFIG=/docker/k3s/kubeconfig.yaml

Might as well add this line to bashrc (optional):

  • so that current user’s shell will automatically set KUBECONFIG variable
1
echo "export KUBECONFIG=/docker/k3s/kubeconfig.yaml" >> ~/.bashrc

Test connection

1
2
kubectl get nodes
kubectl get pods --all-namespaces

If not working, consult docker and other logs

1
2
3
docker ps
docker logs
...

Create Cluster Configs

Attempts to use nginx ingress have been made in vain because of the limited resources.

We shall use our own nginx, plus load balancer service provided by k3s (good to be simple for simple tasks).

Start with the config maps:

  • note: provided nginx configs may not suit every one’s need
1
2
3
mkdir /docker/kube/config-nginx-key      # SSL key, if intend to use https
mkdir /docker/kube/config-nginx-sites    # nginx config for each site
mkdir /docker/kube/config-nginx-snippets # re-useable snippets

Let’s go through each config file

SSL Key

(skip if using http)

Files in /docker/kube/config-nginx-key:

1
2
3
4
5
6
7
8
# SSL certificates
cert.pem
cert.key

# Diffie-Hellman parameters
# used by Nginx
# https://wiki.openssl.org/index.php/Diffie-Hellman_parameters
dhparam.pem # openssl dhparam -out ./dhparam.pem 4096

For security reasons, my own SSL certificates will not be included here.

  • Because I use Cloudflare, my cluster uses Cloudflare Origin CA certificates
  • Also it does not require complex verification, renewal steps etc.
  • If not suitable, might consider a cert manager (linked article uses ingress). That is beyond of the scope of this article

Nginx Snippets

Files in /docker/kube/config-nginx-snippets:

1
2
3
ssl-some.host.conf
ssl-params.conf
security.conf

ssl-some.host.conf

  • just telling nginx which certificates to use
1
2
ssl_certificate     /etc/nginx/ssl-key/cert.pem;
ssl_certificate_key /etc/nginx/ssl-key/cert.key;

ssl-params.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# SSL
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off;

# Diffie-Hellman parameter for DHE ciphersuites
ssl_dhparam /etc/nginx/ssl-key/dhparam.pem;

# Mozilla Intermediate configuration
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;

# OCSP Stapling
ssl_stapling on;
ssl_stapling_verify on;
resolver 1.1.1.1 1.0.0.1 8.8.8.8 8.8.4.4 208.67.222.222 208.67.220.220 valid=60s;
resolver_timeout 2s;

security.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# security headers
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header X-Content-Type-Options "nosniff" always;
add_header Referrer-Policy "no-referrer-when-downgrade" always;
add_header Content-Security-Policy "default-src 'self' http: https: data: blob: 'unsafe-inline'" always;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;

# . files
location ~ /\.(?!well-known) {
	deny all;
}

## misc
## not for security but included here for simplicity

# favicon.ico
location = /favicon.ico {
	log_not_found off;
	access_log off;
}

# robots.txt
location = /robots.txt {
	log_not_found off;
	access_log off;
}

Nginx Site

1
/docker/kube/config-nginx-sites/php.conf

Heavily inspired by:

Choose one of the below:

  • http
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
server {
  index index.php index.html;
  root /var/www;

  location / {
      try_files $uri $uri/ /index.php?$query_string;
  }

  location ~ \.php$ {
      try_files $uri =404;
      fastcgi_split_path_info ^(.+\.php)(/.+)$;
      fastcgi_pass kube-php:9000;
      fastcgi_index index.php;
      include fastcgi_params;
      fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
      fastcgi_param PATH_INFO $fastcgi_path_info;
    }
}
  • https
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
server {
  listen 443 ssl http2;
  listen [::]:443 ssl http2;

  include /etc/nginx/snippets/ssl-some.host.conf;
  include /etc/nginx/snippets/ssl-params.conf;

  root /var/www;
  index  index.html  index.php;

  ### change to your own host name ###
  server_name some.host;

  include /etc/nginx/snippets/security.conf;

  location ~ [^/]\.php(/|$) {
    fastcgi_split_path_info ^(.+?\.php)(/.*)$;
    if (!-f $document_root$fastcgi_script_name) {
        return 404;
    }
    fastcgi_param HTTP_PROXY "";
    fastcgi_pass kube-php:9000;
    fastcgi_index index.php;
    include fastcgi_params;
    fastcgi_param  SCRIPT_FILENAME   $document_root$fastcgi_script_name;
  }
}

Back to our kubernetes cluster, now we need to create:

  • PHP code
  • PHP service
  • PHP deployment
  • nginx service LoadBalancer
  • nginx deployment
1
mkdir -v /docker/kube/objects

Will use one file for related resources

PHP code

For simplicity, the PHP code is directly put into /docker/www

Sample PHP file, put to /docker/www/index.php:

1
2
<?php
echo phpinfo();

PHP Resources

1
editor /docker/kube/objects/php.yaml

Inside the yaml:

  • it exposes port 9000 via ClusterIP (default networking for service) for php-fpm
  • php:7-fpm image is used
  • the php code directory mounted earlier, /var/www, is mounted as a hostPath volume
    • generally not something desired in production
    • A possibly related link and another.
    • might consider a kubernetes init container to set up the code etc.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
apiVersion: v1
kind: Service
metadata:
  name: kube-php
  labels:
    tier: backend
spec:
  ports:
  - protocol: TCP 
    port: 9000
  selector:
    app: kube-php
    tier: backend
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-php
  labels:
    tier: backend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kube-php
      tier: backend
  template:
    metadata:
      labels:
        app: kube-php
        tier: backend
    spec:
      containers:
      - name: php 
        image: php:7-fpm
        volumeMounts:
        - name: www
          mountPath: /var/www
      volumes:
      - name: www
        hostPath:
          path: /var/www
          type: Directory

Nginx Resources

1
editor /docker/kube/objects/nginx.yaml

Inside the yaml:

  • a LoadBalancer service is created with port 80 and 443
  • a deployment is created using nginx:1.16 image
    • a naive and simple approach to test the config online is included
    • the config maps to be created are mounted as directories
  • about nginx conf:
    • nginx will automatically load conf in /etc/nginx/conf.d
    • for our case, our php.conf will include all other config files
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
apiVersion: v1
kind: Service
metadata:
  name: kube-nginx
  labels:
    tier: backend
spec:
  type: LoadBalancer
  ports:
  - protocol: TCP
    port: 80
    name: http
  - protocol: TCP
    port: 443
    name: https
  selector:
    app: kube-nginx
    tier: backend
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-nginx
  labels:
    tier: backend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kube-nginx
      tier: backend
  template:
    metadata:
      labels:
        app: kube-nginx
        tier: backend
    spec:
      containers:
      - name: nginx
        image: nginx:1.16
        ## uncomment this to test the nginx config
        ## use something like `kubectl logs kube-nginx-...` to see the output
        ## use `kubectl delete deployment ...` to remove the test deployment
        #command: ["/bin/bash","-c"]
        #args: ["echo testing-nginx-conf; nginx -t; sleep 10h"]
        ports:
        - containerPort: 80
        - containerPort: 443
        volumeMounts:
        - name: nginx-key
          mountPath: /etc/nginx/ssl-key
        - name: nginx-snippets
          mountPath: /etc/nginx/snippets
        - name: nginx-sites
          mountPath: /etc/nginx/conf.d
        - name: www
          mountPath: /var/www
      volumes:
      - name: nginx-key
        configMap:
          name: config-nginx-key
      - name: nginx-snippets
        configMap:
          name: config-nginx-snippets
      - name: nginx-sites
        configMap:
          name: config-nginx-sites
      - name: www
        hostPath:
          path: /var/www
          type: Directory

Config and Run the Cluster

Now we have both the configs and the Kubernetes resources files ready.

Just a short script will do all the setup

1
2
touch /docker/bin/k3s-setup
chmod u+x /docker/bin/k3s-setup

Inside the setup script

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
#!/bin/bash

# ensure the config works
export KUBECONFIG=/docker/k3s/kubeconfig.yaml

set -e

# test connection (optional)
kubectl get node

# nginx config
kubectl create configmap config-nginx-key      --from-file=/docker/kube/config_nginx-key
kubectl create configmap config-nginx-snippets --from-file=/docker/kube/config_nginx-snippets
kubectl create configmap config-nginx-sites    --from-file=/docker/kube/config_nginx-sites

# apply 
kubectl apply -f /docker/kube/objects

Run the script

1
k3s-setup

Result

The whole server might suffer from a short period of thrashing after starting everything.

Assume all firewall, certificates and so forth all set up, go to http://host-ip or https://some.host (depend on your choice).

The PHP application will show up.

Each access to my host leads to around 40K context switches.

Stop and Clean up

If encounter errors in the previous steps, or need a graceful shutdown, here is a nothing-left-behind clean up script.

1
2
touch /docker/bin/k3s-down
chmod u+x /docker/bin/k3s-down

Inside the down script:

  • shut down via docker-compose
  • remove the used master server volume
1
2
3
4
5
6
7
8
9
10
# the existence of this variable is required by the docker-compose file
export K3S_TOKEN=NOTHING
# if you want the original token, might consider storing it somewhere, e.g.
#export K3S_TOKEN=$(cat ~/.token/K3S_TOKEN)

docker-compose -f /docker/docker-compose.yml down -v

# optional, delete the no-longer-valid kubeconfig
rm -f /docker/k3s/kubeconfig.yaml
#might remove the saved K3S_TOKEN as well

Conclusion and Thoughts

The purpose of Kubernetes is not, clearly, doing things like this.

But the k3s cluster created is perfect for dev and testing, especially for single-machine. And it’s fun to run a fully-functional Kubernetes on the cutest VPS!

Future Works

Reference

Some of documents/tutorials are not updated or no longer working.

Tutorials

K3s

Kubernetes

Kubernetes Volumes

Kubernetes Networking

Nginx config