Skip to content

A Persistent Grafana Server with TLS Encryption

This project will walk through how to deploy a Grafana server that is persistent with TLS encryption.

Note this lab is not for production use.

Lab configuration

Below you will find the specifications for the environment used to run this lab. I am confident the lab is able to run on much less hardware or even on a set of Raspberry Pi's.

Virtualization Environment

Physical Server Hypervisor Physical CPU Physical Memory Physical Storage
Dell R420 ProxMox Virtual Environment 6.2-4 8 32G 2TB RAID 5
Dell R420 ProxMox Virtual Environment 6.2-4 24 32G 4TB RAID 5

Host Machines

Hostname Operating System vCPU Memory Storage
kubernetes-controller-1 Ubuntu 20.04 4 8G 100G
kubernetes-worker-1 Ubuntu 20.04 2 4G 32G
kubernetes-worker-2 Ubuntu 20.04 2 4G 32G
kubernetes-worker-3 Ubuntu 20.04 2 4G 32G
nfs-server-1 CentOS 7 4 4G 250G

Pre-requisites

  • A Kubernetes cluster running v1.19.4.
  • This lab has 3 worker nodes with 4GiB memory and 2 vCPU's.
  • Access to the Kubernetes cluster.
  • Kubectl already configured for use.
  • An already configured NFS server ready to use for persistent storage.

Installation

It is suggested that you go through the first lab before executing this one. This lab requires that you have an NFS server to mount your persistent volume to.

TLS Key Creation

Before we begin the lab lets create a selfsigned TLS certificate for our Grafana server. This will allow encryption in transit from the client to the server.

$ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout privateKey.key -out certificate.crt
 ```

Upon completion of the command you will be left with two files, `privateKey.key` and `certificate.crt`. We will reference these later on in the Grafana server creation.

#### Grafana Namespace

First we must create a namespace for our Grafana server to live. Create a file called `grafana-ns.yaml`.

```yaml
---
apiVersion: v1
kind: Namespace
metadata:
  name: grafana

Then create the namespace.

$ kubectl create -f grafana-ns.yaml
namespace/grafana created

You can confirm it was created by listing the namespaces.

```bash
$ kubectl get namespaces
NAME              STATUS   AGE
default           Active   13d
grafana           Active   28s
kube-node-lease   Active   13d
kube-public       Active   13d
kube-system       Active   13d

Grafana Secrets

Before we begin lets create two secrets that we will use later on in this configuration. Create a file called grafana-secrets.yaml. In this file paste the following.

---
apiVersion: v1
metadata:
  name: grafana-tls
  namespace: grafana
kind: Secret
type: kubernetes.io/tls
data:
  tls.crt: < YOUR BASE64 ENCODED CERT >
  tls.key: < YOUR BASE64 ENCODED KEY >

As you may have noted the we must add the base64 encoded values of our certificate and key.

To get the base64 encoded values for both of these perform the following against the files we created earlier.

$ cat privateKey.key |base64
$ cat certificate.crt |base64

Copy the output of the .crt and paste it into < YOUR BASE64 ENCODED CERT >. Copy the output of the .key and paste it into < YOUR BASE64 ENCODED KEY >.

Now create the secrets by running the following.

$kubectl create -f grafana-secrets.yaml

Persistent Volume

Next we will add a persistent volume for our Grafana server to use. This will allow us to keep the data even if the pod is destroyed.

Like in the NFS Lab 1 we will create a new directory on the nfsshare called pv0006. This is where all the Grafana data will be stored.

Create a file called grafana-pv.yaml and add the following, changing the NFS server address to your NFS server.

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv0006-grafana
  namespace: grafana
spec:
  capacity:
    storage: 20Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  storageClassName: pv0006-grafana
  mountOptions:
    - hard
    - nfsvers=4.1
  nfs:
    path: /nfsshare/pv0006/grafana
    server: 192.168.1.195

Create the persistent volume.

$ kubectl create -f grafana-pv.yaml
persistentvolume/pv0006-grafana created

You can confirm that the persistent volume exists by running the following command.

$ kubectl  get pv
NAME             CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS     REASON   AGE
pv0006-grafana   20Gi       RWO            Recycle          Available           pv0006-grafana            35s

Persistent Volume Claim

Now lets create the persistent volume claim for the Grafana volume that we have just created. Create a file called grafana-pvc.yaml and add the following.

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pv0006-grafana
  namespace: grafana
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: pv0006-grafana
  resources:
    requests:
      storage: 20Gi

Create the persistent volume claim.

$ kubectl create -f grafana-pvc.yaml
persistentvolumeclaim/pv0006-grafana created

You can validate that the persistent volume claim was created by running the following command.

$ kubectl get pvc -n grafana
NAME             STATUS   VOLUME           CAPACITY   ACCESS MODES   STORAGECLASS     AGE
pv0006-grafana  Bound    pv0006-grafana  20Gi       RWO            pv0006-grafana   61s

We can see that our persistent volume claim is bound.

Grafana Configmap

For our grafana server there is a configuration file called grafana.ini. We have made a few changes to this file and would like to be able to call it within our deployment. The configuration file is not necessarily sensitive information, and is not worth of mounting an entire data directory to use it. For that reason we will use a configmap.

One of the easiest ways to create a configmap is using the --from-file command. Try this against the grafana.ini by running the command below.

$ kubectl -n grafana create configmap grafana-config --from-file=./grafana.ini

You will note that it created the configmap from the configuration file we have specified. To see the YAML you can simply run the following command.

$ kubectl -n grafana get configmap grafana-config -o yaml

This will output a YAML file that you can save and apply in the future.

Grafana Deployment

Once we have the persistent volume and claim created we can now deploy the container.

What we are doing is configuring a container with a single replica with a label grafana.

This is going to be deployed to the grafana namespace that we first created. We are running the container on port 3000. The important component here for persistent data between deletion is the volumeMounts.

You may note that we have defined our volume mount as grafana-pv-storage which it gets from the volumes section. The volumes section declares this as the persistent volume claim we created earlier called pv0006-grafana.

This volume is mounted inside the container to /var/lib/grafana.

There is a secret with two entries that we are calling. This secret the grafana-tls secret we created earlier, first we are calling the tls.crt entry and setting the path to certificate.csr and secondly we call the tls.key entry and set it to private_key.key in the container. The certificate is mounted to the directory /etc/ssl/grafana/crt/ and the key is mounted to /etc/ssl/grafana/key/ within the container.

Lastl

Create a file called grafana-deployment.yaml and add the following.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
  namespace: grafana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
        namespace: grafana
    spec:
      containers:
      - name: grafana
        env:
        - name: GF_INSTALL_PLUGINS
          value: "grafana-clock-panel,grafana-simple-json-datasource"
        - name: GF_PATHS_DATA
          value: "/var/lib/grafana"
        image: grafana/grafana:7.3.6-ubuntu
        ports:
          - name: https-port
            containerPort: 3000
        volumeMounts:
          - name: grafana-pv-storage
            mountPath: /var/lib/grafana
          - name: grafana-config
            mountPath: /etc/grafana/
          - name: grafana-cert
            mountPath: /etc/ssl/grafana/crt/
          - name: grafana-key
            mountPath: /etc/ssl/grafana/key/
      volumes:
        - name: grafana-pv-storage
          persistentVolumeClaim:
            claimName: pv0006-grafana
        - name: grafana-config
          configMap:
            name: grafana-config
        - name: grafana-cert
          secret:
            secretName: grafana-tls
            items:
              - key: tls.crt
                path: certificate.csr
        - name: grafana-key
          secret:
            secretName: grafana-tls
            items:
              - key: tls.key
                path: private_key.key

Deploy the grafana container.

$ kubectl create -f grafana-deployment.yaml
deployment.apps/grafana created

You can confirm that it has been created by running the following commands.

$ kubectl -n grafana get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP           NODE                  NOMINATED NODE   READINESS GATES
grafana-6ff8d69b-7rl9d   1/1     Running   0          56s   10.244.1.9   kubernetes-worker-1   <none>           <none>

grafana Service

Here we are creating a service to expose the container port that we declared in the deployment. We declare the port that the service is running on in the container 3000 and what we would like it to be broadcast as on the hosts nodePort which is port 31443.

Create a file called grafana-svc.yaml and add the following.

---
apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: grafana
spec:
  type: NodePort
  ports:
    - port: 3000
      targetPort: 3000
      nodePort: 31443
  selector:
    app: grafana

Create the service by running the following commands.

$ kubectl create -f grafana-svc.yaml
service/grafana created

You can confirm the services exist by running the following commands.

$ kubectl -n grafana get services
NAME      TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
grafana   NodePort   10.97.137.186   <none>        3000:31443/TCP   31m

Accessing The Grafana Server

You can now access the Grafana server by connecting to port 31443 of the IP address of the worker node that the container lives.

To get the node that the container lives on run the following command.

$ kubectl -n grafana get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE     IP           NODE                  NOMINATED NODE   READINESS GATES
grafana-6ff8d69b-7rl9d   1/1     Running   0          5m14s   10.244.1.9   kubernetes-worker-1   <none>           <none>

We can see that the node name is kubernetes-worker-1.

Now run the following command to get details about the node.

$ kubectl get node kubernetes-worker-1 -o wide
NAME                  STATUS   ROLES    AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE           KERNEL-VERSION     CONTAINER-RUNTIME
kubernetes-worker-1   Ready    <none>   13d   v1.19.3   192.168.1.200   <none>        Ubuntu 20.04 LTS   5.4.0-53-generic   docker://19.3.8

Open your web browser and navigate to the address and port 31443. For example https://192.168.1.200:31443.

You will be prompted to log in for the first time which can be done by using the default credentials admin and admin. You will be prompted to change your password.

Back to top