A Persistent Grafana Server with TLS Encryption
This project will walk through how to deploy a Grafana server that is persistent with TLS encryption.
Note this lab is not for production use.
Below you will find the specifications for the environment used to run this lab. I am confident the lab is able to run on much less hardware or even on a set of Raspberry Pi's.
|Physical Server||Hypervisor||Physical CPU||Physical Memory||Physical Storage|
|Dell R420||ProxMox Virtual Environment 6.2-4||8||32G||2TB RAID 5|
|Dell R420||ProxMox Virtual Environment 6.2-4||24||32G||4TB RAID 5|
- A Kubernetes cluster running v1.19.4.
- This lab has 3 worker nodes with 4GiB memory and 2 vCPU's.
- Access to the Kubernetes cluster.
- Kubectl already configured for use.
- An already configured NFS server ready to use for persistent storage.
It is suggested that you go through the first lab before executing this one. This lab requires that you have an NFS server to mount your persistent volume to.
TLS Key Creation
Before we begin the lab lets create a selfsigned TLS certificate for our Grafana server. This will allow encryption in transit from the client to the server.
$ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout privateKey.key -out certificate.crt ``` Upon completion of the command you will be left with two files, `privateKey.key` and `certificate.crt`. We will reference these later on in the Grafana server creation. #### Grafana Namespace First we must create a namespace for our Grafana server to live. Create a file called `grafana-ns.yaml`. ```yaml --- apiVersion: v1 kind: Namespace metadata: name: grafana
Then create the namespace.
$ kubectl create -f grafana-ns.yaml namespace/grafana created You can confirm it was created by listing the namespaces. ```bash $ kubectl get namespaces NAME STATUS AGE default Active 13d grafana Active 28s kube-node-lease Active 13d kube-public Active 13d kube-system Active 13d
Before we begin lets create two secrets that we will use later on in this configuration. Create a file called
grafana-secrets.yaml. In this file paste the following.
--- apiVersion: v1 metadata: name: grafana-tls namespace: grafana kind: Secret type: kubernetes.io/tls data: tls.crt: < YOUR BASE64 ENCODED CERT > tls.key: < YOUR BASE64 ENCODED KEY >
As you may have noted the we must add the base64 encoded values of our certificate and key.
To get the base64 encoded values for both of these perform the following against the files we created earlier.
$ cat privateKey.key |base64 $ cat certificate.crt |base64
Copy the output of the
.crt and paste it into
< YOUR BASE64 ENCODED CERT >.
Copy the output of the
.key and paste it into
< YOUR BASE64 ENCODED KEY >.
Now create the secrets by running the following.
$kubectl create -f grafana-secrets.yaml
Next we will add a persistent volume for our Grafana server to use. This will allow us to keep the data even if the pod is destroyed.
Like in the NFS Lab 1 we will create a new directory on the nfsshare called
pv0006. This is where all the Grafana data will be stored.
Create a file called
grafana-pv.yaml and add the following, changing the NFS server address to your NFS server.
--- apiVersion: v1 kind: PersistentVolume metadata: name: pv0006-grafana namespace: grafana spec: capacity: storage: 20Gi volumeMode: Filesystem accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Recycle storageClassName: pv0006-grafana mountOptions: - hard - nfsvers=4.1 nfs: path: /nfsshare/pv0006/grafana server: 192.168.1.195
Create the persistent volume.
$ kubectl create -f grafana-pv.yaml persistentvolume/pv0006-grafana created
You can confirm that the persistent volume exists by running the following command.
$ kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pv0006-grafana 20Gi RWO Recycle Available pv0006-grafana 35s
Persistent Volume Claim
Now lets create the persistent volume claim for the Grafana volume that we have just created. Create a file called
grafana-pvc.yaml and add the following.
--- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pv0006-grafana namespace: grafana spec: accessModes: - ReadWriteOnce storageClassName: pv0006-grafana resources: requests: storage: 20Gi
Create the persistent volume claim.
$ kubectl create -f grafana-pvc.yaml persistentvolumeclaim/pv0006-grafana created
You can validate that the persistent volume claim was created by running the following command.
$ kubectl get pvc -n grafana NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE pv0006-grafana Bound pv0006-grafana 20Gi RWO pv0006-grafana 61s
We can see that our persistent volume claim is bound.
For our grafana server there is a configuration file called
grafana.ini. We have made a few changes to this file and would like to be able to call it within our deployment. The configuration file is not necessarily sensitive information, and is not worth of mounting an entire data directory to use it. For that reason we will use a
One of the easiest ways to create a configmap is using the
--from-file command. Try this against the
grafana.ini by running the command below.
$ kubectl -n grafana create configmap grafana-config --from-file=./grafana.ini
You will note that it created the configmap from the configuration file we have specified. To see the
YAML you can simply run the following command.
$ kubectl -n grafana get configmap grafana-config -o yaml
This will output a YAML file that you can save and apply in the future.
Once we have the persistent volume and claim created we can now deploy the container.
What we are doing is configuring a container with a single replica with a label
This is going to be deployed to the
grafana namespace that we first created. We are running the container on port
3000. The important component here for persistent data between deletion is the
You may note that we have defined our volume mount as
grafana-pv-storage which it gets from the
volumes section. The
volumes section declares this as the persistent volume claim we created earlier called
This volume is mounted inside the container to
There is a secret with two entries that we are calling. This secret the
grafana-tls secret we created earlier, first we are calling the
tls.crt entry and setting the path to
certificate.csr and secondly we call the
tls.key entry and set it to
private_key.key in the container. The certificate is mounted to the directory
/etc/ssl/grafana/crt/ and the key is mounted to
/etc/ssl/grafana/key/ within the container.
Create a file called
grafana-deployment.yaml and add the following.
--- apiVersion: apps/v1 kind: Deployment metadata: name: grafana namespace: grafana spec: replicas: 1 selector: matchLabels: app: grafana template: metadata: labels: app: grafana namespace: grafana spec: containers: - name: grafana env: - name: GF_INSTALL_PLUGINS value: "grafana-clock-panel,grafana-simple-json-datasource" - name: GF_PATHS_DATA value: "/var/lib/grafana" image: grafana/grafana:7.3.6-ubuntu ports: - name: https-port containerPort: 3000 volumeMounts: - name: grafana-pv-storage mountPath: /var/lib/grafana - name: grafana-config mountPath: /etc/grafana/ - name: grafana-cert mountPath: /etc/ssl/grafana/crt/ - name: grafana-key mountPath: /etc/ssl/grafana/key/ volumes: - name: grafana-pv-storage persistentVolumeClaim: claimName: pv0006-grafana - name: grafana-config configMap: name: grafana-config - name: grafana-cert secret: secretName: grafana-tls items: - key: tls.crt path: certificate.csr - name: grafana-key secret: secretName: grafana-tls items: - key: tls.key path: private_key.key
Deploy the grafana container.
$ kubectl create -f grafana-deployment.yaml deployment.apps/grafana created
You can confirm that it has been created by running the following commands.
$ kubectl -n grafana get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES grafana-6ff8d69b-7rl9d 1/1 Running 0 56s 10.244.1.9 kubernetes-worker-1 <none> <none>
Here we are creating a service to expose the container port that we declared in the deployment. We declare the port that the service is running on in the container
3000 and what we would like it to be broadcast as on the hosts
nodePort which is port
Create a file called
grafana-svc.yaml and add the following.
--- apiVersion: v1 kind: Service metadata: name: grafana namespace: grafana spec: type: NodePort ports: - port: 3000 targetPort: 3000 nodePort: 31443 selector: app: grafana
Create the service by running the following commands.
$ kubectl create -f grafana-svc.yaml service/grafana created
You can confirm the services exist by running the following commands.
$ kubectl -n grafana get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE grafana NodePort 10.97.137.186 <none> 3000:31443/TCP 31m
Accessing The Grafana Server
You can now access the Grafana server by connecting to port
31443 of the IP address of the worker node that the container lives.
To get the node that the container lives on run the following command.
$ kubectl -n grafana get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES grafana-6ff8d69b-7rl9d 1/1 Running 0 5m14s 10.244.1.9 kubernetes-worker-1 <none> <none>
We can see that the node name is
Now run the following command to get details about the node.
$ kubectl get node kubernetes-worker-1 -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME kubernetes-worker-1 Ready <none> 13d v1.19.3 192.168.1.200 <none> Ubuntu 20.04 LTS 5.4.0-53-generic docker://19.3.8
Open your web browser and navigate to the address and port
31443. For example
You will be prompted to log in for the first time which can be done by using the default credentials
admin. You will be prompted to change your password.