A Persistent Jenkins server deployed onto Kubernetes
This project will walk through how to deploy a Jenkins server that is persistent with TLS encryption.
Note this lab is not for production use.
Below you will find the specifications for the environment used to run this lab. I am confident the lab is able to run on much less hardware or even on a set of Raspberry Pi's.
|Physical Server||Hypervisor||Physical CPU||Physical Memory||Physical Storage|
|Dell R420||ProxMox Virtual Environment 6.2-4||8||32G||2TB RAID 5|
|Dell R420||ProxMox Virtual Environment 6.2-4||24||32G||4TB RAID 5|
- A Kubernetes cluster running v1.19.4.
- This lab has 3 worker nodes with 4GiB memory and 2 vCPU's.
- Access to the Kubernetes cluster.
- Kubectl already configured for use.
- An already configured NFS server ready to use for persistent storage.
It is suggested that you go through the first lab before executing this one. This lab requires that you have an NFS server to mount your persistent volume to.
TLS Key Creation
Before we begin the lab lets create a selfsigned TLS certificate for our Jenkins server. This will allow encryption in transit from the client to the server. To do this create a java keystore using the commands below replacing
<yourpassword> with a secure unique password.
keytool -genkey -keyalg RSA -alias selfsigned -keystore jenkins_keystore.jks -storepass <yourpassword> -keysize 2048 ``` Upon completion of the command you will be left with a file called `jenkins_keystore.jks`. This will be important later on when we add this file to Kubernetes as a secret. #### Jenkins Namespace First we must create a namespace for our Jenkins server to live. Create a file called `jenkins-ns.yaml`. ```yaml --- apiVersion: v1 kind: Namespace metadata: name: jenkins
Then create the namespace.
$ kubectl create -f jenkins-ns.yaml namespace/jenkins created
You can confirm it was created by listing the namespaces.
$ kubectl get namespaces NAME STATUS AGE default Active 13d jenkins Active 28s kube-node-lease Active 13d kube-public Active 13d kube-system Active 13d
Before we begin lets create two secrets that we will use later on in this configuration. Create a file called
jenkins-secrets.yaml. In this file paste the following.
--- apiVersion: v1 kind: Secret metadata: name: jenkins namespace: jenkins type: Opaque data: jenkins.jks: | <BASE 64 ENCODED .JKS> --- apiVersion: v1 kind: Secret metadata: name: jenkins-options namespace: jenkins type: Opaque data: jenkins-options: | <BASE 64 ENCODED STARTUP OPTIONS>
As you may have noted the we must add the base64 encoded values of our java key store and our java key store password.
To get the base64 encoded values for your
.jks run the following commands on the
.jks file we created earlier.
$ cat jenkins_keystore.jks |base64
Copy the output and pate it in the
<BASE 64 ENCODED .JKS> section of the secret file.
Now we must do the same for the password. However I am going to take the entire string of Jenkins startup options and pass it as a secret like so.
$ echo "--httpPort=-1 --httpsPort=8083 --httpsKeyStore=/var/lib/jenkins/pki/jenkins.jks --httpsKeyStorePassword=<Your Password Plaintext>" |base64
Copy the output and replace
<BASE 64 ENCODED STARTUP OPTIONS> in the
Next we will add a persistent volume for our Jenkins server to use. This will allow us to keep the data even if the pod is destroyed.
Like in the NFS Lab 1 we will create a new directory on the nfsshare called
pv0004. This is where all the Jenkins volume will be mounted.
Create a file called
jenkins-pv.yaml and add the following, changing the NFS server address to your NFS server.
--- apiVersion: v1 kind: PersistentVolume metadata: name: pv0004-jenkins namespace: jenkins spec: capacity: storage: 20Gi volumeMode: Filesystem accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Recycle storageClassName: pv0004-jenkins mountOptions: - hard - nfsvers=4.1 nfs: path: /nfsshare/pv0004 server: 192.168.1.195
Create the persistent volume.
$ kubectl create -f jenkins-pv.yaml persistentvolume/pv0004-jenkins created
You can confirm that the persistent volume exists by running the following command.
$ kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pv0004-jenkins 20Gi RWO Recycle Available pv0004-jenkins 35s
Persistent Volume Claim
Now lets create the persistent volume claim for the jenkins volume that we have just created. Create a file called
jenkins-pvc.yaml and add the following.
--- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pv0004-jenkins namespace: jenkins spec: accessModes: - ReadWriteOnce storageClassName: pv0004-jenkins resources: requests: storage: 20Gi
Create the persistent volume claim.
$ kubectl create -f jenkins-pvc.yaml persistentvolumeclaim/pv0004-jenkins created
You can validate that the persistent volume claim was created by running the following command.
$ kubectl get pvc -n jenkins NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE pv0004-jenkins Bound pv0004-jenkins 20Gi RWO pv0004-jenkins 61s
We can see that our persistent volume claim is bound.
Jenkins Service Account and Role
Now we will create the permissions for a service account that we will be using at a later date. Create a file called
jenkins-sa.yaml and add the following.
--- apiVersion: v1 kind: ServiceAccount metadata: name: jenkins namespace: jenkins --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "true" labels: kubernetes.io/bootstrapping: rbac-defaults name: jenkins namespace: jenkins rules: - apiGroups: - '*' resources: - statefulsets - services - replicationcontrollers - replicasets - podtemplates - podsecuritypolicies - pods - pods/log - pods/exec - podpreset - poddisruptionbudget - persistentvolumes - persistentvolumeclaims - jobs - endpoints - deployments - deployments/scale - daemonsets - cronjobs - configmaps - namespaces - events - secrets verbs: - create - get - watch - delete - list - patch - apply - update - apiGroups: - "" resources: - nodes verbs: - get - list - watch - update --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "true" labels: kubernetes.io/bootstrapping: rbac-defaults name: jenkins namespace: jenkins roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: jenkins subjects: - apiGroup: rbac.authorization.k8s.io kind: Group name: system:serviceaccounts:jenkins
Create the service account and role by running the following command.
$ kubectl create -f jenkins-sa.yaml serviceaccount/jenkins created clusterrole.rbac.authorization.k8s.io/jenkins created clusterrolebinding.rbac.authorization.k8s.io/jenkins created
You can confirm that the service account was created by running the following command.
$ kubectl -n jenkins get sa NAME SECRETS AGE default 1 8m18s jenkins 1 89s
Once we have the persistent volume and claim created we can now deploy the container.
What we are doing is configuring a container with a single replica with a label
This is going to be deployed to the
jenkins namespace that we first created. We are running the container on port
50000. The important component here for persistent data between deletion is the
You may note that we have defined our volume mount as
jenkins-pv-storage which it gets from the
volumes section. The
volumes section declares this as the persistent volume claim we created earlier called
This volume is mounted inside the container to
We also are mounting a secret
jenkins which is looking at secret
jenkins.jks. This is our Java Key Store that will be used for TLS in the Jenkins configuration. We are mounting this to a directory called
You may also note in the
env we are setting startup options from a secret. This secret
jenkins-options contains the startup options as well as the JKS password required to use the java key store.
Create a file called
jenkins-deployment.yaml and add the following.
--- apiVersion: apps/v1 kind: Deployment metadata: name: jenkins namespace: jenkins spec: replicas: 1 selector: matchLabels: app: jenkins template: metadata: labels: app: jenkins namespace: jenkins spec: containers: - name: jenkins env: - name: JENKINS_OPTS valueFrom: secretKeyRef: name: jenkins-options key: jenkins-options image: jenkins/jenkins:lts ports: - name: https-port containerPort: 8083 - name: jnlp-port containerPort: 50000 volumeMounts: - name: jenkins-pv-storage mountPath: /var/jenkins_home - name: jenkins-secret mountPath: /var/lib/jenkins/pki/ volumes: - name: jenkins-pv-storage persistentVolumeClaim: claimName: pv0004-jenkins - name: jenkins-secret secret: secretName: jenkins items: - key: jenkins.jks path: jenkins.jks
Deploy the jenkins container.
$ kubectl create -f jenkins-deployment.yaml deployment.apps/jenkins created
You can confirm that it has been created by running the following commands.
$ kubectl -n jenkins get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES jenkins-6ff8d69b-7rl9d 1/1 Running 0 56s 10.244.1.9 kubernetes-worker-1 <none> <none>
Here we are creating two services to expose the container ports that we declared in the deployment. We declare the port that the service is running on in the container
8083 and what we would like it to be broadcast as on the hosts
nodePort which is port
We are doing the same thing for the service
jenkins-jnlp however not exposing the port to the node.
Create a file called
jenkins-svc.yaml and add the following.
--- apiVersion: v1 kind: Service metadata: name: jenkins namespace: jenkins spec: type: NodePort ports: - port: 8083 targetPort: 8083 nodePort: 30000 selector: app: jenkins --- apiVersion: v1 kind: Service metadata: name: jenkins-jnlp namespace: jenkins spec: type: ClusterIP ports: - port: 50000 targetPort: 50000 selector: app: jenkins
Create the service by running the following commands.
$ kubectl create -f jenkins-svc.yaml service/jenkins created service/jenkins-jnlp created
You can confirm the services exist by running the following commands.
$ kubectl -n jenkins get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE jenkins NodePort 10.100.195.191 <none> 8080:30000/TCP 8s jenkins-jnlp ClusterIP 10.111.149.244 <none> 50000/TCP 8s
Accessing The Jenkins Server
You can now access the jenkins server by connecting to port
30000 of the IP address of the worker node that the container lives.
To get the node that the container lives on run the following command.
$ kubectl -n jenkins get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES jenkins-6ff8d69b-7rl9d 1/1 Running 0 5m14s 10.244.1.9 kubernetes-worker-1 <none> <none>
We can see that the node name is
Now run the following command to get details about the node.
$ kubectl get node kubernetes-worker-1 -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME kubernetes-worker-1 Ready <none> 13d v1.19.3 192.168.1.200 <none> Ubuntu 20.04 LTS 5.4.0-53-generic docker://19.3.8
Open your web browser and navigate to the address and port
30000. For example
You will be prompted with the following screen if it is your first time logging in.
We will need to connect to the container to get the initial administrator password. We can do this by running the following commands against your pod.
$ kubectl -n jenkins exec -it jenkins-6ff8d69b-7rl9d bash kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead. jenkins@jenkins-6ff8d69b-7rl9d:/$ cat /var/jenkins_home/secrets/initialAdminPassword
This will return a long string which is the initial administrator password. Paste this password into the browser screen.
Install Suggested Plugins.
It will install default plugins that are useful once completed it will ask you to create a username and password, enter the necessary fields and then click "Save and Continue".
Congratulations you are all configured ! You can verify that your persistent volume is operating as expected by performing the next section.
We will delete the Jenkins pod to verify that our volume is truly persistent.
First lets get the pod.
$ kubectl -n jenkins get pods NAME READY STATUS RESTARTS AGE jenkins-6ff8d69b-7rl9d 1/1 Running 0 19m
Now lets delete the pod.
$ kubectl -n jenkins delete pod jenkins-6ff8d69b-7rl9d pod "jenkins-6ff8d69b-7rl9d" deleted
The pod will restart but note that the pod name is different.
$ kubectl -n jenkins get pods NAME READY STATUS RESTARTS AGE jenkins-6ff8d69b-4bp2p 1/1 Running 0 25s
This means that this is a whole new container that has been deployed but the data is carried between the old pod and the new pod through the volume.
Validate that you are able to access the UI and login by navigating to the IP address of the node it lives on and the appropraite port
To uninstall execute the following commands.
$ kubectl delete -f jenkins-svc.yaml service "jenkins" deleted service "jenkins-jnlp" deleted $ kubectl delete -f jenkins-deployment.yaml deployment.apps "jenkins" deleted $ kubectl delete -f jenkins-sa.yaml serviceaccount "jenkins" deleted clusterrole.rbac.authorization.k8s.io "jenkins" deleted clusterrolebinding.rbac.authorization.k8s.io "jenkins" deleted $ kubectl delete -f jenkins-pvc.yaml persistentvolumeclaim "pv0004-jenkins" deleted $ kubectl delete -f jenkins-pv.yaml persistentvolume "pv0004-jenkins" deleted $ kubectl delete -f jenkins-secrets.yaml $ kubectl delete -f jenkins-ns.yaml namespace "jenkins" deleted