How to Manage Jenkins Configuration as Code on Kubernetes
Running Jenkins on Kubernetes is relatively straightforward, but maintaining its configuration across deployments can quickly become challenging. Manually configuring plugins, authentication, permissions, credentials, and pipelines through the Jenkins UI makes rebuilding or migrating an instance unnecessarily complicated.
To solve this problem, I implemented a Jenkins Configuration as Code (JCasC) approach, allowing me to deploy a fully configured Jenkins instance on Kubernetes using a custom Docker image, ConfigMaps, Kubernetes Secrets, and Job DSL.
The goal was simple: make Jenkins configuration reproducible, version-controlled, and easy to restore without manual intervention.
Building a custom Jenkins image with preinstalled plugins
The first step was to create a custom Jenkins Docker image based on the official Jenkins distribution with JDK 21.
Instead of installing plugins manually through the Jenkins UI, I embedded all required plugins directly into the image during the build process.
FROM docker.io/jenkins/jenkins:jdk21
USER root
ENV CASC_JENKINS_CONFIG=/var/jenkins/casc_config
RUN apt-get update && apt-get install -y \
lsb-release curl ca-certificates
RUN curl -fsSLo /usr/share/keyrings/docker-archive-keyring.asc \
https://download.docker.com/linux/debian/gpg
RUN echo "deb [arch=$(dpkg --print-architecture) \
signed-by=/usr/share/keyrings/docker-archive-keyring.asc] \
https://download.docker.com/linux/debian \
$(lsb_release -cs) stable" \
> /etc/apt/sources.list.d/docker.list
RUN apt-get update && apt-get install -y docker-ce-cli
USER jenkins
RUN jenkins-plugin-cli --plugins
"configuration-as-code job-dsl git workflow-job workflow-cps saml oic-auth matrix-auth gitea docker-plugin docker-commons docker-java-api"
The most important plugins in this setup are: * Configuration as Code: Automatically configures Jenkins using YAML files. * Job DSL: Defines and creates Jenkins jobs programmatically. * OpenID Connect Authentication: Integrates Jenkins with an external identity provider such as Keycloak. * Matrix Authorization Strategy: Controls permissions for users and groups. * Docker Plugin: Provides dynamically provisioned Jenkins build agents. I also installed the Docker CLI to support Docker-related operations. The Docker plugin itself communicates with a compatible Docker API endpoint.
By building plugins into the image, I can ensure that every deployment starts with a consistent set of dependencies. For stronger reproducibility, I would pin the Jenkins base image and plugin versions rather than relying on moving tags and latest compatible versions.
Configuring Jenkins with Configuration as Code
After preparing the image, I used the Configuration as Code plugin to manage Jenkins settings through YAML.
The environment variable:
CASC_JENKINS_CONFIG=/var/jenkins/casc_config
tells Jenkins where to find its configuration files.
Here is a simplified example based on my implementation:
jenkins:
numExecutors: 2
mode: NORMAL
authorizationStrategy:
globalMatrix:
entries:
- group:
name: "admin"
permissions:
- "Overall/Administer"
- group:
name: "authenticated"
permissions:
- "Overall/Read"
- "Job/Read"
- "Job/Build"
securityRealm:
oic:
clientId: "jenkins-oic-auth"
clientSecret: "${OIDC_CLIENT_SECRET}"
userNameField: "preferred_username"
groupsFieldName: "groups"
serverConfiguration:
manual:
issuer: "https://keycloak.example.com/realms/master"
scopes: "openid email profile"
clouds:
- docker:
name: "podman"
containerCap: 3
dockerApi:
dockerHost:
uri: "unix:///var/run/docker.sock"
templates:
- labelString: "alpine-jdk21"
name: "alpine-jdk21"
dockerTemplateBase:
image: "jenkins/inbound-agent:latest-alpine-jdk21"
remoteFs: "/home/jenkins/agent"
mode: EXCLUSIVE
connector:
jnlp:
jenkinsUrl: "https://jenkins.example.com/"
user: "1000"
jobs:
- script: >
pipelineJob('Terraform_AWS_account') {
definition {
cpsScm {
lightweight(true)
scm {
git {
remote {
url('ssh://git@git.example.com:7999/infra/project.git')
credentials('nutcracker-rsa-key')
}
branches('*/main')
}
}
scriptPath('path/to/Jenkinsfile')
}
}
}
In my setup, I integrated Jenkins with Keycloak using OpenID Connect, configured role-based permissions through matrix authorisation, and defined Docker-based build agents.
The agent configuration uses a Docker-compatible API endpoint, which can also be provided by Podman. The socket must be available to the Jenkins controller and accessible with appropriate permissions. On Kubernetes, I would avoid mounting a privileged host Docker socket where possible and prefer a dedicated agent provisioning mechanism.
Managing credentials securely with Kubernetes Secrets
One important consideration is handling sensitive information.
Although JCasC supports defining credentials directly in YAML, storing private keys or authentication secrets in a ConfigMap or Git repository is not a secure solution.
Instead, I can store sensitive data in Kubernetes Secrets and reference them from the JCasC configuration.
For example, the SSH credential definition becomes:
credentials:
system:
domainCredentials:
- credentials:
- basicSSHUserPrivateKey:
scope: GLOBAL
id: "nutcracker-rsa-key"
username: "rsa-key"
description: "rsa key for Git access"
privateKeySource:
directEntry:
privateKey: "${JENKINS_RSA_PRIVATE_KEY}"
The ${JENKINSRSAPRIVATE_KEY} expression is resolved by the Configuration as Code plugin from an environment variable.
Now can create the Kubernetes Secrets using an existing SSH private key and OIDC taken secret taken from Keycloak:
kubectl create secret generic jenkins-secrets --namespace jenkins --from-file=rsa-private-key=./id_rsa --from-literal=oidc-client-secret='YOUR_OIDC_SECRET'
The Job DSL configuration is included in the same JCasC YAML file:
jobs:
- script: >
pipelineJob('Terraform_AWS_account') {
definition {
cpsScm {
lightweight(true)
scm {
git {
remote {
url('ssh://git@git.example.com:7999/infra/project.git')
credentials('nutcracker-rsa-key')
}
branches('*/main')
}
}
scriptPath('path/to/Jenkinsfile')
}
}
}
This configuration automatically creates a pipeline named TerraformAWSaccount and connects it to a Git repository containing the Jenkinsfile.
The SSH credential is referenced by its Jenkins credential ID, so the private key does not need to appear in the Job DSL script.
This approach allows me to version-control not only Jenkins system settings but also the pipeline definitions themselves.
Finally, I defined a Kubernetes Deployment that mounts the ConfigMap into the path specified by CASCJENKINSCONFIG
apiVersion: apps/v1
kind: Deployment
metadata:
name: jenkins
namespace: jenkins
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: jenkins
template:
metadata:
labels:
app: jenkins
spec:
containers:
- name: jenkins
image: registry.example.com/jenkins:1.0.0
ports:
- containerPort: 8080
- containerPort: 50000
env:
- name: CASC_JENKINS_CONFIG
value: /var/jenkins/casc_config
- name: JENKINS_RSA_PRIVATE_KEY
valueFrom:
secretKeyRef:
name: jenkins-secrets
key: rsa-private-key
- name: OIDC_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: jenkins-secrets
key: oidc-client-secret
volumeMounts:
- name: jenkins-casc
mountPath: /var/jenkins/casc_config
readOnly: true
- name: jenkins-home
mountPath: /var/jenkins_home
volumes:
- name: jenkins-casc
configMap:
name: jenkins-casc
- name: jenkins-home
persistentVolumeClaim:
claimName: jenkins-home
In this deployment: * The custom image contains all required Jenkins plugins. * The ConfigMap provides the JCasC YAML configuration. * Kubernetes Secrets provide sensitive credentials through environment variables. * A PersistentVolumeClaim stores Jenkins runtime data, build history, and other persistent state. The PVC must be created separately. This example also assumes that Jenkins agents are configured independently or that the Docker-compatible endpoint defined in JCasC is available to the controller.
An important detail is that updating the ConfigMap does not automatically restart Jenkins or guarantee that the new configuration is applied. In my deployment process, I can trigger a rollout restart:
kubectl rollout restart deployment/jenkins -n jenkins
Jenkins then loads the configuration from the mounted YAML file during startup.
Benefits of this approach
By combining Kubernetes, a custom Jenkins image, JCasC, and Job DSL, I achieved a deployment model where most of the Jenkins configuration is defined declarativelyt.
The main benefits are reproducibility, simplified disaster recovery, version-controlled configuration, and reduced manual administration.
It also makes upgrading Jenkins easier: I can build a new image with updated plugins, validate the configuration in a test environment, and deploy the updated version through Kubernetes.
The most important improvement is that Jenkins is no longer configured primarily through its UI. Instead, its configuration becomes part of the infrastructure code, making changes easier to review, track, and reproduce.