Skip to content

Kubernetes Kustomize Tutorial

  • You can find the source code for this video in my GitHub Repo.

Intro

Unless you use Kubernetes for your personal projects or some kind of POC, you need to be able to customize Kubernetes configuration files such deployments, services, ingresses, etc., to a specific environment. For example, if you expose an application to the internet using ingress, you definitely need to update DNS for different environments. Due to the fact that there are so many projects out there, and some of them are widely used, such as Helm, the problem is still considered to be not completely solved.

One approach to configuration reuse is simply copy-paste. However, as with code, it makes it harder to benefit from ongoing improvements to the source.

Another approach to reuse is to express the source material as a parameterized template. A tool is processing the template executing any embedded scripting, and replacing parameters with desired values to generate the configuration. The challenge here is that the templates and value files are not the specification of Kubernetes API resources. They are a necessary new thing, a new language that wraps the Kubernetes API. The problem here with Helm is that different teams want different changes. So almost every specification that you include in the YAML file becomes a parameter that needs a value.

And finally, a template-free approach. Since the Kubernetes configuration is expressed as JSON or YAML, it's relatively easy to target individual keys and merge them with environment-specific values.

That's where kustomize comes into play. It's native to Kubernetes, a template-free way to customize Kubernetes objects. Kustomize has the concept of bases and overlays. A base is a directory with a kustomization.yaml file, which contains a set of Kubernetes resources such as deployments, configMaps, services, etc., and associated customization. Overlays directories contain environment-specific changes. It may be a name prefix, replica count, a new image tag, or other changes.

Kustomization uses an industry-standard declarative approach that can be described in YAML and stored in git. Since it's template free, it expresses the full power of Kubernetes API, with no need to parameterize every single line compared to Helm. It can be used for both bespoke (build in-house apps) and off-the-shelf applications such as Redis, Kafka, Postgres, and other open-source projects. Kustomize supports bulk management operations: creation, deletion, and updates. It is built into kubectl, so you can create and update resources using the kubectl apply -k command and similar command to delete. Since Kustomize is provided as a standalone golang package and cli tool as well as it is built into kubectl, it's easy to integrate with user's tools and workflows.

Convert Deployment to Kustomize Base

1-example/base/deployment.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: default
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
1-example/base/kustomization.yaml
1
2
3
4
5
6
7
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
metadata:
  name: arbitrary
resources:
- deployment.yaml
1-example/environments/staging/kustomization.yaml
1
2
3
4
---
namespace: staging
resources:
- ../../base

kubectl create namespace staging
kubectl apply -k environments/staging
1-example/environments/production/deployment.yaml
1
2
3
4
5
6
7
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 20
1-example/environments/production/kustomization.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
---
# Kustomize Feature List
# https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/#kustomize-feature-list

namespace: production
namePrefix: production-
resources:
- ../../base
- namespace.yaml
patchesStrategicMerge:
- deployment.yaml
1-example/environments/production/namespace.yaml
1
2
3
4
5
---
apiVersion: v1
kind: Namespace
metadata:
  name: production

kubectl apply -k environments/production
kubectl get pods -n production
kubectl delete -k environments/production

Kustomize configMapGenerator

2-example/base/deployment.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: default
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
        env:
        - name: ENV
          valueFrom:
            configMapKeyRef:
              name: config
              key: env
        volumeMounts:
        - name: credentials
          mountPath: /etc/config
      volumes:
        - name: credentials
          configMap:
            name: credentials
2-example/base/kustomization.yaml
1
2
3
---
resources:
- deployment.yaml
2-example/staging/kustomization.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
---
resources:
- ../base
configMapGenerator:
- name: config
  literals:
  - env=staging
- name: credentials
  files:
  - credentials=creds.txt
2-example/staging/creds.txt
1
us-west-1

kubectl apply -k staging
kubectl get pods
kubectl get cm
kubectl get cm config-<id> -o yaml
kubectl exec nginx-deployment-<id> -- cat /etc/config/credentials

Kustomize secret generator

3-example/base/deployment.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: default
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
        env:
        - name: USERNAME
          valueFrom:
            secretKeyRef:
              name: username
              key: username
        - name: REGION
          valueFrom:
            secretKeyRef:
              name: region
              key: region
        - name: PASWD
          valueFrom:
            secretKeyRef:
              name: password
              key: PASSWORD
3-example/base/kustomization.yaml
1
2
3
---
resources:
- deployment.yaml
3-example/production/kustomization.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
---
resources:
- ../base
secretGenerator:
- name: username
  literals:
  - username=admin
- name: region
  files:
  - region=region.txt
- name: password
  envs:
  - .env
3-example/production/.env
1
PASSWORD=123

kubectl apply -k production
kubectl get pods
kubectl exec nginx-deployment-<id> -- env

Kustomize image name or tag

4-example/base/deployment.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: default
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
4-example/base/kustomization.yaml
1
2
3
---
resources:
- deployment.yaml
4-example/staging/kustomization.yaml
1
2
3
4
5
6
7
---
resources:
- ../base
images:
- name: nginx
  # newName: my-nginx
  newTag: 1.21.6

kubectl apply -k staging
kubectl describe pods
Back to top