NGINX Ingress Controller for Kubernetes Tutorial

• You can find the source code for this video in my GitHub Repo.
• If you want to create EKS cluster using terraform, you can follow this tutorial.

Prerequisites

• Kubernetes
• Helm

Deploy Prometheus on Kubernetes Cluster

• Create following CRDs and apply them.
  • prometheus/0-crd/0-alertmanagerconfigs.yaml
  • prometheus/0-crd/1-alertmanagers.yaml
  • prometheus/0-crd/2-podmonitors.yaml
  • prometheus/0-crd/3-probes.yaml
  • prometheus/0-crd/4-prometheuses.yaml
  • prometheus/0-crd/5-prometheusrules.yaml
  • prometheus/0-crd/6-servicemonitors.yaml
  • prometheus/0-crd/7-thanosrulers.yaml

kubectl apply -f prometheus/0-crd

• Deploy Prometheus Operator.
  • prometheus/1-prometheus-operator/0-monitoring-ns.yaml
  • prometheus/1-prometheus-operator/1-crd-cluster-roles.yaml
  • prometheus/1-prometheus-operator/2-service-account.yaml
  • prometheus/1-prometheus-operator/3-cluster-role.yaml
  • prometheus/1-prometheus-operator/4-cluster-role-binding.yaml
  • prometheus/1-prometheus-operator/5-deployment.yaml
  • prometheus/1-prometheus-operator/5-deployment.yaml
  • prometheus/1-prometheus-operator/6-service.yaml
  • prometheus/1-prometheus-operator/7-service-monitor.yaml

kubectl apply -f prometheus/1-prometheus-operator

• Deploy Prometheus.
  • prometheus/2-prometheus/0-service-account.yaml
  • prometheus/2-prometheus/1-cluster-role.yaml
  • prometheus/2-prometheus/2-cluster-role-binding.yaml
  • prometheus/2-prometheus/3-prometheus.yaml

kubectl apply -f prometheus/2-prometheus

• Check Prometheus pods.

kubectl get pods -n monitoring

Deploy Nginx Ingress Controller (YAML & HELM)

• Add Nginx ingress Helm repo.

helm repo add ingress-nginx \
  https://kubernetes.github.io/ingress-nginx

• Update Helm repository.

helm repo update

• Search for nginx Helm Chart.

helm search repo nginx

• Create values.yaml to override default parameters

• Generate YAML from the Helm chart.

helm template my-ing ingress-nginx/ingress-nginx \
  --namespace ingress \
  --version 3.35.0 \
  --values values.yaml \
  --output-dir my-ing

• Deploy Nginx ingress with Helm.

helm install my-ing ingress-nginx/ingress-nginx \
  --namespace ingress \
  --version 3.35.0 \
  --values values.yaml \
  --create-namespace

• List Helm releases.

helm list -n ingress

• Get nginx pods.

kubectl get pods -n ingress

• Get Kubernetes services.

kubectl get svc -n ingress

Monitor Nginx Ingress with Prometheus

• Port forward Prometheus to localhost for now.

kubectl get svc -n monitoring
kubectl port-forward svc/prometheus-operated 9090 -n monitoring

• Add monitoring label to ingress namespace.

kubectl edit namespace ingress

monitoring: prometheus

Deploy Grafana on Kubernetes Cluster

• Generate admin password for Grafana.

echo -n "devops123" | base64

• Decode the password to base64 string.

echo "devops123" | base64 -d

• Deploy grafana.

kubectl apply -f grafana

• Get Kubernetes pods.

kubectl get pods -n monitoring

Import Nginx Ingress Controller Grafana Dashboard

• Port forward Grafana to localhost for now.

kubectl get svc -n monitoring
kubectl port-forward svc/grafana 3000 -n monitoring

• Login to Grafana, user: admin, password: devops123.

• Add Prometheus datasource.

http://prometheus-operated:9090

• To find Grafana dashboard google: nginx ingress grafana dashboard.

• Import 9614 dashboard

Create Ingress for Prometheus

• Get services.

kubectl get svc -n monitoring

• Create ingress.

kubectl apply -f example-1/prometheus.yaml

• Get ingresses.

kubectl get ing -n monitoring
kubectl get svc -n ingress

• Create CNAME record for prometheus.devopsbyexample.io

• Go to http://prometheus.devopsbyexample.io

Test Nginx Ingress Admission Webhook

• Add valid ngin directive annotation to Prometheus ingress.

annotations:
  nginx.ingress.kubernetes.io/configuration-snippet: |
    more_set_headers 'Foo: bar';

kubectl apply -f example-1/prometheus.yaml

• Make a mistake in the directive and apply.
• Render the full nginx config.

kubectl get pods -n ingress
kubectl exec <pod> -n ingress -- cat /etc/nginx/nginx.conf

• Search for more_set_headers

Create Ingress for Grafana

• Get services.

kubectl get svc -n monitoring

• Create ingress.

kubectl apply -f example-2/grafana.yaml

• Get ingresses.

kubectl get ing -n monitoring

• Create CNAME record for grafana.devopsbyexample.io

• Go to http://grafana.devopsbyexample.io

Simple Fanout Ingress Example

• Create app directory.

• Create example-3 directory.

• Apply example-3 folder.

kubectl apply -f example-3

• Get ingresses in staging namespace.

kubectl get ing -n staging

• Create CNAME record for api.

• Test with ingresses with cURL.

curl http://api.devopsbyexample.io/foo/ytafsd
curl http://api.devopsbyexample.io/bar/123

Virtual Hosting Ingress Example

• Create example-4 folder.
• Apply example-4.

kubectl apply -f example-4

• Get ingress.

kubectl get ing -n staging

• Create CNAME record for foo and bar.

• Test with cURL.

curl http://foo.devopsbyexample.io/hello
curl http://bar.devopsbyexample.io/blog

Nginx Ingress TLS Example

• Install cfssl using homebrew if you are on mac.

brew install cfssl

• Create config certs/0-config.json.
• Create CA certificate request certs/1-ca-csr.json.
• Change directory to certs and generate CA.

cfssl gencert -initca 1-ca-csr.json | cfssljson -bare ca
openssl x509 -in ca.pem -text -noout

• Create certificate request for foo-api.devopsbyexample.io domain certs/2-foo-api-csr.json.
• Generate certificate.

cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=0-config.json \
  -profile=demo \
  2-foo-api-csr.json | cfssljson -bare foo-api

• Open with OpenSSL.

openssl x509 -in foo-api.pem -text -noout

• Create Kubernetes secret example-5/7-tls-secret.yaml.

---
apiVersion: v1
kind: Secret
metadata:
  name: foo-api-devopsbyexample-io-tls
  namespace: staging
type: kubernetes.io/tls
data:
  tls.crt: base64
  tls.key: base64

• Encode tls certificate certs/foo-api.pem to base64.

echo -n "tls-cert-content" | base64

• Encode private key certs/foo-api.pem to base64.

echo -n "tls-private-key-content" | base64

• Create ingress example-5/8-tls-ingress.yaml.
• Switch directory and apply.

cd ..
kubectl apply -f example-5

• Get ingress

kubectl get ing -n staging

• Create CNAME record for foo-api.
• Go to https://foo-api.devopsbyexample.io
• Add CA to KeyChain.

Nginx Ingress Different Namespaces Example

• Create example-6 folder.
• Apply example-6.

kubectl apply -f example-6

• Use local service.
• Get pods in both namespaces.

kubectl get pods -n foo
kubectl get pods -n bar

• Get ingresses in foo namespace.

kubectl get ing -n foo

• Create CNAME for api-ns.
• Test using cURL.

curl http://api-ns.devopsbyexample.io/foo/asd
curl http://api-ns.devopsbyexample.io/bar/asd

• Create example-6/6-bar-external.yaml.
• Update example-6/5-ingress.yaml.
• Apply example-6.

kubectl apply -f example-6

• Test with cURL.

curl http://api-ns.devopsbyexample.io/bar/asd

Nginx Ingress TCP Example

• Create following files.

  • example-7/0-namespace.yaml
  • example-7/1-secrets.yaml
  • example-7/2-statefulset.yaml
  • example-7/3-service.yaml

• Apply example-7 folder and get pods in database namespace.

kubectl apply -f example-7

kubectl get pods -n database

• Create example-7/4-configmap.yaml.

kubectl apply -f example-7/4-configmap.yaml

- Add tcp configmap flag.

- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services

kubectl get svc my-ing-ingress-nginx-controller -n ingress
kubectl get deployment -n ingress
kubectl edit deployment -n ingress my-ing-ingress-nginx-controller
kubectl edit svc my-ing-ingress-nginx-controller -n ingress

- name: postgres
  port: 5444
  protocol: TCP

kubectl get svc my-ing-ingress-nginx-controller -n ingress

• Go to AWS open LB and security group.
• Create CNAME for postgres.

kubectl get svc my-ing-ingress-nginx-controller -n ingress

psql --host postgres.devopsbyexample.io \
  --port 5444 \
  --username postgres \
  --password

\l

Monitor Nginx Ingress with Grafana

• Open http://grafana.devopsbyexample.io.
• Create some traffic.

curl http://api.devopsbyexample.io/bar/123asd
curl http://bar.devopsbyexample.io/bar
curl http://foo-ns.devopsbyexample.io/bar
curl http://api-ns.devopsbyexample.io/foo/asdad

Links

• K8s Ingress
• NGINX Ingress
• NGINX inc version
• kubernetes/ingress-nginx
• Installation Guide
• Which Ingress Controller Do I Need?
• values.yaml
• Configuration options
• Exposing TCP and UDP services
• Admission web hook

Clean

• Remove Helm repos

  helm repo remove ingress-nginx
  helm repo remove bitnami
  

• remove ca from keychain

  brew remove cfssl