Logo

Personal Ops Runbook

Personal runbook covering infrastructure operations for Cloud, Kubernetes, OpenStack, and Ceph environments. Includes deployment and teardown procedures, node management, cluster monitoring setup, and incident response workflows compiled from day-to-day operational work. Intended strictly for personal reference — configurations and scripts are environment-specific and not guaranteed to work as-is elsewhere.

Documentation for Kubernetes Cluster Setup

This document provides a step-by-step guide to set up a Kubernetes cluster on Ubuntu nodes, including the installation of necessary packages, configuration of container runtime, and setup of Kubernetes components. Additionally, it covers the setup of load balancing using Keepalived and HAProxy, and the deployment of monitoring tools like Prometheus and Grafana.

Requement.

  • OS version: Ubuntu 22.04.4 LTS
  • Proxy: 10.237.7.250:3128
  • Kube version: kubelet=1.28.8-1.1, kubectl=1.28.8-1.1, kubeadm=1.28.8-1.1
  • HAProxy version: 2.4.24
  • Keepalived verison: v2.2.4
  • Metallb version: v0.13.5
  • Calico versionv3.29.2
  • Kube Prometheus stack: chart version 34.9.0, app version 0.55.0
  • Helm version: chart version 2.0.1, app version 4.0.1
  • NGINX Ingress Controller 4.0.1

Summary steps:

  1. Update and Install Dependencies: - Update package lists and install required packages such as curl, gnupg2, software-properties-common, apt-transport-https, and ca-certificates. - Disable and stop the firewall (ufw).

  2. Install and Configure Containerd: - Install containerd and configure it to use systemd as the cgroup driver. - Set up HTTP proxy for containerd if required.

  3. Install Kubernetes Components: - Add Kubernetes apt repository and install specific versions of kubeadm, kubelet, and kubectl. - Enable and configure kubelet.

  4. Kernel and System Configuration: - Configure kernel parameters for Kubernetes networking. - Disable swap.

  5. Load Balancer Setup: - Install and configure Keepalived for high availability. - Install and configure HAProxy for load balancing Kubernetes API server and worker nodes.

  6. Initialize Kubernetes Cluster: - Initialize the Kubernetes control plane using kubeadm. - Set up kubeconfig for kubectl access. - Join worker nodes to the cluster.

  7. Network Plugin Installation: - Install Calico as the network plugin for the cluster.

  8. Install Helm: - Download and install Helm for managing Kubernetes applications.

  9. Deploy NGINX Ingress Controller: - Install NGINX Ingress Controller using Helm and configure it to use host ports.

  10. Monitoring Setup: - Install Prometheus and Grafana using Helm for monitoring the cluster. - Configure MetalLB for load balancing services.

  11. Prometheus Deployment: - Deploy Prometheus using a custom YAML configuration. - Configure the service type for Prometheus to LoadBalancer or NodePort as needed.

This guide ensures a comprehensive setup of a Kubernetes cluster with high availability, load balancing, and monitoring capabilities.

Detail steps:

Update package lists on all nodes.

apt-get update

Install necessary packages:

apt install -y curl gnupg2 software-properties-common apt-transport-https ca-certificates

isable UFW (Uncomplicated Firewall):

ufw disable
systemctl stop ufw
systemctl disable ufw

Install Docker and containerd:

These commands install Docker's GPG key, add Docker's repository, update the package list, and install containerd, a container runtime.

apt-get install ca-certificates curl gnupg lsb-release -y
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list
apt-get update
apt-get install containerd.io -y
containerd  --version

Add Kubernetes repository and install Kubernetes components:

These commands add the Kubernetes repository, update the package list, and install specific versions of kubelet, kubectl, and kubeadm.

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /' | tee /etc/apt/sources.list.d/kubernetes.list
apt update
apt list -a kubeadm
apt-get install -y kubelet=1.28.8-1.1 kubectl=1.28.8-1.1 kubeadm=1.28.8-1.1
systemctl enable  kubelet

Configure sysctl for Kubernetes

This creates a sysctl configuration file to enable packet forwarding and bridge network traffic.

tee /etc/sysctl.d/kubernetes.conf<<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

Increase connection tracking table size:

This increases the maximum number of tracked connections.

echo "net.netfilter.nf_conntrack_max=1000000" >> /etc/sysctl.conf

Load necessary kernel modules:

These commands load the kernel modules and apply the sysctl settings.

tee /etc/modules-load.d/containerd.conf <<EOF
overlay
br_netfilter
EOF

Load the modules immediately and apply sysctl settings:

These commands load the kernel modules and apply the sysctl settings.

modprobe overlay
modprobe br_netfilter
sysctl --system

Disable swap:

These commands disable swap, which is necessary for Kubernetes to function properly.

sed -i "/ swap / s/^\(.*\)$/#\1/g" /etc/fstab || sed -i '/swap/d' /etc/fstab
swapoff -a

Configure containerd:

These commands create the containerd configuration directory, generate a default configuration file, and modify it to use systemd as the cgroup driver.

mkdir -p /etc/containerd 
containerd config default > /etc/containerd/config.toml
sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml

Set up HTTP proxy for containerd:

These commands create a systemd drop-in configuration file to set HTTP proxy environment variables for containerd.

mkdir -p /etc/systemd/system/containerd.service.d
cat > /etc/systemd/system/containerd.service.d/http-proxy.conf << 'OEF'
[Service]
Environment="HTTP_PROXY=http://10.237.7.250:3128"
Environment="HTTPS_PROXY=http://10.237.7.250:3128"
Environment="NO_PROXY=10.237.7.0/24,127.0.0.1,localhost,10.96.0.0/12,10.244.0.0/16"
OEF

These commands reload systemd configurations, enable containerd to start on boot, restart containerd, check its status, and display its environment variables.

systemctl daemon-reload
systemctl enable containerd
systemctl restart containerd
systemctl is-active containerd
systemctl show --property=Environment containerd

Pull Kubernetes images:

Pull all images

kubeadm config images pull

Install Keepalived

apt-get install linux-headers-$(uname -r)
apt-get update
apt-get install keepalived -y

Create /etc/keepalived/keepalived.conf

cat >  /etc/keepalived/keepalived.conf << 'OEF'
vrrp_instance apiserver {
        interface ens160
        virtual_router_id 101
        virtual_ipaddress {
            10.237.7.79
        }
    }
OEF

Restart Keepalived

systemctl restart keepalived
systemctl enable keepalived
systemctl status keepalived

Install HAProxy

apt-get update
apt-get install haproxy -y

Create /etc/haproxy/haproxy.cfg

cat > /etc/haproxy/haproxy.cfg << 'OEF'
frontend stats
    bind *:8080
    mode http
    stats enable
    stats uri /stats
    stats refresh 10s
    stats admin if LOCALHOST

frontend fe-apiserver
    bind 0.0.0.0:6443
    mode tcp
    option tcplog
    default_backend be-apiserver

backend be-apiserver
    mode tcp
    option tcplog
    option tcp-check
    balance roundrobin
    default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
    server k8s-master1 10.237.7.71:6443 check
    server k8s-master2 10.237.7.72:6443 check
    server k8s-master3 10.237.7.73:6443 check

frontend http_frontend
    bind *:80
    mode tcp
    option tcplog
    default_backend http_backend

backend http_backend
    mode tcp
    balance roundrobin
    server k8s-worker1 10.237.7.74:30100 check
    server k8s-worker2 10.237.7.75:30100 check
    server k8s-worker3 10.237.7.76:30100 check

frontend https_frontend
    bind *:443
    mode tcp
    option tcplog
    default_backend https_backend

backend https_backend
    mode tcp
    balance roundrobin
    server k8s-worker1 10.237.7.74:30101 check
    server k8s-worker2 10.237.7.75:30101 check
    server k8s-worker3 10.237.7.76:30101 check
OEF

Restart HAProxy

systemctl enable haproxy
systemctl restart haproxy
systemctl status haproxy

Set hostname all nodes

hostnamectl set-hostname k8s-master1
hostnamectl set-hostname k8s-master2
hostnamectl set-hostname k8s-master3
hostnamectl set-hostname k8s-worker1
hostnamectl set-hostname k8s-worker2
hostnamectl set-hostname k8s-worker3

Set hostfile

cat >> /etc/hosts << 'OEF'
10.237.7.71 k8s-master1
10.237.7.72 k8s-master2
10.237.7.73 k8s-master3
10.237.7.74 k8s-worker1
10.237.7.75 k8s-worker2
10.237.7.76 k8s-worker3
OEF

Init cluster

kubeadm init --control-plane-endpoint "10.237.7.79:6443" --upload-certs --pod-network-cidr="10.244.0.0/16" --v=5

Create env

mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

Create master token

shell> kubeadm token create --print-join-command --certificate-key $(kubeadm init phase upload-certs --upload-certs | tail -1)
kubeadm join 10.237.7.79:6443 --token rxvhse.c171fivq1209642u --discovery-token-ca-cert-hash sha256:80f472a531bd08542cee645e8ad8fdf95b5f54e7663fbaad606d73a15cd35d5a --control-plane --certificate-key 7dae022de6eac3f6fa3256748e0058d4a4f523667969aa060784d8b7db14537d

Create workder token

shell> kubeadm token create --print-join-command
kubeadm join 10.237.7.79:6443 --token 5v78oh.e3zm61i2d2dk2bbl --discovery-token-ca-cert-hash sha256:80f472a531bd08542cee645e8ad8fdf95b5f54e7663fbaad606d73a15cd35d5a

Deploy Calico network

kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.29.2/manifests/calico.yaml
wget https://raw.githubusercontent.com/projectcalico/calico/v3.29.2/manifests/calico.yaml
sed -i 's/# - name: CALICO_IPV4POOL_CIDR/- name: CALICO_IPV4POOL_CIDR/' calico.yaml
sed -i 's|#   value:.*|  value: "10.244.0.0/16"|g' calico.yaml
kubectl apply -f calico.yaml
kubectl get pods -A -o wide
kubectl get node -A -o wide

Heml install

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh && ./get_helm.sh
helm version

Make nginx-ingress namespace

kubectl create namespace nginx-ingress
helm repo add nginx-stable https://helm.nginx.com/stable
helm install ingress nginx-stable/nginx-ingress --namespace nginx-ingress
git clone https://github.com/nginxinc/kubernetes-ingress.git

Set proxy

export http_proxy=http://10.237.7.250:3128
export https_proxy=http://10.237.7.250:3128
export no_proxy=localhost,127.0.0.1,10.237.7.0/24

Verify

helm list --namespace nginx-ingress
helm status ingress --namespace nginx-ingress

Make custom-values.yaml

cat > custom-values.yaml << 'OEF'
controller:
  kind: daemonset
  daemonset:
    useHostPort: true
    hostPorts:
      http: 80
      https: 443
  service:
    type: NodePort
    nodePorts:
      http: 30100
      https: 30101
OEF

Upgrade nginx-ingress

helm upgrade ingress nginx-stable/nginx-ingress --namespace nginx-ingress -f ./custom-values.yaml

Verify

kubectl get po -n nginx-ingress
kubectl get ds -n nginx-ingress
kubectl get svc -n nginx-ingress

Patch config map

echo '{
  "data": {
    "use-proxy-protocol": "true",
    "proxy-connect-timeout": "10s",
    "proxy-read-timeout": "10s",
    "client-max-body-size": "2m",
    "external-status-address": "10.237.7.79"
  }
}' > ./patch-configmap.json

Apply new config map

kubectl patch configmap ingress-nginx-ingress -n nginx-ingress --patch "$(cat ./patch-configmap.json)"
kubectl -n nginx-ingress get cm
kubectl -n nginx-ingress get cm/ingress-nginx-ingress -o yaml
kubectl -n nginx-ingress patch svc ingress-nginx-ingress-controller --patch '{"spec": { "type": "NodePort", "ports": [ { "port": 80, "nodePort": 30100 }, { "port": 443, "nodePort": 30101 } ] } }'
kubectl get svc -n nginx-ingress

Create metric server

wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Apply metric server

vi ./components.yaml 
add line ```kubelet-insecure-tls``
kubectl apply -f ./components.yaml 

### PROMETHEUS ###
mkdir -p /home/prometheus
cd /home/prometheus

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add stable https://charts.helm.sh/stable
helm repo update
helm search repo prometheus |egrep "stack|CHART"
helm pull prometheus-community/kube-prometheus-stack --version 34.9.0
tar -xzf kube-prometheus-stack-34.9.0.tgz
cp kube-prometheus-stack/values.yaml values-prometheus.yaml
kubectl create ns kube-monitoring
helm -n kube-monitoring install prometheus-grafana-stack -f /home/prometheus/values-prometheus.yaml kube-prometheus-stack
helm -n kube-monitoring uninstall prometheus-grafana-stack -f /home/prometheus/values-prometheus.yaml kube-prometheus-stack
helm -n kube-monitoring upgrade prometheus-grafana-stack -f /home/prometheus/values-prometheus.yaml kube-prometheus-stack
kubectl -n kube-monitoring get all
kubectl get ingress -n kube-monitoring -owide

Metallb

Change strictARP: false -> strictARP: true

# see what changes would be made, returns nonzero returncode if different
kubectl get configmap kube-proxy -n kube-system -o yaml | \
sed -e "s/strictARP: false/strictARP: true/" | \
kubectl diff -f - -n kube-system

# actually apply the changes, returns nonzero returncode on errors only
kubectl get configmap kube-proxy -n kube-system -o yaml | \
sed -e "s/strictARP: false/strictARP: true/" | \
kubectl apply -f - -n kube-system

Apply MetaBL

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.5/config/manifests/metallb-native.yaml

Create IPAddressPool

cat << OEF | kubectl apply -f -
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: first-pool
  namespace: metallb-system
spec:
  addresses:
  - 10.237.7.80-10.237.7.90
OEF

cat << OEF | kubectl apply -f -
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: example
  namespace: metallb-system
spec:
  ipAddressPools:
  - first-pool
OEF

cat > ./prometheus_deployment.yaml << OEF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
spec:
  selector:
    matchLabels:
      app: monitor
  replicas: 1
  template:
    metadata:
      labels:
        app: monitor
    spec:
      containers:
        - name: prometheus
          image: prom/prometheus
          ports:
            - containerPort: 9090
          resources:
            limits:
              memory: "150M"
              cpu: "100m"
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus-service
spec:
  selector:
     app: monitor
  type: LoadBalancer
  ports:
    - name: http
      port: 9090
      targetPort: 9090
OEF

Apply

kubectl apply -f ./prometheus_deployment.yaml 
kubectl get svc/prometheus-service

Patch LoadBalancer -> NodePort

kubectl patch svc prometheus-service -p '{"spec": {"type": "NodePort"}}'
kubectl patch svc prometheus-service -p '{"spec": {"type": "NodePort", "ports": [{"port": 9090, "nodePort": 30090}]}}'

Rollback to LoadBalancer

kubectl patch svc prometheus-service -p '{"spec": {"type": "LoadBalancer"}}'

© Copyright Infra Notes contributors. Last updated on September 07, 2026.