Logo

Personal Ops Runbook

Personal runbook covering infrastructure operations for Cloud, Kubernetes, OpenStack, and Ceph environments. Includes deployment and teardown procedures, node management, cluster monitoring setup, and incident response workflows compiled from day-to-day operational work. Intended strictly for personal reference — configurations and scripts are environment-specific and not guaranteed to work as-is elsewhere.

BƯỚC 0 - Xác nhận kit có file monitoring-docker-images.tar:

ls -lh ~/lightos-offline-kit/03-controller/monitoring/monitoring-docker-images.tar 2>/dev/null

BƯỚC 1 — Xóa sạch monitoring cũ (container + volume + config + image)

# Remove old containers
docker rm -f lightbits-grafana lightbits-prometheus 2>/dev/null

# Remove old data volumes (metric cu cluster A/B)
docker volume rm grafana_data prometheus_data 2>/dev/null

# Remove old config/targets (tro cluster cu)
rm -rf /var/lib/monitoring-clustering/file_sd_configs/*

# Remove old images
docker rmi grafana/grafana:9.3.6 prom/prometheus:v2.42.0 2>/dev/null

echo -n "containers: "; docker ps -a | grep -icE 'grafana|prometheus'
echo -n "volumes: "; docker volume ls | grep -icE 'grafana|prometheus'
echo -n "images: "; docker images | grep -icE 'grafana|prometheus'

Kỳ vọng containers 0, volumes 0, images 0 — trắng hoàn toàn.

BƯỚC 2 — Load image từ KIT (test air-gap)

docker load -i ~/lightos-offline-kit/03-controller/monitoring/monitoring-docker-images.tar
docker images | grep -iE 'grafana|prometheus'

Kỳ vọng có 2 image — prom/prometheus:v2.42.0 + grafana/grafana:9.3.6.

BƯỚC 3 — Deploy stack + add cluster mới

# Deploy container moi
/var/lib/monitoring-images/deploy.sh deploy
sleep 5
docker ps | grep -iE 'grafana|prometheus'

# Add cluster moi 01/02/03
/var/lib/monitoring-images/deploy.sh add_cluster \
  -c oss-cluster \
  -i 10.237.94.101,10.237.94.102,10.237.94.103

# Fix permission targets (theo doc cua ban)
chmod 644 /var/lib/monitoring-clustering/file_sd_configs/*/*.yaml
cat /var/lib/monitoring-clustering/file_sd_configs/*/*.yaml

Lưu ý: deploy.sh deploy có thể tự tìm image tar trong /var/lib/monitoring-images/*.tar.gz để load. Nếu nó tự load được thì bước 2 thừa — nhưng cứ load từ kit trước để test kit, rồi deploy sẽ thấy image sẵn, không cần load lại.

BƯỚC 4 - Login Grafana

Login admin/foobar