Logo

Personal Ops Runbook

Personal runbook covering infrastructure operations for Cloud, Kubernetes, OpenStack, and Ceph environments. Includes deployment and teardown procedures, node management, cluster monitoring setup, and incident response workflows compiled from day-to-day operational work. Intended strictly for personal reference — configurations and scripts are environment-specific and not guaranteed to work as-is elsewhere.

9.10 — Diễn tập "node chết, đưa node dự phòng vào thay" (04 thế chỗ 05)

Kịch bản production thật: oss-nvme-05 chết vì journal (node Unattached, record SSD Failed, không có đường mềm) → đưa node dự phòng oss-nvme-04 (đã làm sạch) vào cụm → thử lệnh chính quy replace node → giải thể hoàn toàn 05.

Trạng thái xuất phát: - Cluster: 01/02/03 khỏe + server 35b83b77 (05) còn trong cụm, node 8861a8fc Unattached - etcd: 4 member started (01/02/03/05 — etcd của 05 vẫn sống, độc lập với tầng LightOS) - 04: đã ép sạch, chờ vào cụm

3 terminal: [ADMIN] vm-admin-01 · [NEW04] oss-nvme-04 · [REF] oss-nvme-01 · (+[OLD05] khi giải thể)

PHA A — Đưa 04 vào cụm (theo runbook v3 đã nghiệm thu, giá trị điền sẵn)

A0 — Định vị + kiểm xuất phát điểm

### [NEW04] — node sach? ky vong 0/8/0/0/0
echo -n "pkg: ";   rpm -qa | grep -icE "lightos|duros|etcd"
echo -n "disks: "; lsblk | grep -cE "nvme[0-7]n1"
echo -n "repos: "; ls /etc/yum.repos.d/*.repo 2>/dev/null | wc -l
echo -n "raid0: "; grep -E "^md[0-9]+ :" /proc/mdstat | grep -c raid0
echo -n "leftover: "; ls -d /var/lib/etcd /etc/lightos* /etc/node-manager /etc/cluster-manager /etc/duroslight /var/cache/nm_service /usr/lib/systemd/system/duroslight-*.service 2>/dev/null | wc -l
# Chua dat -> chay khoi EP SACH 5 nhip (Buoc 0 runbook v3-PerNode)

### [ADMIN] — cluster: 4 server (05 con xac), khong server00
lbcli --endpoint $ENDPOINT -J "$LIGHTOS_JWT" list servers
lbcli --endpoint $ENDPOINT -J "$LIGHTOS_JWT" list nodes      # 3 Active + 05 Unattached

### [REF] — member 4 started; registry DON RAC truoc khi them nguoi moi
etcdctl member list
etcdctl get --prefix /registry/discovery-service --keys-only
# Ky vong key: 88940976(03) a0d5bf2a(01) a17b6c72(02) 35b83b77(05-con-song-tam)
# NEU con a173b57d (server 04 DOI CU, da delete hom 02/08) -> xoa:
etcdctl del /registry/discovery-service/a173b57d-648b-4e52-9c14-a6c04dfd6c8b

A1 — Cài package + disable toàn bộ

### [ADMIN]
NEW=10.237.94.104
ssh root@$NEW 'rm -rf /tmp/rpms && mkdir -p /tmp/rpms'
scp ~/lightos-offline/target_rpms/*.rpm ~/lightos-offline/deps/*.rpm root@$NEW:/tmp/rpms/

### [NEW04]
yum install -y --nogpgcheck /tmp/rpms/createrepo_c-libs*.rpm /tmp/rpms/createrepo_c-[0-9]*.rpm /tmp/rpms/drpm*.rpm 2>/dev/null
createrepo /tmp/rpms
printf "[lightos]\nname=lightos\nbaseurl=file:///tmp/rpms\ngpgcheck=0\nenabled=1\n" > /etc/yum.repos.d/lightos.repo
dnf clean all
yum install -y --nogpgcheck etcd
yum install -y --nogpgcheck lightos --allowerasing
yum install -y --nogpgcheck tar net-tools yum-utils
for s in etcd node-manager cluster-manager api-service discovery-service upgrade-manager lightbox-exporter profile-generator irqbalance; do systemctl disable --now $s 2>/dev/null; done
reboot

### [NEW04] — sau reboot ~90s, CHECKPOINT 1.5
systemctl is-active etcd node-manager discovery-service cluster-manager   # toan inactive/failed
ps -C etcd,node-manager -o args=                                          # TRONG

### [ADMIN] — khong moc them server00 (van 4 server)
lbcli --endpoint $ENDPOINT -J "$LIGHTOS_JWT" list servers | tail -5
### [ADMIN] — 1.6 verify package
diff <(ssh root@10.237.94.101 'rpm -qa --qf "%{NAME}\n" | sort') \
     <(ssh root@$NEW 'rpm -qa --qf "%{NAME}\n" | sort') | grep '^<'       # khong lightos/duros/etcd
ssh root@$NEW 'uname -r'                                                   # = REF

A2 — Cert mới (CA cụm + clusterId + danh tính 04)

### [ADMIN]
cd ~/lightos-certificates-clusterNew
NEWHOST=oss-nvme-04; NEW_DATA_IP=10.237.95.104
CLUSTER_ID=$(ssh root@10.237.94.101 "grep clusterId /etc/cluster-manager/cluster-manager.yaml | awk '{print \$2}'")
echo "CLUSTER_ID=$CLUSTER_ID"    # f49c7acd-...
mkdir -p old-$NEWHOST && mv $NEWHOST-cert-etcd-peer*.pem old-$NEWHOST/ 2>/dev/null

cat > /tmp/$NEWHOST.cnf <<EOF
[req]
default_bits = 2048
distinguished_name = req_distinguished_name
req_extensions = req_ext
x509_extensions = v3_req
prompt = no
[req_distinguished_name]
countryName = IL
stateOrProvinceName = N/A
localityName = KS
organizationName = Lightbits
commonName = $NEWHOST
serialNumber = $CLUSTER_ID
[req_ext]
subjectAltName = @alt_names
[v3_req]
subjectAltName = @alt_names
basicConstraints = CA:FALSE
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer:always
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth,serverAuth
[alt_names]
IP.1=$NEW_DATA_IP
IP.2=127.0.0.1
DNS.1=$NEWHOST
DNS.2=localhost
EOF

openssl req -nodes -newkey rsa:2048 -config /tmp/$NEWHOST.cnf \
  -keyout $NEWHOST-cert-etcd-peer-key.pem -out $NEWHOST-cert-etcd-peer-csr.pem
openssl x509 -req -days 3650 -set_serial 0x$(openssl rand -hex 20) \
  -in $NEWHOST-cert-etcd-peer-csr.pem -out $NEWHOST-cert-etcd-peer.pem \
  -extensions v3_req -extfile /tmp/$NEWHOST.cnf -CA etcd-ca.pem -CAkey etcd-ca-key.pem
chmod 600 $NEWHOST-cert-etcd-peer*

openssl x509 -in $NEWHOST-cert-etcd-peer.pem -noout -text | grep -A2 'Subject Alternative Name'
openssl verify -CAfile etcd-ca.pem $NEWHOST-cert-etcd-peer.pem
diff <(openssl x509 -in $NEWHOST-cert-etcd-peer.pem -noout -pubkey) \
     <(openssl pkey -in $NEWHOST-cert-etcd-peer-key.pem -pubout) && echo "KEY-CERT MATCH"
# 3/3 xanh moi upload:
ssh root@$NEW 'mkdir -p /etc/etcd/ssl && chmod 700 /etc/etcd/ssl'
scp etcd-ca.pem etcd-ca-key.pem root@$NEW:/etc/etcd/ssl/
scp $NEWHOST-cert-etcd-peer.pem     root@$NEW:/etc/etcd/ssl/cert-etcd-peer.pem
scp $NEWHOST-cert-etcd-peer-key.pem root@$NEW:/etc/etcd/ssl/cert-etcd-peer-key.pem

### [NEW04]
chmod 600 /etc/etcd/ssl/*; ls -la /etc/etcd/ssl/

A3 — etcd join

### [NEW04]
set +H
NEWHOST=$(hostname); NEW_DATA_IP=10.237.95.104
REF_HOST=oss-nvme-01; REF_DATA_IP=10.237.95.101
mkdir -p /var/lib/etcd && chmod 700 /var/lib/etcd
grep -q ETCDCTL_API ~/.bashrc || sed -i "1i export ETCDCTL_API=3" ~/.bashrc

### [ADMIN] — trung chuyen unit tu REF sang NEW04
scp root@10.237.94.101:/etc/systemd/system/etcd.service /tmp/etcd.service.orig
scp /tmp/etcd.service.orig root@$NEW:/tmp/etcd.service.orig

### [REF] — chuoi initial-cluster tu member list that (4 member ke ca 05 — van hop le!)
etcdctl member list | grep -v unstarted | awk -F', ' '{printf "%s%s=%s",sep,$3,$4; sep=","}'; echo

### [NEW04] — sed + kiem 4 dieu
INITIAL_CLUSTER='<chuoi tu REF>,'"$NEWHOST=https://$NEW_DATA_IP:2380"
echo "$INITIAL_CLUSTER"    # 5 entry
for v in NEWHOST NEW_DATA_IP REF_HOST REF_DATA_IP INITIAL_CLUSTER; do [ -z "${!v}" ] && echo "!! $v RONG"; done
sed -e "s/--name=$REF_HOST/--name=$NEWHOST/" \
    -e "s/${REF_DATA_IP//./\\.}/$NEW_DATA_IP/g" \
    -e "s/--initial-cluster-state=new/--initial-cluster-state=existing/" \
    -e "s#--initial-cluster=[^ ]*#--initial-cluster=$INITIAL_CLUSTER#" \
    /tmp/etcd.service.orig > /etc/systemd/system/etcd.service
grep -E 'name=|initial-cluster|urls' /etc/systemd/system/etcd.service   # name=04 / .104 / existing / 5 entry
systemctl is-active firewalld >/dev/null 2>&1 && firewall-cmd --add-port=2379/tcp --add-port=2380/tcp --permanent && firewall-cmd --reload

### [REF]
etcdctl member add oss-nvme-04 --peer-urls=https://10.237.95.104:2380
etcdctl member list                       # 04 unstarted

### [NEW04] — quet ma + start
ps -C etcd -o args=                       # co gi tran trui -> con ma
systemctl stop etcd 2>/dev/null; rm -rf /var/lib/etcd/*
systemctl daemon-reload && systemctl enable --now etcd
sleep 8; systemctl is-active etcd

### [REF]
etcdctl member list                       # 5 member, 04 started

A4 — Config

### [ADMIN] — trung chuyen 8 thu muc REF -> NEW04
rm -rf /tmp/cfg && mkdir -p /tmp/cfg && cd /tmp/cfg
for d in node-manager cluster-manager api-service upgrade-manager discovery-service lbcli lightbox-exporter profile-generator; do
  scp -q -r root@10.237.94.101:/etc/$d ./ && echo "got $d"
done
ssh root@$NEW 'rm -rf /tmp/cfg && mkdir -p /tmp/cfg'
scp -q -r /tmp/cfg/* root@$NEW:/tmp/cfg/ && echo "pushed"

### [NEW04] — OLD_* dong, UUID moi, sed, kiem
cd /tmp/cfg; set +H
OLD_SUUID=$(grep '^serverUUID:' node-manager/node-manager.yaml | awk '{print $2}')
OLD_NUUID=$(grep -m1 '^- uuid:' node-manager/node-manager.yaml | awk '{print $3}')
OLD_CMID=$(grep '^id:' cluster-manager/cluster-manager.yaml | awk '{print $2}' | tr -d '"')
SUUID=$(uuidgen); NUUID=$(uuidgen)
echo "OLD: $OLD_SUUID / $OLD_NUUID / $OLD_CMID"; echo "NEW: SUUID=$SUUID NUUID=$NUUID"   # LUU LAI
for v in NEWHOST NEW_DATA_IP REF_HOST REF_DATA_IP SUUID NUUID OLD_SUUID OLD_NUUID OLD_CMID; do [ -z "${!v}" ] && echo "!! $v RONG"; done

grep -rl "$REF_HOST" . | xargs -r sed -i "s/$REF_HOST/$NEWHOST/g"
grep -rl "$REF_DATA_IP" . | xargs -r sed -i "s/${REF_DATA_IP//./\\.}/$NEW_DATA_IP/g"
grep -rl "$OLD_SUUID" . | xargs -r sed -i "s/$OLD_SUUID/$SUUID/g"
grep -rl "$OLD_NUUID" . | xargs -r sed -i "s/$OLD_NUUID/$NUUID/g"
sed -i "s/^id: $OLD_CMID/id: \"\"/" cluster-manager/cluster-manager.yaml
printf 'profileGeneratorVersion: 0\n' > profile-generator/system-profile.yaml
rm -f node-manager/gftl.db node-manager/node-manager.yaml.rpmsave

grep -rn "$REF_HOST\|${REF_DATA_IP//./\\.}\|$OLD_SUUID\|$OLD_NUUID" .   # residue TRONG
grep -E '^name:|serverUUID:' node-manager/node-manager.yaml
grep -E 'serverName:|^id:|clusterId:' cluster-manager/cluster-manager.yaml
grep -iE 'serverUUID' discovery-service/*.yaml                          # = SUUID moi

# DON DICH truoc, copy sau
for d in node-manager cluster-manager api-service upgrade-manager discovery-service lbcli lightbox-exporter profile-generator; do rm -rf /etc/$d; done
cp -r /tmp/cfg/* /etc/
ls /etc/node-manager/gftl.db 2>/dev/null && echo "!! CON GFTL.DB" || echo OK

A5 — Start đúng thứ tự

### [NEW04]
systemctl daemon-reload
systemctl enable --now discovery-service; sleep 5; systemctl is-active discovery-service
etcdctl get --prefix /registry/discovery-service --keys-only
# CHECKPOINT: phai co key $SUUID moi (5 key: 01/02/03/05 + 04moi). Khong co -> DUNG.
for s in profile-generator node-manager cluster-manager upgrade-manager api-service; do
  systemctl enable --now $s; sleep 3; echo -n "$s: "; systemctl is-active $s
done
journalctl -fu node-manager    # format 8 o -> journal -> md -> duroslight -> Active

A6 — Verify

### [ADMIN]
lbcli --endpoint $ENDPOINT -J "$LIGHTOS_JWT" list servers    # 5 server: 01/02/03/04moi/05chet
lbcli --endpoint $ENDPOINT -J "$LIGHTOS_JWT" list nodes      # 04 Active 61TiB; 05 van Unattached
lbcli --endpoint $ENDPOINT -J "$LIGHTOS_JWT" list nvme-devices | grep $SUUID | awk '{print $NF}' | sort | uniq -c   # 6 Data + 2 Journal
### [NEW04]
grep -E "^md" /proc/mdstat; systemctl is-active duroslight-0

PHA B — Thí nghiệm replace node (giáo dục, 0 volume nên vô hại)

Prereq theo docs: source Inactive, target Unattached. Hiện trạng ta có NGƯỢC vai: 05 (nguồn muốn bỏ) = Unattached, 04 (mới) = Active → dự đoán bị từ chối. Chạy để lấy thông điệp lỗi làm tư liệu máy trạng thái:

### [ADMIN]
NODE04=<node uuid moi cua 04, tu list nodes>
lbcli --endpoint $ENDPOINT -J "$LIGHTOS_JWT" replace node --src-node-uuid=8861a8fc-b72b-4129-b9e9-1fe77a5a6e82 --target-node-uuid=$NODE04
# Ghi lai nguyen van loi tra ve (ky vong: tu choi vi src khong Inactive / target khong Unattached)

Kết luận ghi sổ: replace node dành cho kịch bản "node NGUỒN vừa Inactive còn dữ liệu cần chuyển + node ĐÍCH dự phòng đang Unattached chờ sẵn" — không phải công cụ dọn node chết đã Unattached. Unattached có vai trò kép: ga cuối trước delete VÀ ghế chờ của node dự phòng.

PHA C — Giải thể hoàn toàn oss-nvme-05

C1 — Tắt nguồn phát trước (bài học server hồi sinh)

### [OLD05]
for s in node-manager cluster-manager api-service discovery-service upgrade-manager lightbox-exporter; do
  systemctl stop $s 2>/dev/null
done
systemctl is-active node-manager discovery-service    # failed/inactive
# etcd TREN 05 TAM GIU — go member sau khi delete server

C2 — Gỡ server khỏi cụm (node đã Unattached → không timer)

### [ADMIN]
lbcli --endpoint $ENDPOINT -J "$LIGHTOS_JWT" disable server --uuid 35b83b77-a97d-43a5-8325-39f84ac67605
sleep 10
lbcli --endpoint $ENDPOINT -J "$LIGHTOS_JWT" delete server --uuid 35b83b77-a97d-43a5-8325-39f84ac67605
lbcli --endpoint $ENDPOINT -J "$LIGHTOS_JWT" list servers                     # 4: 01/02/03/04
lbcli --endpoint $ENDPOINT -J "$LIGHTOS_JWT" list nvme-devices | grep -c 35b83b77   # 0 — record Failed chet theo server

C3 — Gỡ etcd member của 05

### [REF]
etcdctl member list        # delete server co the DA tu go member 05 (kiem truoc)
# Neu member 05 (ID hien tai, vd 35049cb5...) VAN CON:
### [OLD05]
systemctl disable --now etcd
### [REF]
etcdctl member remove <ID-cua-05>
etcdctl member list        # 4 member: 01/02/03/04

# Don key registry mo coi cua 05 (discovery da stop, key co the li):
etcdctl get --prefix /registry/discovery-service --keys-only
etcdctl del /registry/discovery-service/35b83b77-a97d-43a5-8325-39f84ac67605   # neu con

C4 — Ép sạch máy 05 (5 nhịp)

### [OLD05]
for s in duroslight-0 node-manager cluster-manager api-service discovery-service upgrade-manager lightbox-exporter profile-generator etcd; do
  systemctl disable --now $s 2>/dev/null
done
PKGS=$(rpm -qa | grep -iE "lightos|duros|node-manager|cluster-manager|api-service|discovery|upgrade-manager|profile-generator|lightbox|lbcli|etcd")
[ -n "$PKGS" ] && dnf remove -y $PKGS 2>&1 | tail -2
cd /etc/yum.repos.d && for f in *.repo; do [ -e "$f" ] && mv "$f" "$f.disabled"; done
for m in /dev/md*; do [ -e "$m" ] && mdadm --detail "$m" 2>/dev/null | grep -q raid0 && mdadm --stop "$m"; done
rm -rf /var/lib/etcd /etc/lightos* /etc/node-manager /etc/cluster-manager /etc/api-service \
       /etc/discovery-service /etc/upgrade-manager /etc/profile-generator /etc/duroslight \
       /etc/lbcli /etc/lightbox-exporter /var/cache/nm_service /usr/lib/systemd/system/duroslight-*.service
for d in 0 1 2 3 4 5 6 7; do wipefs -a /dev/nvme${d}n1 >/dev/null 2>&1; done
# Kiem: 0/8/0/0/0
echo -n "pkg: ";   rpm -qa | grep -icE "lightos|duros|etcd"
echo -n "disks: "; lsblk | grep -cE "nvme[0-7]n1"
echo -n "repos: "; ls /etc/yum.repos.d/*.repo 2>/dev/null | wc -l
echo -n "raid0: "; grep -E "^md[0-9]+ :" /proc/mdstat | grep -c raid0
echo -n "leftover: "; ls -d /var/lib/etcd /etc/lightos* /etc/node-manager /etc/cluster-manager /etc/duroslight /var/cache/nm_service /usr/lib/systemd/system/duroslight-*.service 2>/dev/null | wc -l

C5 — Verify cụm cuối cùng

### [ADMIN]
lbcli --endpoint $ENDPOINT -J "$LIGHTOS_JWT" list servers    # 4, NoRisk het
lbcli --endpoint $ENDPOINT -J "$LIGHTOS_JWT" list nodes      # 4 Active 61TiB
lbcli --endpoint $ENDPOINT -J "$LIGHTOS_JWT" list events | head -5
### [REF]
etcdctl member list                                          # 4 started
etcdctl get --prefix /registry/discovery-service --keys-only # 4 key khop 4 server

C6 — Hậu kỳ

  • Cập nhật inventory ansible: 04 vào, 05 ra (3 section + host_vars)
  • Timer về mặc định nếu có hạ: update cluster-config --parameter=DurationToTurnIntoPermanentFailure --value=4h
  • 05 giờ là node dự phòng sạch — sẵn sàng cho vòng diễn tập kế tiếp

Bài học kịch bản này bổ sung cho sổ tay

  1. Node chết vì journal + record SSD Failed = không có đường mềm; đường production thật: đưa node dự phòng vào (add server thường) + giải thể node chết (delete server) — chính là tổ hợp hai runbook đã thuộc, không cần công cụ mới.
  2. replace node là công cụ CHUYỂN DỮ LIỆU src-Inactive → target-Unattached (node dự phòng chờ sẵn) — không phải công cụ dọn node đã Unattached.
  3. Record SSD Failed chết theo server khi delete server — cơ chế "gỡ record" duy nhất.
  4. etcd member của node chết vẫn started bình thường cho tới khi chủ động gỡ — hai tầng (etcd membership / LightOS server) sống chết độc lập nhau.

© Copyright Infra Notes contributors. Last updated on September 07, 2026.