Logo

Personal Ops Runbook

Personal runbook covering infrastructure operations for Cloud, Kubernetes, OpenStack, and Ceph environments. Includes deployment and teardown procedures, node management, cluster monitoring setup, and incident response workflows compiled from day-to-day operational work. Intended strictly for personal reference — configurations and scripts are environment-specific and not guaranteed to work as-is elsewhere.

Xóa node khỏi cluster

Điều kiện

  • Cluster phải có ≥ 4 server (dưới 4: liên hệ Lightbits Support).
  • Gate của delete server: node phải ở trạng thái Unattached. Delete sớm sẽ báo lỗi cannot delete server whose nodes are not in Unattached state.

Máy trạng thái

disable server → node Inactive → [chờ hết DurationToTurnIntoPermanentFailure, mặc định 4h]
              → node Unattached → delete server (xóa entry server + node + etcd)

Thiết lập biến môi trường thông tin các node liên quan

ADMIN_NODE=10.237.94.100  
CURRENT_NODE=10.237.94.104  
OLD_NODE_IPADDR=10.237.94.104  
OLD_NODE_HOSTNAME=oss-nvme-04  
source ~/lightos-system-jwt
EP=https://${CURRENT_NODE}:443

Quy trình chuẩn

# 1.1. Xem list servers và lấy thông tin UUID tại đây (lưu ý không lấy thông Node UUID)
[root@vm-admin-01 ~]# lbcli --endpoint $EP -J "$LIGHTOS_JWT" list servers
NAME          UUID                                   State     RiskOfServiceLoss State   LightOSVersion
oss-nvme-01   a0d5bf2a-2ef6-5ff9-953b-592531939f78   Enabled   NoRiskOfServiceLoss       3.18.2b26868114363
oss-nvme-02   a17b6c72-5a59-5c33-a699-61bdc24b9b54   Enabled   NoRiskOfServiceLoss       3.18.2b26868114363
oss-nvme-03   88940976-0aa4-5506-852b-e5861d3816cb   Enabled   NoRiskOfServiceLoss       3.18.2b26868114363
oss-nvme-04   a173b57d-648b-4e52-9c14-a6c04dfd6c8b   Enabled   NoRiskOfServiceLoss       3.18.2b26868114363
oss-nvme-05   35b83b77-a97d-43a5-8325-39f84ac67605   Enabled   NoRiskOfServiceLoss       3.18.2b26868114363

# 1.2. Xem lại list nodes
[root@vm-admin-01 ~]# lbcli --endpoint $EP -J "$LIGHTOS_JWT" list nodes
Name            UUID                                   State      NVMe endpoint        Failure domains   Capacity   Used      Local rebuild progress
oss-nvme-01-0   e8bc0737-a4af-5538-8952-3cdf63a20f96   Active     10.237.95.101:4420   [oss-nvme-01]     61 TiB     0 B       None
oss-nvme-02-0   ef732732-4c12-5e22-8bf4-eb3dddbd0cc7   Active     10.237.95.102:4420   [oss-nvme-02]     61 TiB     0 B       None
oss-nvme-03-0   5c5b6d5d-21c0-51e2-9c02-359936ae9c70   Active     10.237.95.103:4420   [oss-nvme-03]     61 TiB     0 B       None
oss-nvme-04-0   f2a4f333-9aaf-45fe-bcab-d479c20bb634   Active     10.237.95.104:4420   [oss-nvme-04]     61 TiB     0 B       None
oss-nvme-05-0   8861a8fc-b72b-4129-b9e9-1fe77a5a6e82   Active     10.237.95.105:4420   [oss-nvme-05]     61 TiB     0 B       None

# 1.3. Lấy nhanh SRV_UUID node cần gỡ
SRV_UUID=$(lbcli --endpoint $EP -J "$LIGHTOS_JWT" list servers | grep ${OLD_NODE_HOSTNAME} | awk '{print $2}')

# 2. Hạ Ha timer 4h -> 1h
## 2.1 (Tùy chọn) Hạ timer 4h -> 1h để rút ngắn thời gian chờ
[root@vm-admin-01 ~]# lbcli --endpoint $EP -J "$LIGHTOS_JWT" update cluster-config --parameter=DurationToTurnIntoPermanentFailure --value=1h

## 2.2. Verify lại kết quả
[root@vm-admin-01 ~]# lbcli --endpoint $EP -J "$LIGHTOS_JWT" get cluster-config --parameter=DurationToTurnIntoPermanentFailure
Cluster Config Parameter             Value
DurationToTurnIntoPermanentFailure   1h0m0s

# 4. Disable
[root@vm-admin-01 ~]# lbcli --endpoint $EP -J "$LIGHTOS_JWT" disable server --uuid ${SRV_UUID}

# 5. Sau khi disable, node chuyển sang trạng thái Inactive
[root@vm-admin-01 ~]# lbcli --endpoint $EP -J "$LIGHTOS_JWT" list nodes
Name            UUID                                   State      NVMe endpoint        Failure domains   Capacity   Used      Local rebuild progress
oss-nvme-01-0   e8bc0737-a4af-5538-8952-3cdf63a20f96   Active     10.237.95.101:4420   [oss-nvme-01]     61 TiB     0 B       None
oss-nvme-02-0   ef732732-4c12-5e22-8bf4-eb3dddbd0cc7   Active     10.237.95.102:4420   [oss-nvme-02]     61 TiB     0 B       None
oss-nvme-03-0   5c5b6d5d-21c0-51e2-9c02-359936ae9c70   Active     10.237.95.103:4420   [oss-nvme-03]     61 TiB     0 B       None
oss-nvme-04-0   f2a4f333-9aaf-45fe-bcab-d479c20bb634   Inactive   10.237.95.104:4420   [oss-nvme-04]     61 TiB     0 B       None
oss-nvme-05-0   8861a8fc-b72b-4129-b9e9-1fe77a5a6e82   Active     10.237.95.105:4420   [oss-nvme-05]     61 TiB     0 B       None

# 6. Delete khi Unattached
## 6.1 Chắc chắn node đã chuyển sang trạng thái Unattached
[root@vm-admin-01 ~]# lbcli --endpoint $EP -J "$LIGHTOS_JWT" list nodes
Name            UUID                                   State        NVMe endpoint        Failure domains   Capacity   Used      Local rebuild progress
oss-nvme-01-0   e8bc0737-a4af-5538-8952-3cdf63a20f96   Active       10.237.95.101:4420   [oss-nvme-01]     61 TiB     0 B       None
oss-nvme-02-0   ef732732-4c12-5e22-8bf4-eb3dddbd0cc7   Active       10.237.95.102:4420   [oss-nvme-02]     61 TiB     0 B       None
oss-nvme-03-0   5c5b6d5d-21c0-51e2-9c02-359936ae9c70   Active       10.237.95.103:4420   [oss-nvme-03]     61 TiB     0 B       None
oss-nvme-04-0   f2a4f333-9aaf-45fe-bcab-d479c20bb634   Unattached   10.237.95.104:4420   [oss-nvme-04]     61 TiB     0 B       None
oss-nvme-05-0   8861a8fc-b72b-4129-b9e9-1fe77a5a6e82   Active       10.237.95.105:4420   [oss-nvme-05]     61 TiB     0 B       None

## 6.2 Delete node đã Unattached và verify lại
[root@vm-admin-01 ~]# lbcli --endpoint $EP -J "$LIGHTOS_JWT" delete server --uuid ${SRV_UUID}
[root@vm-admin-01 ~]# lbcli --endpoint $EP -J "$LIGHTOS_JWT" list nodes
Name            UUID                                   State     NVMe endpoint        Failure domains   Capacity   Used      Local rebuild progress
oss-nvme-01-0   e8bc0737-a4af-5538-8952-3cdf63a20f96   Active    10.237.95.101:4420   [oss-nvme-01]     61 TiB     0 B       None
oss-nvme-02-0   ef732732-4c12-5e22-8bf4-eb3dddbd0cc7   Active    10.237.95.102:4420   [oss-nvme-02]     61 TiB     0 B       None
oss-nvme-03-0   5c5b6d5d-21c0-51e2-9c02-359936ae9c70   Active    10.237.95.103:4420   [oss-nvme-03]     61 TiB     0 B       None
oss-nvme-05-0   8861a8fc-b72b-4129-b9e9-1fe77a5a6e82   Active    10.237.95.105:4420   [oss-nvme-05]     61 TiB     0 B       None

Chọn hướng để action

Hướng A — Chờ: không làm gì, chờ node đã bị disable từ trạng thái Inactive chuyển sang trạng thái Unattached.
Hướng B — Evict ngay: chạy lệnh dưới để kiểm flag trước xem version có hỗ trợ hay không rồi mới đi tiếp:

[root@vm-admin-01 light-app]# lbcli --endpoint $EP -J "$LIGHTOS_JWT" disable server --help 2>&1 | grep -iE "evict|force"
To force disabling a server, even if it may lead to risk of service loss, use the --force flag
      --evict-data                         Evict data from the server (optional, default: false). NOTE: This option is not currently supported
      --force                              Force disable, bypass risk of service loss checks (optional, default: false)

Ở version hiện tại thì output trên thì không có cách nào xóa node ngay lập tức, bắt buộc phải chờ đủ DurationToTurnIntoPermanentFailure. Phương án evict chỉ có từ 3.19.x. Cách nhanh duy nhất trên 3.18.2 là hạ timer xuống 1h trước khi disable.

Quy trình nhanh cho hướng B (bỏ chờ — evict, Technology Preview)

  • Phải enable Evict Data feature trước (xem doc "Enabling the Evict Data Feature").
  • Chỉ áp dụng cho server đang Enabled (đã lỡ disable thì enable lại rồi mới evict).
  • Evict có thể fail nếu các server còn lại thiếu dung lượng trống chứa dữ liệu dời sang.
lbcli disable server --name=<server_name> --evict-data           # migrate ngay, xong -> Unattached -> delete
lbcli disable server --name=<server_name> --force --evict-data   # node da chet/RiskOfServiceLoss (hong phan cung)

C3 — Gỡ etcd member của oss-nvme-04

### [REF]
etcdctl member list

### [NODE REMOVE]
systemctl disable --now etcd

### [REF]
etcdctl member remove <id>
etcdctl member list

# Dọn key registry
etcdctl get --prefix /registry/discovery-service --keys-only
etcdctl del /registry/discovery-service/${SRV_UUID}

Verify sau delete

Command thực hiện

[root@vm-admin-01 ~]# lbcli --endpoint $EP -J "$LIGHTOS_JWT" list servers
NAME          UUID                                   State     RiskOfServiceLoss State   LightOSVersion
oss-nvme-01   a0d5bf2a-2ef6-5ff9-953b-592531939f78   Enabled   NoRiskOfServiceLoss       3.18.2b26868114363
oss-nvme-02   a17b6c72-5a59-5c33-a699-61bdc24b9b54   Enabled   NoRiskOfServiceLoss       3.18.2b26868114363
oss-nvme-03   88940976-0aa4-5506-852b-e5861d3816cb   Enabled   NoRiskOfServiceLoss       3.18.2b26868114363
oss-nvme-05   35b83b77-a97d-43a5-8325-39f84ac67605   Enabled   NoRiskOfServiceLoss       3.18.2b26868114363

[root@vm-admin-01 ~]# lbcli --endpoint $EP -J "$LIGHTOS_JWT" list nodes
Name            UUID                                   State     NVMe endpoint        Failure domains   Capacity   Used      Local rebuild progress
oss-nvme-01-0   e8bc0737-a4af-5538-8952-3cdf63a20f96   Active    10.237.95.101:4420   [oss-nvme-01]     61 TiB     0 B       None
oss-nvme-02-0   ef732732-4c12-5e22-8bf4-eb3dddbd0cc7   Active    10.237.95.102:4420   [oss-nvme-02]     61 TiB     0 B       None
oss-nvme-03-0   5c5b6d5d-21c0-51e2-9c02-359936ae9c70   Active    10.237.95.103:4420   [oss-nvme-03]     61 TiB     0 B       None
oss-nvme-05-0   8861a8fc-b72b-4129-b9e9-1fe77a5a6e82   Active    10.237.95.105:4420   [oss-nvme-05]     61 TiB     0 B       None

[root@oss-nvme-01 ~]# etcdctl member list
35049cb5f985d69e, started, oss-nvme-05, https://10.237.95.105:2380, http://localhost:2379,https://10.237.95.105:2379, false
49675b756efdbdb4, started, oss-nvme-02, https://10.237.95.102:2380, http://localhost:2379,https://10.237.95.102:2379, false
92786a305fae6ea5, started, oss-nvme-03, https://10.237.95.103:2380, http://localhost:2379,https://10.237.95.103:2379, false
dcea60084de74739, started, oss-nvme-01, https://10.237.95.101:2380, http://localhost:2379,https://10.237.95.101:2379, false

Dừng service + gỡ package + xóa jwt/CA cũ trên node 10.237.94.104

Stop LightOS services

systemctl stop api-service cluster-manager node-manager duroslight-0 gftl lightbox-exporter discovery-service etcd 2>/dev/null

Remove all LightOS packages

dnf remove -y lightos duroslight management-node-manager management-cluster-manager management-api-service management-profile-generator management-lbcli discovery-service discovery-client management-lightbox-exporter management-upgrade-manager klight_utils userlbe etcd gftl 2>/dev/null

Remove leftover JWT + old CA/etcd data that poison next deploy

rm -f /root/lightos-system-jwt /root/lightos-default-admin-jwt
rm -rf /etc/lightos /etc/etcd/ssl /var/lib/etcd /var/lib/lightos* 2>/dev/null
ls /root/lightos-*jwt* 2>/dev/null | wc -l
ls /etc/etcd/ssl 2>/dev/null | wc -l
rpm -qa | grep -icE "lightos|duros|node-manager|cluster-manager|api-service|discovery|etcd|gftl|userlbe"

Kỳ vọng kết quả: 0/0/0.

Reboot node để nhả md126

Reboot qua Ansible từ ADMIN

cat > /tmp/all3.ini << 'EOF'
10.237.94.104
EOF

cd ~/light-app
ansible -i /tmp/all3.ini all -m reboot -a "reboot_timeout=1200" -b -u root

Reboot trực tiếp trên node

reboot

Sau reboot wipe các ổ NVMe

# Stop leftover raid0 (NOT md127 = OS raid1)
for m in /dev/md12[0-6]; do
  [ -e "$m" ] && mdadm --detail "$m" 2>/dev/null | grep -q "raid0" && mdadm --stop "$m" && echo "stopped $m"
done

# Wipe 8 data drives (nvme0-7), NOT sda/sdb/md127
for d in 0 1 2 3 4 5 6 7; do wipefs -a /dev/nvme${d}n1 >/dev/null 2>&1; done
for d in 0 1 2 3 4 5 6 7; do nvme format /dev/nvme${d}n1 --lbaf=1 --force >/dev/null 2>&1; done
lsblk | grep -cE "nvme[0-7]n1"
grep -q md126 /proc/mdstat && echo 1 || echo 0
grep -q md127 /proc/mdstat && echo 0 || echo 1

Kỳ vọng 8/0/0.

Hậu kỳ

  1. Inventory: xóa node khỏi 3 section trong hosts + xóa file host_vars/<node>.yml nếu sử dụng deploy bằng Ansible.
  2. Trả cluster-config: set DurationToTurnIntoPermanentFailure về giá trị gốc (vd 4h0m0s) nếu đã hạ.
  3. Nếu tái sử dụng, dọn node đã xóa cho sạch sẽ: gỡ package lightos/etcd, rm -rf /var/lib/etcd /etc/lightos*, wipe + format 8 ổ NVMe, tắt repo online.

Lưu ý

  • Endpoint lbcli phải trỏ vào 1 trong các node còn lại, không trỏ node sắp xóa.
  • Timer là tham số toàn cluster: để giá trị thấp = node down quá hạn là bị coi permanent failure và kích rebuild → production nên giữ mặc định, chỉ hạ tạm khi thao tác.
  • Khi xóa node phải xác định capaciy, cluster phải stable, số lượng node còn lại để tránh thiếu quorum.

© Copyright Infra Notes contributors. Last updated on September 07, 2026.