Logo

Personal Ops Runbook

Personal runbook covering infrastructure operations for Cloud, Kubernetes, OpenStack, and Ceph environments. Includes deployment and teardown procedures, node management, cluster monitoring setup, and incident response workflows compiled from day-to-day operational work. Intended strictly for personal reference — configurations and scripts are environment-specific and not guaranteed to work as-is elsewhere.

9.8 - Add node bằng tay vào cluster Lightbits

Label Node Role
[ADMIN] vm-admin-01 giữ bundle RPM, CA + JWT, chạy lbcli
[NEW] node mới (vd 10.237.94.105 / oss-nvme-05) storage node
[REF] storage node hiện tại (vd 10.237.94.101 / oss-nvme-01) nguồn config + thao tác etcd cluster

Vì mỗi terminal là một shell riêng, biến phải khai trên từng node:

### [ADMIN] ###
set +H
source ~/lightos-system-jwt
ENDPOINT=https://10.237.94.101:443
CERTS=~/lightos-certificates-clusterNew
NEW=10.237.94.105

### [NEW] ###
set +H
NEWHOST=$(hostname)
NEW_DATA_IP=10.237.95.105
REF=10.237.94.101
REF_HOST=oss-nvme-01
REF_DATA_IP=10.237.95.101

### [REF] ###
set +H
NEWHOST=oss-nvme-05
NEW_DATA_IP=10.237.95.105

BƯỚC 0 — Verify node mới sạch và cluster đang stable

[NEW] — Kiểm tra tình trạng của node mới

# Check pkg
[root@oss-nvme-05 ~]# rpm -qa | grep -icE "lightos|duros|etcd"
0

# Check disks
[root@oss-nvme-05 ~]# lsblk | grep -cE "nvme[0-7]n1"
8

# Check repos
[root@oss-nvme-05 ~]# ls /etc/yum.repos.d/*.repo 2>/dev/null | wc -l
0

# Check number count raid0
[root@oss-nvme-05 ~]# grep -E "^md[0-9]+ :" /proc/mdstat | grep -c raid0
0

[root@oss-nvme-05 ~]# grep -E "^md[0-9]+ :" /proc/mdstat
md127 : active raid1 sda2[0] sdb1[1]

# Check leftover
[root@oss-nvme-05 ~]# ls -d /var/lib/etcd /etc/lightos* /etc/node-manager /etc/cluster-manager /etc/duroslight /var/cache/nm_service /usr/lib/systemd/system/duroslight-*.service 2>/dev/null | wc -l
0

→ Kỳ vọng tất lần lượt là 0/8/0/0/0. Nếu kết quả khác, chạy các command dưới để dọn

[NEW] — Các command dưới đây để ép dọn sạch (gỡ service -> gỡ RPM -> xóa config/data -> wipe disk)

# 1. Dừng + disable mọi service LightOS (chạy trước dnf remove, tránh service đang chạy lock file)
for s in duroslight-0 node-manager cluster-manager api-service discovery-service \
         upgrade-manager lightbox-exporter profile-generator etcd; do
  systemctl disable --now $s 2>/dev/null
done

# 2. Gỡ RPM
PKGS=$(rpm -qa | grep -iE "lightos|duros|node-manager|cluster-manager|api-service|discovery|upgrade-manager|profile-generator|lightbox|lbcli|etcd")
[ -n "$PKGS" ] && dnf remove -y $PKGS 2>&1 | tail -2

# 3. Dọn repo + md rác + config/data (dnf remove giữ lại config đã sửa + .rpmsave -> vẫn phải rm)
cd /etc/yum.repos.d && for f in *.repo; do [ -e "$f" ] && mv "$f" "$f.disabled"; done
for m in /dev/md*; do [ -e "$m" ] && mdadm --detail "$m" 2>/dev/null | grep -q raid0 && mdadm --stop "$m"; done
rm -rf /var/lib/etcd /etc/lightos* /etc/node-manager /etc/cluster-manager /etc/api-service \
       /etc/discovery-service /etc/upgrade-manager /etc/profile-generator /etc/duroslight \
       /etc/lbcli /etc/lightbox-exporter /var/cache/nm_service /usr/lib/systemd/system/duroslight-*.service

# 4. Wipe 8 disk NVMe (không được đụng tới sda/sdb của OS)
for d in 0 1 2 3 4 5 6 7; do wipefs -a /dev/nvme${d}n1 >/dev/null 2>&1; done

Sử dụng lại các command để verify lại

rpm -qa | grep -icE "lightos|duros|etcd"
lsblk | grep -cE "nvme[0-7]n1"
ls /etc/yum.repos.d/*.repo 2>/dev/null | wc -l
grep -E "^md[0-9]+ :" /proc/mdstat | grep -c raid0
ls -d /var/lib/etcd /etc/lightos* /etc/node-manager /etc/cluster-manager /etc/duroslight /var/cache/nm_service /usr/lib/systemd/system/duroslight-*.service 2>/dev/null | wc -l

→ Kết quả phải là 0/8/0/0/0

[ADMIN] — Check cluster thông qua node admin

Verify danh sách server hiện tại trong cluster

[root@vm-admin-01 ~]# lbcli --endpoint $ENDPOINT -J "$LIGHTOS_JWT" list servers
NAME          UUID                                   State     RiskOfServiceLoss State   LightOSVersion
oss-nvme-01   a0d5bf2a-2ef6-5ff9-953b-592531939f78   Enabled   NoRiskOfServiceLoss       3.18.2b26868114363
oss-nvme-02   a17b6c72-5a59-5c33-a699-61bdc24b9b54   Enabled   NoRiskOfServiceLoss       3.18.2b26868114363
oss-nvme-03   88940976-0aa4-5506-852b-e5861d3816cb   Enabled   NoRiskOfServiceLoss       3.18.2b26868114363
oss-nvme-04   a173b57d-648b-4e52-9c14-a6c04dfd6c8b   Enabled   NoRiskOfServiceLoss       3.18.2b26868114363

Verify state danh sách nvme-devices hiện tại trong cluster, chắc chắn không có nvme-devices nào trạng thái Failed/UnManaged

[root@vm-admin-01 ~]# lbcli --endpoint $ENDPOINT -J "$LIGHTOS_JWT" list nvme-devices | grep -icE "Failed|UnManaged"
0

[root@vm-admin-01 ~]# lbcli --endpoint $ENDPOINT -J "$LIGHTOS_JWT" list nvme-devices
Name      Size      NUMA ID   Serial           State     Server UUID                            Node UUID                              Usage
nvme7n1   14 TiB    0         S6VANN0L105910   Healthy   a0d5bf2a-2ef6-5ff9-953b-592531939f78   e8bc0737-a4af-5538-8952-3cdf63a20f96   Journal
nvme2n1   14 TiB    0         S6VANN0L105906   Healthy   a0d5bf2a-2ef6-5ff9-953b-592531939f78   e8bc0737-a4af-5538-8952-3cdf63a20f96   Data
nvme5n1   14 TiB    0         S6VANN0L105909   Healthy   a0d5bf2a-2ef6-5ff9-953b-592531939f78   e8bc0737-a4af-5538-8952-3cdf63a20f96   Journal
nvme6n1   14 TiB    0         S6VANN0L105948   Healthy   a0d5bf2a-2ef6-5ff9-953b-592531939f78   e8bc0737-a4af-5538-8952-3cdf63a20f96   Data
nvme1n1   14 TiB    0         S6VANN0L105907   Healthy   a0d5bf2a-2ef6-5ff9-953b-592531939f78   e8bc0737-a4af-5538-8952-3cdf63a20f96   Data
nvme4n1   14 TiB    0         S6VANN0L105908   Healthy   a0d5bf2a-2ef6-5ff9-953b-592531939f78   e8bc0737-a4af-5538-8952-3cdf63a20f96   Data
nvme0n1   14 TiB    0         S6VANN0L105949   Healthy   a0d5bf2a-2ef6-5ff9-953b-592531939f78   e8bc0737-a4af-5538-8952-3cdf63a20f96   Data
nvme3n1   14 TiB    0         S6VANN0L107087   Healthy   a0d5bf2a-2ef6-5ff9-953b-592531939f78   e8bc0737-a4af-5538-8952-3cdf63a20f96   Data
nvme0n1   14 TiB    0         S6VANN0L107098   Healthy   a17b6c72-5a59-5c33-a699-61bdc24b9b54   ef732732-4c12-5e22-8bf4-eb3dddbd0cc7   Data
nvme7n1   14 TiB    0         S6VANN0L106479   Healthy   a17b6c72-5a59-5c33-a699-61bdc24b9b54   ef732732-4c12-5e22-8bf4-eb3dddbd0cc7   Data
nvme4n1   14 TiB    0         S6VANN0L107096   Healthy   a17b6c72-5a59-5c33-a699-61bdc24b9b54   ef732732-4c12-5e22-8bf4-eb3dddbd0cc7   Journal
nvme6n1   14 TiB    0         S6VANN0L106470   Healthy   a17b6c72-5a59-5c33-a699-61bdc24b9b54   ef732732-4c12-5e22-8bf4-eb3dddbd0cc7   Journal
nvme2n1   14 TiB    0         S6VANN0L106516   Healthy   a17b6c72-5a59-5c33-a699-61bdc24b9b54   ef732732-4c12-5e22-8bf4-eb3dddbd0cc7   Data
nvme3n1   14 TiB    0         S6VANN0L106975   Healthy   a17b6c72-5a59-5c33-a699-61bdc24b9b54   ef732732-4c12-5e22-8bf4-eb3dddbd0cc7   Data
nvme1n1   14 TiB    0         S6VANN0L107100   Healthy   a17b6c72-5a59-5c33-a699-61bdc24b9b54   ef732732-4c12-5e22-8bf4-eb3dddbd0cc7   Data
nvme5n1   14 TiB    0         S6VANN0L107259   Healthy   a17b6c72-5a59-5c33-a699-61bdc24b9b54   ef732732-4c12-5e22-8bf4-eb3dddbd0cc7   Data
nvme6n1   14 TiB    0         S6VANN0L107043   Healthy   a173b57d-648b-4e52-9c14-a6c04dfd6c8b   f2a4f333-9aaf-45fe-bcab-d479c20bb634   Journal
nvme0n1   14 TiB    0         S6VANN0L107019   Healthy   a173b57d-648b-4e52-9c14-a6c04dfd6c8b   f2a4f333-9aaf-45fe-bcab-d479c20bb634   Journal
nvme4n1   14 TiB    0         S6VANN0L107039   Healthy   a173b57d-648b-4e52-9c14-a6c04dfd6c8b   f2a4f333-9aaf-45fe-bcab-d479c20bb634   Data
nvme7n1   14 TiB    0         S6VANN0L107042   Healthy   a173b57d-648b-4e52-9c14-a6c04dfd6c8b   f2a4f333-9aaf-45fe-bcab-d479c20bb634   Data
nvme1n1   14 TiB    0         S6VANN0L107020   Healthy   a173b57d-648b-4e52-9c14-a6c04dfd6c8b   f2a4f333-9aaf-45fe-bcab-d479c20bb634   Data
nvme3n1   14 TiB    0         S6VANN0L107035   Healthy   a173b57d-648b-4e52-9c14-a6c04dfd6c8b   f2a4f333-9aaf-45fe-bcab-d479c20bb634   Data
nvme2n1   14 TiB    0         S6VANN0L107032   Healthy   a173b57d-648b-4e52-9c14-a6c04dfd6c8b   f2a4f333-9aaf-45fe-bcab-d479c20bb634   Data
nvme5n1   14 TiB    0         S6VANN0L107036   Healthy   a173b57d-648b-4e52-9c14-a6c04dfd6c8b   f2a4f333-9aaf-45fe-bcab-d479c20bb634   Data
nvme7n1   14 TiB    0         S6VANN0L107091   Healthy   88940976-0aa4-5506-852b-e5861d3816cb   5c5b6d5d-21c0-51e2-9c02-359936ae9c70   Journal
nvme3n1   14 TiB    0         S6VANN0L105943   Healthy   88940976-0aa4-5506-852b-e5861d3816cb   5c5b6d5d-21c0-51e2-9c02-359936ae9c70   Data
nvme2n1   14 TiB    0         S6VANN0L107101   Healthy   88940976-0aa4-5506-852b-e5861d3816cb   5c5b6d5d-21c0-51e2-9c02-359936ae9c70   Data
nvme0n1   14 TiB    0         S6VANN0L107099   Healthy   88940976-0aa4-5506-852b-e5861d3816cb   5c5b6d5d-21c0-51e2-9c02-359936ae9c70   Data
nvme6n1   14 TiB    0         S6VANN0L107102   Healthy   88940976-0aa4-5506-852b-e5861d3816cb   5c5b6d5d-21c0-51e2-9c02-359936ae9c70   Data
nvme4n1   14 TiB    0         S6VANN0L105950   Healthy   88940976-0aa4-5506-852b-e5861d3816cb   5c5b6d5d-21c0-51e2-9c02-359936ae9c70   Journal
nvme1n1   14 TiB    0         S6VANN0L107095   Healthy   88940976-0aa4-5506-852b-e5861d3816cb   5c5b6d5d-21c0-51e2-9c02-359936ae9c70   Data
nvme5n1   14 TiB    0         S6VANN0L107097   Healthy   88940976-0aa4-5506-852b-e5861d3816cb   5c5b6d5d-21c0-51e2-9c02-359936ae9c70   Data

[REF] — Kiểm tra các discovery-service mà node đã đăng ký endpoint dưới key = serverUUID của nó với ETC, node-manager sẽ dùng key này khi dựng datapath.

[root@oss-nvme-01 ~]# etcdctl get --prefix /registry/discovery-service --keys-only
/registry/discovery-service/88940976-0aa4-5506-852b-e5861d3816cb
/registry/discovery-service/a0d5bf2a-2ef6-5ff9-953b-592531939f78
/registry/discovery-service/a173b57d-648b-4e52-9c14-a6c04dfd6c8b
/registry/discovery-service/a17b6c72-5a59-5c33-a699-61bdc24b9b54

[REF] — Xem trạng thái membership của etcd, đã started (đồng bộ dữ liệu, tham gia bầu cử) chưa. Một node storage muốn vào cluster Lightbits thì trước hết phải là member etcd.

[root@oss-nvme-01 ~]# etcdctl member list
2969ce575aecaac3, started, oss-nvme-04, https://10.237.95.104:2380, http://localhost:2379,https://10.237.95.104:2379, false
49675b756efdbdb4, started, oss-nvme-02, https://10.237.95.102:2380, http://localhost:2379,https://10.237.95.102:2379, false
92786a305fae6ea5, started, oss-nvme-03, https://10.237.95.103:2380, http://localhost:2379,https://10.237.95.103:2379, false
dcea60084de74739, started, oss-nvme-01, https://10.237.95.101:2380, http://localhost:2379,https://10.237.95.101:2379, false
f547908c105ada5b, started, oss-nvme-05, https://10.237.95.105:2380, http://localhost:2379,https://10.237.95.105:2379, false

BƯỚC 1 — Cài package, disable toàn bộ service cũ

[NEW] — Lấy bundle từ node ADMIN

# Xóa và tạo lại thư mục bundle
rm -rf /tmp/rpms
mkdir -p /tmp/rpms

# Copy bundle từ node ADMIN sang
scp ~/lightos-offline/target_rpms/*.rpm ~/lightos-offline/deps/*.rpm root@$NEW:/tmp/rpms/

[NEW] — Dựng repo local và cài đặt package

yum install -y --nogpgcheck /tmp/rpms/createrepo_c-libs*.rpm /tmp/rpms/createrepo_c-[0-9]*.rpm /tmp/rpms/drpm*.rpm 2>/dev/null
createrepo /tmp/rpms
printf "[lightos]\nname=lightos\nbaseurl=file:///tmp/rpms\ngpgcheck=0\nenabled=1\n" > /etc/yum.repos.d/lightos.repo
dnf clean all
yum install -y --nogpgcheck etcd
yum install -y --nogpgcheck lightos --allowerasing
yum install -y --nogpgcheck tar net-tools yum-utils

[NEW] — 1.4 Disable hết dịch vụ cũ rồi reboot server

# Disable service
for s in etcd node-manager cluster-manager api-service discovery-service upgrade-manager lightbox-exporter profile-generator irqbalance; do
  systemctl disable --now $s 2>/dev/null
done

# Reboot
reboot

[NEW] — 1.5 Checkpoint sau khi server reboot xong, tất cả dịch vụ phải inactive/failed

# Kỳ vọng kết quả inactive/failed
[root@oss-nvme-05 ~]# systemctl is-active etcd node-manager discovery-service cluster-manager
inactive
inactive
inactive
inactive

# Process kỳ vọng kết quả trống
[root@oss-nvme-05 ~]# ps -C etcd,node-manager -o args= | wc -l
0

[ADMIN] — Verify lại cluster vẫn phải stable như đã verify ở bước trên

[root@vm-admin-01 ~]# lbcli --endpoint $ENDPOINT -J "$LIGHTOS_JWT" list servers
NAME          UUID                                   State     RiskOfServiceLoss State   LightOSVersion
oss-nvme-01   a0d5bf2a-2ef6-5ff9-953b-592531939f78   Enabled   NoRiskOfServiceLoss       3.18.2b26868114363
oss-nvme-02   a17b6c72-5a59-5c33-a699-61bdc24b9b54   Enabled   NoRiskOfServiceLoss       3.18.2b26868114363
oss-nvme-03   88940976-0aa4-5506-852b-e5861d3816cb   Enabled   NoRiskOfServiceLoss       3.18.2b26868114363
oss-nvme-04   a173b57d-648b-4e52-9c14-a6c04dfd6c8b   Enabled   NoRiskOfServiceLoss       3.18.2b26868114363

[ADMIN] — 1.6 Verify package từ ADMIN

diff <(ssh root@$REF 'rpm -qa --qf "%{NAME}\n" | sort') \
     <(ssh root@$NEW 'rpm -qa --qf "%{NAME}\n" | sort') | grep '^<'

Thang đọc kết quả diff ở 3 mức:

  • Rỗng = hoàn hảo
  • Chỉ tiện ích OS (epel, sysstat, pcp, nload...) = đạt, chấp nhận được
  • Có bất kỳ gói nào match lightos|duros|etcd|management- = FAIL, phải cài bổ sung từ /tmp/rpms trước khi đi tiếp

[NEW] — Kernel phải giống với REF

# REF
[root@oss-nvme-01 ~]# uname -r
5.14.0-687.5.3.el9_8.x86_64

# NEW
[root@oss-nvme-05 ~]# uname -r
5.14.0-687.5.3.el9_8.x86_64

BƯỚC 2 — Tạo cert

Do CA + key CA nằm ở ADMIN nên quy trình sẽ sinh cert tại ADMIN

[REF] — Cho ADMIN biet clusterId

[root@oss-nvme-01 ~]# grep clusterId /etc/cluster-manager/cluster-manager.yaml
clusterId: f49c7acd-4ffd-59ab-91ad-9946a01f2cba

[ADMIN] — Tạo key + cert mới, ký bằng CA của cluster

Set CLUSTER_ID

CLUSTER_ID=f49c7acd-4ffd-59ab-91ad-9946a01f2cba

Lưu lại cert cũ nếu có

cd $CERTS
mkdir -p old-$NEWHOST && mv $NEWHOST-cert-etcd-peer*.pem old-$NEWHOST/ 2>/dev/null

Verify

[root@vm-admin-01 lightos-certificates-clusterNew]# ls -lh old-$NEWHOST/
total 12K
-rw-------. 1 root root 1.2K Jul 31 15:57 oss-nvme-05-cert-etcd-peer-csr.pem
-rw-------. 1 root root 1.7K Jul 31 15:57 oss-nvme-05-cert-etcd-peer-key.pem
-rw-------. 1 root root 1.8K Jul 31 15:57 oss-nvme-05-cert-etcd-peer.pem

Tạo file cnf khai báo thông tin cấp cert — openssl đọc nó để biết thông tin tạo ra cert mới

cat > /tmp/$NEWHOST.cnf <<EOF
[req]
default_bits = 2048
distinguished_name = req_distinguished_name
req_extensions = req_ext
x509_extensions = v3_req
prompt = no
[req_distinguished_name]
countryName = IL
stateOrProvinceName = N/A
localityName = KS
organizationName = Lightbits
commonName = $NEWHOST
serialNumber = $CLUSTER_ID
[req_ext]
subjectAltName = @alt_names
[v3_req]
subjectAltName = @alt_names
basicConstraints = CA:FALSE
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer:always
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth,serverAuth
[alt_names]
IP.1=$NEW_DATA_IP
IP.2=127.0.0.1
DNS.1=$NEWHOST
DNS.2=localhost
EOF

Verify

[root@vm-admin-01 lightos-certificates-clusterNew]# cat /tmp/$NEWHOST.cnf
[req]
default_bits = 2048
distinguished_name = req_distinguished_name
req_extensions = req_ext
x509_extensions = v3_req
prompt = no
[req_distinguished_name]
countryName = IL
stateOrProvinceName = N/A
localityName = KS
organizationName = Lightbits
commonName = oss-nvme-05
serialNumber = f49c7acd-4ffd-59ab-91ad-9946a01f2cba
[req_ext]
subjectAltName = @alt_names
[v3_req]
subjectAltName = @alt_names
basicConstraints = CA:FALSE
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer:always
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth,serverAuth
[alt_names]
IP.1=10.237.95.105
IP.2=127.0.0.1
DNS.1=oss-nvme-05
DNS.2=localhost

Sinh KEY mới, xin cấp csr theo cnf

# Sinh key
openssl req -nodes -newkey rsa:2048 -config /tmp/$NEWHOST.cnf -keyout $NEWHOST-cert-etcd-peer-key.pem -out $NEWHOST-cert-etcd-peer-csr.pem

# Verify
[root@vm-admin-01 lightos-certificates-clusterNew]# ls -l  $NEWHOST-cert-etcd-peer-*
-rw-r--r--. 1 root root 1127 Aug  1 12:53 oss-nvme-05-cert-etcd-peer-csr.pem
-rw-------. 1 root root 1704 Aug  1 12:53 oss-nvme-05-cert-etcd-peer-key.pem

Sinh CA (Certificate Authority) của cụm = cặp file etcd-ca.pem (cert gốc) + etcd-ca-key.pem (khóa ký), sinh đúng một lần lúc deploy cụm, nằm ở admin. Nó là con dấu chung của cluster. Trong Lightbits các etcd member nói chuyện với nhau qua mTLS — mỗi kết nối peer (port 2380), hai bên đều phải có cert và kiểm cert của nhau bằng cùng một câu hỏi dạng cert có do CA của cluster ký không? (unit file: --peer-trusted-ca-file=/etc/etcd/ssl/etcd-ca.pem, --peer-client-cert-auth=true). Nếu ok, tiếp tục đi vào nội bộ etcd; nếu không có cert do CA này ký sẽ bị từ chối bắt tay ngay ở tầng TLS.

# Command
openssl x509 -req -days 3650 -set_serial 0x$(openssl rand -hex 20) \
  -in $NEWHOST-cert-etcd-peer-csr.pem -out $NEWHOST-cert-etcd-peer.pem \
  -extensions v3_req -extfile /tmp/$NEWHOST.cnf -CA etcd-ca.pem -CAkey etcd-ca-key.pem

# Output
[root@vm-admin-01 lightos-certificates-clusterNew]# openssl x509 -req -days 3650 -set_serial 0x$(openssl rand -hex 20) \
  -in $NEWHOST-cert-etcd-peer-csr.pem -out $NEWHOST-cert-etcd-peer.pem \
  -extensions v3_req -extfile /tmp/$NEWHOST.cnf -CA etcd-ca.pem -CAkey etcd-ca-key.pem
Certificate request self-signature ok
subject=C=IL, ST=N/A, L=KS, O=Lightbits, CN=oss-nvme-05, serialNumber=f49c7acd-4ffd-59ab-91ad-9946a01f2cba

Phân quyền 

chmod 600 $NEWHOST-cert-etcd-peer*

Verify 3 điều (quyết định cert đúng/sai — không đối chiếu đồ cũ)

[root@vm-admin-01 lightos-certificates-clusterNew]# openssl x509 -in $NEWHOST-cert-etcd-peer.pem -noout -text | grep -A2 'Subject Alternative Name'
            X509v3 Subject Alternative Name:
                IP Address:10.237.95.105, IP Address:127.0.0.1, DNS:oss-nvme-05, DNS:localhost
            X509v3 Basic Constraints:

[root@vm-admin-01 lightos-certificates-clusterNew]# openssl verify -CAfile etcd-ca.pem $NEWHOST-cert-etcd-peer.pem
oss-nvme-05-cert-etcd-peer.pem: OK

[root@vm-admin-01 lightos-certificates-clusterNew]# diff <(openssl x509 -in $NEWHOST-cert-etcd-peer.pem -noout -pubkey) <(openssl pkey -in $NEWHOST-cert-etcd-peer-key.pem -pubout) && echo "OK"
OK

[ADMIN] — Nếu mọi thứ ổn cả, copy chúng sáng NEW

ssh root@$NEW 'mkdir -p /etc/etcd/ssl && chmod 700 /etc/etcd/ssl'
scp etcd-ca.pem etcd-ca-key.pem root@$NEW:/etc/etcd/ssl/
scp $NEWHOST-cert-etcd-peer.pem root@$NEW:/etc/etcd/ssl/cert-etcd-peer.pem
scp $NEWHOST-cert-etcd-peer-key.pem root@$NEW:/etc/etcd/ssl/cert-etcd-peer-key.pem

[NEW] — Chmod lại 600 trên NEW

[root@oss-nvme-05 ~]# chmod 600 /etc/etcd/ssl/*
[root@oss-nvme-05 ~]# ls -la /etc/etcd/ssl/
total 16
drwx------. 2 root root  104 Jul  9 14:59 .
drwx------. 3 root root   17 Jul  9 14:59 ..
-rw-------. 1 root root 1704 Aug  1 13:03 cert-etcd-peer-key.pem
-rw-------. 1 root root 1757 Aug  1 13:03 cert-etcd-peer.pem
-rw-------. 1 root root 1675 Aug  1 13:03 etcd-ca-key.pem
-rw-------. 1 root root 1671 Aug  1 13:03 etcd-ca.pem

Kết quả: - SAN đủ 4 mục: 10.237.95.105, 127.0.0.1, oss-nvme-05, localhost ✅ - verify: OK — đúng CA cụm ký ✅ - Key-cert match ✅ - 4 file nằm đúng /etc/etcd/ssl/, mode 600, thư mục 700 ✅ (timestamp Aug 1 13:03 = bộ mới vừa sinh, không phải bộ cũ)

BƯỚC 3 — ETCD join

[NEW] - Chuẩn bị trên node mới

Set ETCDCTL_API=3, vì LightOS chỉ sử dụng ETCD API Version 3, nên mọi phiên làm việc phải khóa sẵn v3

[root@oss-nvme-05 ~]# grep -q ETCDCTL_API ~/.bashrc || sed -i "1i export ETCDCTL_API=3" ~/.bashrc
[root@oss-nvme-05 ~]# head -1 ~/.bashrc
export ETCDCTL_API=3

Tạo file /var/lib/etcd và phân quyền 700 cho nó

mkdir -p /var/lib/etcd
chmod 700 /var/lib/etcd

[ADMIN] — Chuyển unit file từ REF sang NEW

scp root@$REF:/etc/systemd/system/etcd.service /tmp/etcd.service.orig
scp /tmp/etcd.service.orig root@$NEW:/tmp/etcd.service.orig

[REF] — Build danh sách initial-cluster từ member list đang có

[root@oss-nvme-01 ~]# etcdctl member list | grep -v unstarted | awk -F', ' '{printf "%s%s=%s",sep,$3,$4; sep=","}'
oss-nvme-04=https://10.237.95.104:2380,oss-nvme-02=https://10.237.95.102:2380,oss-nvme-03=https://10.237.95.103:2380,oss-nvme-01=https://10.237.95.101:2380

[NEW] — Sed unit file

Set biến INITIAL_CLUSTER

[root@oss-nvme-05 ~]# INITIAL_CLUSTER='oss-nvme-04=https://10.237.95.104:2380,oss-nvme-02=https://10.237.95.102:2380,oss-nvme-03=https://10.237.95.103:2380,oss-nvme-01=https://10.237.95.101:2380,'"$NEWHOST=https://$NEW_DATA_IP:2380"
[root@oss-nvme-05 ~]# echo "$INITIAL_CLUSTER"
oss-nvme-04=https://10.237.95.104:2380,oss-nvme-02=https://10.237.95.102:2380,oss-nvme-03=https://10.237.95.103:2380,oss-nvme-01=https://10.237.95.101:2380,oss-nvme-05=https://10.237.95.105:2380

Verify các biến đã được gán

[root@oss-nvme-05 ~]# echo "NEWHOST=$NEWHOST NEW_DATA_IP=$NEW_DATA_IP REF_HOST=$REF_HOST REF_DATA_IP=$REF_DATA_IP"
NEWHOST=oss-nvme-05 NEW_DATA_IP=10.237.95.105 REF_HOST=oss-nvme-01 REF_DATA_IP=10.237.95.101

Sau khi thu thập đầy đủ các biến, dùng sed để thay thế các biến trong unit file 

sed -e "s/--name=$REF_HOST/--name=$NEWHOST/" \
    -e "s/${REF_DATA_IP//./\\.}/$NEW_DATA_IP/g" \
    -e "s/--initial-cluster-state=new/--initial-cluster-state=existing/" \
    -e "s#--initial-cluster=[^ ]*#--initial-cluster=$INITIAL_CLUSTER#" \
    /tmp/etcd.service.orig > /etc/systemd/system/etcd.service

Verify

[root@oss-nvme-05 ~]# grep -E 'name=|initial-cluster|urls' /etc/systemd/system/etcd.service
  --advertise-client-urls=http://localhost:2379,https://10.237.95.105:2379 \
  --initial-advertise-peer-urls=https://10.237.95.105:2380 \
  --initial-cluster=oss-nvme-04=https://10.237.95.104:2380,oss-nvme-02=https://10.237.95.102:2380,oss-nvme-03=https://10.237.95.103:2380,oss-nvme-01=https://10.237.95.101:2380,oss-nvme-05=https://10.237.95.105:2380 \
  --initial-cluster-state=existing \
  --initial-cluster-token=etcd-cluster-0 \
  --listen-client-urls=http://localhost:2379,https://10.237.95.105:2379 \
  --listen-peer-urls=https://10.237.95.105:2380 \
  --name=oss-nvme-05 \

Config Firewalld nếu có

[root@oss-nvme-05 ~]# systemctl is-active firewalld
active
[root@oss-nvme-05 ~]# firewall-cmd --add-port=2379/tcp --add-port=2380/tcp --permanent
Warning: ALREADY_ENABLED: 2379:tcp
Warning: ALREADY_ENABLED: 2380:tcp
success
[root@oss-nvme-05 ~]# firewall-cmd --reload
success

[REF] — Member add

Add NEW vào ETCD

[root@oss-nvme-01 ~]# etcdctl member add $NEWHOST --peer-urls=https://$NEW_DATA_IP:2380
Member 35049cb5f985d69e added to cluster 38972ecc953e575c
ETCD_NAME="oss-nvme-05"
ETCD_INITIAL_CLUSTER="oss-nvme-04=https://10.237.95.104:2380,oss-nvme-05=https://10.237.95.105:2380,oss-nvme-02=https://10.237.95.102:2380,oss-nvme-03=https://10.237.95.103:2380,oss-nvme-01=https://10.237.95.101:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.237.95.105:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"

NEW xuất hiện với trạng thái unstarted

[root@oss-nvme-01 ~]# etcdctl member list
2969ce575aecaac3, started, oss-nvme-04, https://10.237.95.104:2380, http://localhost:2379,https://10.237.95.104:2379, false
35049cb5f985d69e, unstarted, , https://10.237.95.105:2380, , false
49675b756efdbdb4, started, oss-nvme-02, https://10.237.95.102:2380, http://localhost:2379,https://10.237.95.102:2379, false
92786a305fae6ea5, started, oss-nvme-03, https://10.237.95.103:2380, http://localhost:2379,https://10.237.95.103:2379, false
dcea60084de74739, started, oss-nvme-01, https://10.237.95.101:2380, http://localhost:2379,https://10.237.95.101:2379, false

[NEW] — quet ma + start

Kết quả phải rồng

[root@oss-nvme-05 ~]# ps -C etcd -o args=
/usr/bin/etcd --data-dir /var/lib/etcd

Nếu khác rỗng như trên chạy ngay systemctl stop etcd

systemctl stop etcd 2>/dev/null
rm -rf /var/lib/etcd/*

Và start lại

systemctl daemon-reload
systemctl enable --now etcd
systemctl is-active etcd

Verify trên NEW

[root@oss-nvme-05 ~]# ps -o args= -C etcd | tr " " "\n" | grep -E "^--name|initial-cluster-state"
--initial-cluster-state=existing
--name=oss-nvme-05

[REF] — Verify trên REF

[root@oss-nvme-01 ~]# etcdctl member list
2969ce575aecaac3, started, oss-nvme-04, https://10.237.95.104:2380, http://localhost:2379,https://10.237.95.104:2379, false
35049cb5f985d69e, started, oss-nvme-05, https://10.237.95.105:2380, http://localhost:2379,https://10.237.95.105:2379, false
49675b756efdbdb4, started, oss-nvme-02, https://10.237.95.102:2380, http://localhost:2379,https://10.237.95.102:2379, false
92786a305fae6ea5, started, oss-nvme-03, https://10.237.95.103:2380, http://localhost:2379,https://10.237.95.103:2379, false
dcea60084de74739, started, oss-nvme-01, https://10.237.95.101:2380, http://localhost:2379,https://10.237.95.101:2379, false

BƯỚC 4 — Config

[ADMIN] — Kéo config từ REF sang ADMIN rồi đẩy qua NEW

rm -rf /tmp/cfg
mkdir -p /tmp/cfg && cd /tmp/cfg

for d in node-manager cluster-manager api-service upgrade-manager discovery-service lbcli lightbox-exporter profile-generator; do
  scp -q -r root@$REF:/etc/$d ./ && echo "$d"
done

# Verify kèo từ REF về ADMIN thành công
[root@vm-admin-01 cfg]# ls -l ./
total 4
drwxr-xr-x. 2 root root   62 Aug  1 13:37 api-service
drwxr-xr-x. 2 root root   98 Aug  1 13:37 cluster-manager
drwxr-xr-x. 2 root root   74 Aug  1 13:37 discovery-service
drwxr-xr-x. 2 root root   50 Aug  1 13:37 lbcli
drwxr-xr-x. 3 root root  129 Aug  1 13:37 lightbox-exporter
drwxr-xr-x. 3 root root 4096 Aug  1 13:37 node-manager
drwxr-xr-x. 2 root root   96 Aug  1 13:37 profile-generator
drwxr-xr-x. 2 root root  101 Aug  1 13:37 upgrade-manager

# Đẩy từ ADMIN sang NEW
ssh root@$NEW 'rm -rf /tmp/cfg && mkdir -p /tmp/cfg'
scp -q -r /tmp/cfg/* root@$NEW:/tmp/cfg/

# Verify kết quả đẩy từ ADMIN sang NEW thành công
[root@vm-admin-01 cfg]# ssh root@$NEW 'ls -lh /tmp/cfg'
total 4.0K
drwxr-xr-x. 2 root root   62 Aug  1 13:39 api-service
drwxr-xr-x. 2 root root   98 Aug  1 13:39 cluster-manager
drwxr-xr-x. 2 root root   74 Aug  1 13:39 discovery-service
drwxr-xr-x. 2 root root   50 Aug  1 13:39 lbcli
drwxr-xr-x. 3 root root  129 Aug  1 13:39 lightbox-exporter
drwxr-xr-x. 3 root root 4.0K Aug  1 13:39 node-manager
drwxr-xr-x. 2 root root   96 Aug  1 13:39 profile-generator
drwxr-xr-x. 2 root root  101 Aug  1 13:39 upgrade-manager

[NEW] — Thực hiện lấy OLD_* trên NEW

# Lấy OLD_*
cd /tmp/cfg
OLD_SUUID=$(grep '^serverUUID:' node-manager/node-manager.yaml | awk '{print $2}')
OLD_NUUID=$(grep -m1 '^- uuid:' node-manager/node-manager.yaml | awk '{print $3}')
OLD_CMID=$(grep '^id:' cluster-manager/cluster-manager.yaml | awk '{print $2}' | tr -d '"')
SUUID=$(uuidgen)
NUUID=$(uuidgen)

# Verify
[root@oss-nvme-05 cfg]# echo "OLD: $OLD_SUUID / $OLD_NUUID / $OLD_CMID"
OLD: a0d5bf2a-2ef6-5ff9-953b-592531939f78 / e8bc0737-a4af-5538-8952-3cdf63a20f96 / f8297806-dfd0-4aba-b8d3-6ed4a0d799bc

[root@oss-nvme-05 cfg]# echo "NEW: SUUID=$SUUID NUUID=$NUUID"
NEW: SUUID=35b83b77-a97d-43a5-8325-39f84ac67605 NUUID=8861a8fc-b72b-4129-b9e9-1fe77a5a6e82

for v in NEWHOST NEW_DATA_IP REF_HOST REF_DATA_IP SUUID NUUID OLD_SUUID OLD_NUUID OLD_CMID; do
  [ -z "${!v}" ] && echo "→ Warning: Variable $v is empty. "
done

[NEW] — Sed + reset các trường runtime

grep -rl "$REF_HOST" . | xargs -r sed -i "s/$REF_HOST/$NEWHOST/g"
grep -rl "$REF_DATA_IP" . | xargs -r sed -i "s/${REF_DATA_IP//./\\.}/$NEW_DATA_IP/g"
grep -rl "$OLD_SUUID" . | xargs -r sed -i "s/$OLD_SUUID/$SUUID/g"
grep -rl "$OLD_NUUID" . | xargs -r sed -i "s/$OLD_NUUID/$NUUID/g"
sed -i "s/^id: $OLD_CMID/id: \"\"/" cluster-manager/cluster-manager.yaml
printf 'profileGeneratorVersion: 0\n' > profile-generator/system-profile.yaml
rm -f node-manager/gftl.db node-manager/node-manager.yaml.rpmsave

[NEW] — Verify

Kỳ vọng trống vì đã thay thế bởi NEW

[root@oss-nvme-05 cfg]# grep -rn "$REF_HOST\|${REF_DATA_IP//./\\.}\|$OLD_SUUID\|$OLD_NUUID" . | wc -l
0

Kỳ vọng là thông tin NEW

[root@oss-nvme-05 cfg]# grep -E '^name:|serverUUID:' node-manager/node-manager.yaml
name: oss-nvme-05
serverUUID: 35b83b77-a97d-43a5-8325-39f84ac67605

Kỳ vọng serverName là NEW, id rỗng và clusterId không đổi

[root@oss-nvme-05 cfg]# grep -E 'serverName:|^id:|clusterId:' cluster-manager/cluster-manager.yaml
serverName: oss-nvme-05
id: ""
clusterId: f49c7acd-4ffd-59ab-91ad-9946a01f2cba

Kỳ vọng có SUUID server NEW

[root@oss-nvme-05 cfg]# grep -iE 'serverUUID' discovery-service/*.yaml
serverUUID: 35b83b77-a97d-43a5-8325-39f84ac67605

Copy config đã lấy từ REF vào /etc/

cp -r /tmp/cfg/* /etc/

Nếu gặp config cũ đã tồn tại như dưới ở bước này, nên dọn nó trước khi copy mới

[root@oss-nvme-05 cfg]# cp -r /tmp/cfg/* /etc/
cp: overwrite '/etc/api-service/api-service.yaml'?

Sử dụng for để dọn cho nhanh sau đó copy lại

for d in node-manager cluster-manager api-service upgrade-manager discovery-service lbcli lightbox-exporter profile-generator; do
  rm -rf /etc/$d
done

cp -r /tmp/cfg/* /etc/

File gftl.db phải chắc chắn không tồn tại mới OK vì node mới sẽ phải tạo gftl.db mới từ đầu

[root@oss-nvme-05 cfg]# ls /etc/node-manager/gftl.db 2>/dev/null && echo "!! CON GFTL.DB" || echo OK
OK

BƯỚC 5 — [NEW] Start đúng thứ tự

### [NEW] ###
# 5.1 discovery-service
systemctl daemon-reload
systemctl enable --now discovery-service
systemctl is-active discovery-service

# 5.2 Registry phải có key $SUUID trước khi node-manager chạy lần đầu, lần này là 5 key tương đương với 5 node (trước đó là 4 key)
[root@oss-nvme-05 ~]# etcdctl get --prefix /registry/discovery-service --keys-only
/registry/discovery-service/35b83b77-a97d-43a5-8325-39f84ac67605
/registry/discovery-service/88940976-0aa4-5506-852b-e5861d3816cb
/registry/discovery-service/a0d5bf2a-2ef6-5ff9-953b-592531939f78
/registry/discovery-service/a173b57d-648b-4e52-9c14-a6c04dfd6c8b
/registry/discovery-service/a17b6c72-5a59-5c33-a699-61bdc24b9b54

# Có thể dùng cách này so sánh với output trene
[root@oss-nvme-05 ~]# echo /registry/discovery-service/$SUUID
/registry/discovery-service/35b83b77-a97d-43a5-8325-39f84ac67605

# 5.3 Start các service còn lại
[root@oss-nvme-05 ~]# for s in profile-generator node-manager cluster-manager upgrade-manager api-service; do
  systemctl enable --now $s
  sleep 5
  echo -n "$s: "systemctl is-active $s
done
Created symlink /etc/systemd/system/multi-user.target.wants/profile-generator.service → /usr/lib/systemd/system/profile-generator.service.
profile-generator: systemctl is-active profile-generatorCreated symlink /etc/systemd/system/multi-user.target.wants/node-manager.service → /usr/lib/systemd/system/node-manager.service.
node-manager: systemctl is-active node-managerCreated symlink /etc/systemd/system/multi-user.target.wants/cluster-manager.service → /usr/lib/systemd/system/cluster-manager.service.
cluster-manager: systemctl is-active cluster-managerCreated symlink /etc/systemd/system/multi-user.target.wants/upgrade-manager.service → /usr/lib/systemd/system/upgrade-manager.service.
upgrade-manager: systemctl is-active upgrade-managerCreated symlink /etc/systemd/system/multi-user.target.wants/api-service.service → /usr/lib/systemd/system/api-service.service.

# Tất cả đều sẽ phải active ngoại trừ profile-generator
[root@oss-nvme-05 ~]# for s in discovery-service profile-generator node-manager cluster-manager upgrade-manager api-service; do
  echo -n "$s: "; systemctl is-active $s
done
discovery-service: active
profile-generator: inactive
node-manager: active
cluster-manager: active
upgrade-manager: active

# 5.4 Check log nếu cần
journalctl -fu node-manager

# CANH BAO: dong bang sau loat "detected new pg" -> systemctl stop node-manager NGAY

BƯỚC 6 — Verify

[NEW]

Check md journaling (raid0)

[root@oss-nvme-05 ~]# grep -E "^md" /proc/mdstat
md126 : active raid0 nvme2n1[1] nvme0n1[0]
md127 : active raid1 sda2[0] sdb1[1]

Check service duroslight-0 phải active

[root@oss-nvme-05 ~]# systemctl is-active duroslight-0
active

### [REF] ###
[root@oss-nvme-01 ~]# etcdctl member list
2969ce575aecaac3, started, oss-nvme-04, https://10.237.95.104:2380, http://localhost:2379,https://10.237.95.104:2379, false
35049cb5f985d69e, started, oss-nvme-05, https://10.237.95.105:2380, http://localhost:2379,https://10.237.95.105:2379, false
49675b756efdbdb4, started, oss-nvme-02, https://10.237.95.102:2380, http://localhost:2379,https://10.237.95.102:2379, false
92786a305fae6ea5, started, oss-nvme-03, https://10.237.95.103:2380, http://localhost:2379,https://10.237.95.103:2379, false
dcea60084de74739, started, oss-nvme-01, https://10.237.95.101:2380, http://localhost:2379,https://10.237.95.101:2379, false

### [ADMIN] ###
[root@vm-admin-01 cfg]# lbcli --endpoint $ENDPOINT -J "$LIGHTOS_JWT" list servers
NAME          UUID                                   State     RiskOfServiceLoss State   LightOSVersion
oss-nvme-01   a0d5bf2a-2ef6-5ff9-953b-592531939f78   Enabled   NoRiskOfServiceLoss       3.18.2b26868114363
oss-nvme-02   a17b6c72-5a59-5c33-a699-61bdc24b9b54   Enabled   NoRiskOfServiceLoss       3.18.2b26868114363
oss-nvme-03   88940976-0aa4-5506-852b-e5861d3816cb   Enabled   NoRiskOfServiceLoss       3.18.2b26868114363
oss-nvme-04   a173b57d-648b-4e52-9c14-a6c04dfd6c8b   Enabled   NoRiskOfServiceLoss       3.18.2b26868114363
oss-nvme-05   35b83b77-a97d-43a5-8325-39f84ac67605   Enabled   NoRiskOfServiceLoss       3.18.2b26868114363

[root@vm-admin-01 cfg]# lbcli --endpoint $ENDPOINT -J "$LIGHTOS_JWT" list nodes
Name            UUID                                   State     NVMe endpoint        Failure domains   Capacity   Used      Local rebuild progress
oss-nvme-01-0   e8bc0737-a4af-5538-8952-3cdf63a20f96   Active    10.237.95.101:4420   [oss-nvme-01]     61 TiB     0 B       None
oss-nvme-02-0   ef732732-4c12-5e22-8bf4-eb3dddbd0cc7   Active    10.237.95.102:4420   [oss-nvme-02]     61 TiB     0 B       None
oss-nvme-03-0   5c5b6d5d-21c0-51e2-9c02-359936ae9c70   Active    10.237.95.103:4420   [oss-nvme-03]     61 TiB     0 B       None
oss-nvme-04-0   f2a4f333-9aaf-45fe-bcab-d479c20bb634   Active    10.237.95.104:4420   [oss-nvme-04]     61 TiB     0 B       None
oss-nvme-05-0   8861a8fc-b72b-4129-b9e9-1fe77a5a6e82   Active    10.237.95.105:4420   [oss-nvme-05]     61 TiB     0 B       None

# 6 Data + 2 Journal
[root@vm-admin-01 cfg]# lbcli --endpoint $ENDPOINT -J "$LIGHTOS_JWT" list nvme-devices | grep 8861a8fc-b72b-4129-b9e9-1fe77a5a6e82 | awk '{print $NF}' | sort | uniq -c
      6 Data
      2 Journal

Bản đồ "việc gì thuộc máy nào"

Việc Máy Vì sao
lbcli (mọi thao tác cluster-level) ADMIN JWT nằm ở admin; endpoint trỏ node sống
Sinh cert ADMIN CA key (etcd-ca-key.pem) chỉ nằm ở admin — không bao giờ rời admin
Đọc clusterId, build initial-cluster, etcdctl member add/list REF phải hỏi thành viên đang sống của cluster
Cài package, sed unit/config, start services, checkpoint 5.2 NEW việc của chính node; etcdctl ở 5.2 chạy được vì etcd local đã join
scp bundle ADMIN → NEW bundle nằm ở admin
scp cert ADMIN → NEW cert sinh ở admin
scp unit + config ADMIN trung chuyển REF → NEW key SSH hình sao: chỉ ADMIN có key tới mọi node; NEW và REF không tin nhau

© Copyright Infra Notes contributors. Last updated on September 07, 2026.