Yêu cầu 3 điều kiện: giữ nguyên settings Ansible gốc + generated files, master jwt chung giữa các node và chạy từ thư mục đã extract light-app — hiện tại có đủ cả 3 (~/light-app, ~/lightos-system-jwt, ~/lightos-certificates-clusterNew).
Bước 0 — Chuẩn bị node 04, đây là điều kiện bắt buộc với offline
Lý do là node 04 từng làm cluster test 1-node + client, nó phải sạch sẽ và offline như 01/02/03 trước đây:
ssh root@10.237.94.104 '
# Go package LightOS cu
dnf remove -y $(rpm -qa | grep -iE "lightos|duros|node-manager|cluster-manager|api-service|discovery|profile-generator|upgrade-manager|lightbox|lbcli|etcd" | tr "\n" " ") 2>&1 | tail -2
rm -rf /var/lib/etcd /etc/lightos* 2>/dev/null
# Wipe 8 o NVMe (KHONG dung OS disk)
for m in /dev/md12[0-6]; do [ -e "$m" ] && mdadm --detail "$m" 2>/dev/null | grep -q raid0 && mdadm --stop "$m"; done
for d in 0 1 2 3 4 5 6 7; do wipefs -a /dev/nvme${d}n1 >/dev/null 2>&1; nvme format /dev/nvme${d}n1 --lbaf=1 --force >/dev/null 2>&1; done
# Tat repo online (ep offline)
cd /etc/yum.repos.d && for f in *.repo; do [ -e "$f" ] && mv "$f" "$f.disabled"; done
echo -n "pkg:"; rpm -qa|grep -icE "lightos|duros|node-manager"
echo -n " disks:"; lsblk|grep -cE "nvme[0-7]n1"
echo -n " repos:"; ls *.repo 2>/dev/null|wc -l'
Kỳ vọng pkg:0 disks:8 repos:0.
Bước 1 — Sửa inventory để thêm 04 vào 3 chỗ trong hosts
Thêm thông tin server mới vào 3 section của hosts file — top section, [duros_nodes] và [etcd]; hosts file phải chứa toàn bộ các node hiện có dưới [etcd] để nhận biết cluster cần join.
cat > ~/light-app/ansible/inventories/clusterNew/hosts << 'EOF'
oss-nvme-01 ansible_host=10.237.94.101 ansible_connection=ssh ansible_ssh_user=root ansible_become_user=root
oss-nvme-02 ansible_host=10.237.94.102 ansible_connection=ssh ansible_ssh_user=root ansible_become_user=root
oss-nvme-03 ansible_host=10.237.94.103 ansible_connection=ssh ansible_ssh_user=root ansible_become_user=root
oss-nvme-04 ansible_host=10.237.94.104 ansible_connection=ssh ansible_ssh_user=root ansible_become_user=root
[duros_nodes]
oss-nvme-01
oss-nvme-02
oss-nvme-03
oss-nvme-04
[etcd]
oss-nvme-01
oss-nvme-02
oss-nvme-03
oss-nvme-04
EOF
Doc có lưu ý mỗi lần chỉ thêm được một server và hosts file chỉ được chứa các node hiện có + đúng một node mới — 3 cũ + 1 mới là hợp lệ.
Bước 2 — Tạo host_vars cho 04 (layout giống 3 node kia: 6 data + 2 journal, EC on)
cat > ~/light-app/ansible/inventories/clusterNew/host_vars/oss-nvme-04.yml << 'EOF'
---
name: oss-nvme-04
nodes:
- instanceID: 0
data_ip: 10.237.95.104
ec_enabled: true
failure_domains:
- oss-nvme-04
storageDeviceLayout:
allowCrossNumaDevices: false
deviceMatchers:
- partition == false
initialDeviceCount: 6
maxDeviceCount: 6
journalDeviceLayout:
deviceMatchers:
- partition == false
numDevices: 2
EOF
Lưu ý, doc có nói không thể thêm node dùng SSD Journaling vào cluster không dùng SSD Journaling — cluster hiện tại cả 3 node đều SSD journaling nên 04 giống hệt là ok.
Bước 3 — Pre-check nhanh
cd ~/light-app
INV=ansible/inventories/clusterNew/hosts
ansible -i $INV all -m ping | grep -E 'SUCCESS|UNREACHABLE'
# Cluster hien tai van khoe truoc khi dung vao
source ~/lightos-system-jwt
lbcli --endpoint https://10.237.94.101:443 -J "$LIGHTOS_JWT" list nodes
Kết quả 3 node hiện tại phải Active hết rồi mới thêm node mới.
Bước 4 — Chạy playbook với --limit + 2 flag etcd
Chạy ansible-playbook với tham số như lần deploy trước, thêm -e new_etcd_member=true -e extend_etcd_cluster=true --limit <tên server mới>:
cd ~/light-app
ansible-playbook -i ansible/inventories/clusterNew/hosts \
playbooks/deploy-lightos.yml -vvv \
-e new_etcd_member=true \
-e extend_etcd_cluster=true \
--limit oss-nvme-04 2>&1 | tee ~/add-node04.log
Lưu ý:
- --limit oss-nvme-04 dùng tên friendly ở cột đầu hosts file, không phải hostname hay IP.
- Lệnh này dựa vào các file jwt và thư mục lightos-certificate đã tạo lúc deploy, nằm ở $HOME là /root/lightos-system-jwt và certificates_directory trong all.yml đã trỏ ~/lightos-certificates-clusterNew, khỏi thêm -e.
- source_type: offline nằm trong group_vars/all.yml nên node 04 sẽ cài từ bundle ~/lightos-offline như 3 node kia.
- all.yml đã có ipacl_explicit_allow: false, IPACL sẽ tắt ngay từ lúc deploy cluster.
Bước 5 — Verify sau khi xong
Cài xong server mới sẽ reboot, etcd mới sync với cluster hiện có, cluster tự động mở rộng. Đợi 04 reboot xong rồi:
source ~/lightos-system-jwt
EP=https://10.237.94.101:443
lbcli --endpoint $EP -J "$LIGHTOS_JWT" list nodes
lbcli --endpoint $EP -J "$LIGHTOS_JWT" list nvme-devices | head -40
ssh root@10.237.94.101 'etcdctl member list'
Kết quả
[root@vm-admin-01 light-app]# source ~/lightos-system-jwt
[root@vm-admin-01 light-app]# EP=https://10.237.94.101:443
[root@vm-admin-01 light-app]# lbcli --endpoint $EP -J "$LIGHTOS_JWT" list nodes
Name UUID State NVMe endpoint Failure domains Capacity Used Local rebuild progress
oss-nvme-01-0 e742dcaf-c905-5659-b91b-97df007ecfc5 Active 10.237.95.101:4420 [oss-nvme-01] 61 TiB 0 B None
oss-nvme-02-0 4638f9a7-9fd5-58ae-8274-d7abdec644d1 Active 10.237.95.102:4420 [oss-nvme-02] 61 TiB 0 B None
oss-nvme-03-0 efa65c6d-ec40-505a-b087-0de72a26f5ca Active 10.237.95.103:4420 [oss-nvme-03] 61 TiB 0 B None
oss-nvme-04-0 a16923bd-bfc2-5cb0-8df9-024c2634bba6 Active 10.237.95.104:4420 [oss-nvme-04] 61 TiB 0 B None
[root@vm-admin-01 light-app]# lbcli --endpoint $EP -J "$LIGHTOS_JWT" list nvme-devices | head -40
Name Size NUMA ID Serial State Server UUID Node UUID Usage
nvme1n1 14 TiB 0 S6VANN0L105948 Healthy d8eaed10-dade-513f-afb2-83bf7c684321 e742dcaf-c905-5659-b91b-97df007ecfc5 Data
nvme5n1 14 TiB 0 S6VANN0L105908 Healthy d8eaed10-dade-513f-afb2-83bf7c684321 e742dcaf-c905-5659-b91b-97df007ecfc5 Journal
nvme3n1 14 TiB 0 S6VANN0L105906 Healthy d8eaed10-dade-513f-afb2-83bf7c684321 e742dcaf-c905-5659-b91b-97df007ecfc5 Data
nvme2n1 14 TiB 0 S6VANN0L105907 Healthy d8eaed10-dade-513f-afb2-83bf7c684321 e742dcaf-c905-5659-b91b-97df007ecfc5 Data
nvme7n1 14 TiB 0 S6VANN0L105910 Healthy d8eaed10-dade-513f-afb2-83bf7c684321 e742dcaf-c905-5659-b91b-97df007ecfc5 Data
nvme6n1 14 TiB 0 S6VANN0L105909 Healthy d8eaed10-dade-513f-afb2-83bf7c684321 e742dcaf-c905-5659-b91b-97df007ecfc5 Data
nvme4n1 14 TiB 0 S6VANN0L107087 Healthy d8eaed10-dade-513f-afb2-83bf7c684321 e742dcaf-c905-5659-b91b-97df007ecfc5 Journal
nvme0n1 14 TiB 0 S6VANN0L105949 Healthy d8eaed10-dade-513f-afb2-83bf7c684321 e742dcaf-c905-5659-b91b-97df007ecfc5 Data
nvme7n1 14 TiB 0 S6VANN0L106479 Healthy f1506a9c-4a55-5948-b904-c3ea35da4fc8 4638f9a7-9fd5-58ae-8274-d7abdec644d1 Data
nvme1n1 14 TiB 0 S6VANN0L107098 Healthy f1506a9c-4a55-5948-b904-c3ea35da4fc8 4638f9a7-9fd5-58ae-8274-d7abdec644d1 Data
nvme2n1 14 TiB 0 S6VANN0L106516 Healthy f1506a9c-4a55-5948-b904-c3ea35da4fc8 4638f9a7-9fd5-58ae-8274-d7abdec644d1 Data
nvme0n1 14 TiB 0 S6VANN0L107100 Healthy f1506a9c-4a55-5948-b904-c3ea35da4fc8 4638f9a7-9fd5-58ae-8274-d7abdec644d1 Journal
nvme4n1 14 TiB 0 S6VANN0L107096 Healthy f1506a9c-4a55-5948-b904-c3ea35da4fc8 4638f9a7-9fd5-58ae-8274-d7abdec644d1 Data
nvme5n1 14 TiB 0 S6VANN0L107259 Healthy f1506a9c-4a55-5948-b904-c3ea35da4fc8 4638f9a7-9fd5-58ae-8274-d7abdec644d1 Journal
nvme3n1 14 TiB 0 S6VANN0L106975 Healthy f1506a9c-4a55-5948-b904-c3ea35da4fc8 4638f9a7-9fd5-58ae-8274-d7abdec644d1 Data
nvme6n1 14 TiB 0 S6VANN0L106470 Healthy f1506a9c-4a55-5948-b904-c3ea35da4fc8 4638f9a7-9fd5-58ae-8274-d7abdec644d1 Data
nvme3n1 14 TiB 0 S6VANN0L107035 Healthy e62ef023-6c7b-536c-9a3c-5da3bf8bb2f1 a16923bd-bfc2-5cb0-8df9-024c2634bba6 Data
nvme4n1 14 TiB 0 S6VANN0L107039 Healthy e62ef023-6c7b-536c-9a3c-5da3bf8bb2f1 a16923bd-bfc2-5cb0-8df9-024c2634bba6 Data
nvme0n1 14 TiB 0 S6VANN0L107019 Healthy e62ef023-6c7b-536c-9a3c-5da3bf8bb2f1 a16923bd-bfc2-5cb0-8df9-024c2634bba6 Data
nvme2n1 14 TiB 0 S6VANN0L107032 Healthy e62ef023-6c7b-536c-9a3c-5da3bf8bb2f1 a16923bd-bfc2-5cb0-8df9-024c2634bba6 Data
nvme1n1 14 TiB 0 S6VANN0L107020 Healthy e62ef023-6c7b-536c-9a3c-5da3bf8bb2f1 a16923bd-bfc2-5cb0-8df9-024c2634bba6 Journal
nvme7n1 14 TiB 0 S6VANN0L107042 Healthy e62ef023-6c7b-536c-9a3c-5da3bf8bb2f1 a16923bd-bfc2-5cb0-8df9-024c2634bba6 Data
nvme6n1 14 TiB 0 S6VANN0L107043 Healthy e62ef023-6c7b-536c-9a3c-5da3bf8bb2f1 a16923bd-bfc2-5cb0-8df9-024c2634bba6 Journal
nvme5n1 14 TiB 0 S6VANN0L107036 Healthy e62ef023-6c7b-536c-9a3c-5da3bf8bb2f1 a16923bd-bfc2-5cb0-8df9-024c2634bba6 Data
nvme6n1 14 TiB 0 S6VANN0L107099 Healthy b7ef33ba-df1e-50b1-b1b9-bec29b441dbc efa65c6d-ec40-505a-b087-0de72a26f5ca Data
nvme5n1 14 TiB 0 S6VANN0L107102 Healthy b7ef33ba-df1e-50b1-b1b9-bec29b441dbc efa65c6d-ec40-505a-b087-0de72a26f5ca Data
nvme4n1 14 TiB 0 S6VANN0L107097 Healthy b7ef33ba-df1e-50b1-b1b9-bec29b441dbc efa65c6d-ec40-505a-b087-0de72a26f5ca Data
nvme0n1 14 TiB 0 S6VANN0L107095 Healthy b7ef33ba-df1e-50b1-b1b9-bec29b441dbc efa65c6d-ec40-505a-b087-0de72a26f5ca Data
nvme7n1 14 TiB 0 S6VANN0L107091 Healthy b7ef33ba-df1e-50b1-b1b9-bec29b441dbc efa65c6d-ec40-505a-b087-0de72a26f5ca Data
nvme3n1 14 TiB 0 S6VANN0L105950 Healthy b7ef33ba-df1e-50b1-b1b9-bec29b441dbc efa65c6d-ec40-505a-b087-0de72a26f5ca Journal
nvme1n1 14 TiB 0 S6VANN0L107101 Healthy b7ef33ba-df1e-50b1-b1b9-bec29b441dbc efa65c6d-ec40-505a-b087-0de72a26f5ca Data
nvme2n1 14 TiB 0 S6VANN0L105943 Healthy b7ef33ba-df1e-50b1-b1b9-bec29b441dbc efa65c6d-ec40-505a-b087-0de72a26f5ca Journal
[root@vm-admin-01 light-app]# ssh root@10.237.94.101 'etcdctl member list'
Warning: Permanently added '10.237.94.101' (ED25519) to the list of known hosts.
49675b756efdbdb4, started, oss-nvme-02, https://10.237.95.102:2380, http://localhost:2379,https://10.237.95.102:2379, false
81939c40990fdc83, started, oss-nvme-04, https://10.237.95.104:2380, http://localhost:2379,https://10.237.95.104:2379, false
92786a305fae6ea5, started, oss-nvme-03, https://10.237.95.103:2380, http://localhost:2379,https://10.237.95.103:2379, false
dcea60084de74739, started, oss-nvme-01, https://10.237.95.101:2380, http://localhost:2379,https://10.237.95.101:2379, false
Nếu fail giữa chừng — theo doc thì cleanup rồi làm lại
Chạy cleanup playbook với --limit node mới để dọn metadata + software Lightbits, rồi etcdctl member list lấy ID của server fail và etcdctl member remove <id> xóa khỏi etcd:
cd ~/light-app
ansible-playbook -i ansible/inventories/clusterNew/hosts \
playbooks/cleanup-lightos-playbook.yml -t cleanup --limit oss-nvme-04
ssh root@10.237.94.101 'etcdctl member list'
ssh root@10.237.94.101 'etcdctl member remove <ID_cua_04>'
Lưu ý: Việc cleanup chỉ dùng cho server thêm vào bị fail (tức chỉ
--limit oss-nvme-04, đừng bao giờ chạy không limit).
Chạy Bước 0 → 3 trước, paste output pre-check, xanh thì bấm Bước 4.
