Lightbits Client

Mục tiêu: Cài đặt oss-nvme-06 thành client NVMe/TCP

Thông tin


Client	oss-nvme-06 — mgmt 10.237.94.106, data 10.237.95.106
Storage data IP	10.237.95.101 / .102 / .103 (port 4420)
Controller (lbcli)	vm-admin-01 10.237.94.100
Proxy	http://10.237.7.250:3128 (no_proxy 10.237.0.0/16)

Chú ý: Các lệnh ở phần này chỉ chạy trên oss-nvme-06 hoặc trên controller vm-admin-01.

================================================================

PHẦN A — CHUẨN BỊ CLIENT (trên oss-nvme-06)

================================================================

A.0 Cài package, disable toàn bộ service cũ

[CLIENT] — Lấy bundle từ node ADMIN

# Xóa và tạo lại thư mục bundle
rm -rf /tmp/rpms
mkdir -p /tmp/rpms

# Copy bundle từ node ADMIN sang
scp ~/lightos-offline/target_rpms/*.rpm ~/lightos-offline/deps/*.rpm root@$NEW:/tmp/rpms/

[NEW] — Dựng repo local và cài đặt package

yum install -y --nogpgcheck /tmp/rpms/createrepo_c-libs*.rpm /tmp/rpms/createrepo_c-[0-9]*.rpm /tmp/rpms/drpm*.rpm 2>/dev/null
createrepo /tmp/rpms
printf "[lightos]\nname=lightos\nbaseurl=file:///tmp/rpms\ngpgcheck=0\nenabled=1\n" > /etc/yum.repos.d/lightos.repo
dnf clean all

A.1 Kiểm tra kernel (cần >=5.3.5 cho nvme-tcp)

uname -r # AlmaLinux 9 kernel 5.14 → OK

A.2 Kiểm tra data network tới 3 node

[root@oss-nvme-06 ~]# for ip in 10.237.95.101 10.237.95.102 10.237.95.103; do ping -c1 -W1 $ip; done
PING 10.237.95.101 (10.237.95.101) 56(84) bytes of data.
64 bytes from 10.237.95.101: icmp_seq=1 ttl=64 time=0.213 ms

--- 10.237.95.101 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.213/0.213/0.213/0.000 ms
PING 10.237.95.102 (10.237.95.102) 56(84) bytes of data.
64 bytes from 10.237.95.102: icmp_seq=1 ttl=64 time=0.199 ms

--- 10.237.95.102 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.199/0.199/0.199/0.000 ms
PING 10.237.95.103 (10.237.95.103) 56(84) bytes of data.
64 bytes from 10.237.95.103: icmp_seq=1 ttl=64 time=0.182 ms

--- 10.237.95.103 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.182/0.182/0.182/0.000 ms

# MTU9000: test jumbo frame không phân mảnh, nếu fail → MTU path chưa thông 9000
[root@oss-nvme-06 ~]# ping -M do -s 8972 -c2 10.237.95.101
PING 10.237.95.101 (10.237.95.101) 8972(9000) bytes of data.
8980 bytes from 10.237.95.101: icmp_seq=1 ttl=64 time=0.142 ms
8980 bytes from 10.237.95.101: icmp_seq=2 ttl=64 time=0.155 ms

--- 10.237.95.101 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1062ms
rtt min/avg/max/mdev = 0.142/0.148/0.155/0.006 ms

Nếu jumbo fail nhưng ping thường ok bench vẫn được nhưng kết quả sẽ không tốt.

A.3 Cài gói cần thiết

dnf install -y nvme-cli fio

A.4 Nạp module nvme-tcp

[root@oss-nvme-06 ~]# modprobe nvme-tcp
[root@oss-nvme-06 ~]# unset http_proxy; unset https_proxy
[root@oss-nvme-06 ~]# lsmod | grep nvme # kỳ vọng nvme_tcp / nvme_fabrics / nvme_core
nvme                   77824  0
nvme_tcp              102400  0
nvme_fabrics           49152  1 nvme_tcp
nvme_core             274432  3 nvme_tcp,nvme,nvme_fabrics
nvme_keyring           20480  3 nvme_tcp,nvme_core,nvme_fabrics
nvme_auth              36864  1 nvme_core

# boot-persistent
echo nvme_tcp > /etc/modules-load.d/nvme_tcp.conf

A.5 Bật NVMe native multipath

# kiểm tra hiện trạng
[root@oss-nvme-06 ~]# grep -r "" /sys/module/nvme_core/parameters | grep multipath
/sys/module/nvme_core/parameters/multipath:Y

# lấy đường dẫn kernel
[root@oss-nvme-06 ~]# grubby --default-kernel
/boot/vmlinuz-5.14.0-687.5.3.el9_8.x86_64

grubby --args=nvme_core.multipath=Y --update-kernel $(grubby --default-kernel)

Nếu không có grubby chạy echo "options nvme_core multipath=Y" > /etc/modprobe.d/50-nvme_core.conf && dracut -f

A.6 Reboot để áp dụng multipath

reboot

================================================================

PHẦN B — LẤY HOSTNQN + TẠO VOLUME (ACL)

================================================================

B.1 Lấy hostNQN của client (trên oss-nvme-06)

# Cách 1
[root@oss-nvme-06 ~]# cat /etc/nvme/hostnqn
nqn.2014-08.org.nvmexpress:uuid:6f63c000-1464-11f2-8000-30560f906fe2

# Cách 2 khi sử dụng cách 1 file trống/không có:
[root@oss-nvme-06 ~]# nvme gen-hostnqn | tee /etc/nvme/hostnqn
nqn.2014-08.org.nvmexpress:uuid:6f63c000-1464-11f2-8000-30560f906fe2

Ghi lại nqn.2014-08.org.nvmexpress:uuid:.... — đây là ACL để volume cho phép client này.

B.2 Lấy subsystem NQN của cluster (trên CONTROLLER vm-admin-01)

[root@vm-admin-01 ~]# lbcli get cluster
UUID                                   Subsystem NQN                                                             Current max replicas   Supported max replicas   MinVersionInCluster   MinAllowedVersion   MaxAllowedVersion
f49c7acd-4ffd-59ab-91ad-9946a01f2cba   nqn.2016-01.com.lightbitslabs:uuid:a1bf3a50-1b6e-47aa-b5c2-3e8ae454f916   3                      3                        3.18.2b26868114363    3.18.X              3.21.X
# tìm dòng "subsystem NQN" / "discoveryEndpoints"

Ghi lại subsysNQN dạng nqn.2016-01.com.lightbitslabs:uuid:....

B.3 Tạo nhiều volume để bench MAX (trên CONTROLLER)

Bench trần cluster cần nhiều volume rải primary trên 3 node.
Tạo 6 volume replica 2, ACL = hostNQN client.

[root@vm-admin-01 ~]# HOSTNQN="nqn.2014-08.org.nvmexpress:uuid:6f63c000-1464-11f2-8000-30560f906fe2"
[root@vm-admin-01 ~]# for i in $(seq 1 6); do lbcli create volume --name bench-vol-$i --size 200Gib --replica-count 2 --acl "$HOSTNQN" --project-name default; done
Name          UUID                                   State      Protection State   NSID      Size      Replicas   Compression   ACL                                                                             Rebuild Progress   Missing Nodes Access
bench-vol-1   6e329c5f-84fe-40da-9632-33cd3213e045   Creating   Unknown            0         200 GiB   2          false         values:"nqn.2014-08.org.nvmexpress:uuid:6f63c000-1464-11f2-8000-30560f906fe2"
Name          UUID                                   State      Protection State   NSID      Size      Replicas   Compression   ACL                                                                             Rebuild Progress   Missing Nodes Access
bench-vol-2   e05c6c1c-1062-4e26-963d-710cb1fe0ad6   Creating   Unknown            0         200 GiB   2          false         values:"nqn.2014-08.org.nvmexpress:uuid:6f63c000-1464-11f2-8000-30560f906fe2"
Name          UUID                                   State      Protection State   NSID      Size      Replicas   Compression   ACL                                                                             Rebuild Progress   Missing Nodes Access
bench-vol-3   42bda73c-4a12-40e6-9f12-db11f375f896   Creating   Unknown            0         200 GiB   2          false         values:"nqn.2014-08.org.nvmexpress:uuid:6f63c000-1464-11f2-8000-30560f906fe2"
Name          UUID                                   State      Protection State   NSID      Size      Replicas   Compression   ACL                                                                             Rebuild Progress   Missing Nodes Access
bench-vol-4   a1a23dd8-f9fb-4746-b502-b79fc8d31226   Creating   Unknown            0         200 GiB   2          false         values:"nqn.2014-08.org.nvmexpress:uuid:6f63c000-1464-11f2-8000-30560f906fe2"
Name          UUID                                   State      Protection State   NSID      Size      Replicas   Compression   ACL                                                                             Rebuild Progress   Missing Nodes Access
bench-vol-5   30f1050f-79b1-4c35-abc7-5ab148d10a3a   Creating   Unknown            0         200 GiB   2          false         values:"nqn.2014-08.org.nvmexpress:uuid:6f63c000-1464-11f2-8000-30560f906fe2"
Name          UUID                                   State      Protection State   NSID      Size      Replicas   Compression   ACL                                                                             Rebuild Progress   Missing Nodes Access
bench-vol-6   f67d9374-7487-48e0-ac07-bd2c98a9781e   Creating   Unknown            0         200 GiB   2          false         values:"nqn.2014-08.org.nvmexpress:uuid:6f63c000-1464-11f2-8000-30560f906fe2"

[root@vm-admin-01 ~]# lbcli list volumes --project-name default
Name          UUID                                   State       Protection State   NSID      Size      Replicas   Compression   ACL                                                                             Rebuild Progress   Missing Nodes Access
bench-vol-5   30f1050f-79b1-4c35-abc7-5ab148d10a3a   Available   FullyProtected     5         200 GiB   2          false         values:"nqn.2014-08.org.nvmexpress:uuid:6f63c000-1464-11f2-8000-30560f906fe2"   None
bench-vol-3   42bda73c-4a12-40e6-9f12-db11f375f896   Available   FullyProtected     3         200 GiB   2          false         values:"nqn.2014-08.org.nvmexpress:uuid:6f63c000-1464-11f2-8000-30560f906fe2"   None
bench-vol-1   6e329c5f-84fe-40da-9632-33cd3213e045   Available   FullyProtected     1         200 GiB   2          false         values:"nqn.2014-08.org.nvmexpress:uuid:6f63c000-1464-11f2-8000-30560f906fe2"   None
bench-vol-4   a1a23dd8-f9fb-4746-b502-b79fc8d31226   Available   FullyProtected     4         200 GiB   2          false         values:"nqn.2014-08.org.nvmexpress:uuid:6f63c000-1464-11f2-8000-30560f906fe2"   None
bench-vol-2   e05c6c1c-1062-4e26-963d-710cb1fe0ad6   Available   FullyProtected     2         200 GiB   2          false         values:"nqn.2014-08.org.nvmexpress:uuid:6f63c000-1464-11f2-8000-30560f906fe2"   None
bench-vol-6   f67d9374-7487-48e0-ac07-bd2c98a9781e   Available   FullyProtected     6         200 GiB   2          false         values:"nqn.2014-08.org.nvmexpress:uuid:6f63c000-1464-11f2-8000-30560f906fe2"   None

ACL = hostNQN nghĩa là chỉ client này được mount vào 6 vol × 200GiB vừa tạo ở trên, cluster tự rải primary đều 3 node.

[root@vm-admin-01 ~]# lbcli list volumes --project-name default -o json | grep -iE "name|node|primary"
      "nodeList": [
      "name": "bench-vol-5",
      "projectName": "default",
      "sourceSnapshotName": "",
      "qosPolicyName": "no-limit-policy",
      "primaryNodeUUID": "ef732732-4c12-5e22-8bf4-eb3dddbd0cc7",
      "missingNodesAccess": []
      "nodeList": [
      "name": "bench-vol-3",
      "projectName": "default",
      "sourceSnapshotName": "",
      "qosPolicyName": "no-limit-policy",
      "primaryNodeUUID": "5c5b6d5d-21c0-51e2-9c02-359936ae9c70",
      "missingNodesAccess": []
      "nodeList": [
      "name": "bench-vol-1",
      "projectName": "default",
      "sourceSnapshotName": "",
      "qosPolicyName": "no-limit-policy",
      "primaryNodeUUID": "ef732732-4c12-5e22-8bf4-eb3dddbd0cc7",
      "missingNodesAccess": []
      "nodeList": [
      "name": "bench-vol-4",
      "projectName": "default",
      "sourceSnapshotName": "",
      "qosPolicyName": "no-limit-policy",
      "primaryNodeUUID": "e8bc0737-a4af-5538-8952-3cdf63a20f96",
      "missingNodesAccess": []
      "nodeList": [
      "name": "bench-vol-2",
      "projectName": "default",
      "sourceSnapshotName": "",
      "qosPolicyName": "no-limit-policy",
      "primaryNodeUUID": "5c5b6d5d-21c0-51e2-9c02-359936ae9c70",
      "missingNodesAccess": []
      "nodeList": [
      "name": "bench-vol-6",
      "projectName": "default",
      "sourceSnapshotName": "",
      "qosPolicyName": "no-limit-policy",
      "primaryNodeUUID": "e8bc0737-a4af-5538-8952-3cdf63a20f96",
      "missingNodesAccess": []

Phân bố primary gần như cân bằng hoàn hảo — đúng cái cần để ép đều 3 node. Đối chiếu UUID node từ lbcli list nodes dưới:

[root@vm-admin-01 ~]# lbcli list nodes
Name            UUID                                   State     NVMe endpoint        Failure domains   Capacity   Used      Local rebuild progress
oss-nvme-01-0   e8bc0737-a4af-5538-8952-3cdf63a20f96   Active    10.237.95.101:4420   [oss-nvme-01]     61 TiB     0 B       None
oss-nvme-02-0   ef732732-4c12-5e22-8bf4-eb3dddbd0cc7   Active    10.237.95.102:4420   [oss-nvme-02]     61 TiB     0 B       None
oss-nvme-03-0   5c5b6d5d-21c0-51e2-9c02-359936ae9c70   Active    10.237.95.103:4420   [oss-nvme-03]     61 TiB     0 B       None

Node UUID	Node	Volume primary
`e8bc0737...`	oss-nvme-01	bench-vol-4, bench-vol-6
`ef732732...`	oss-nvme-02	bench-vol-1, bench-vol-5
`5c5b6d5d...`	oss-nvme-03	bench-vol-2, bench-vol-3

2 volume bench primary mỗi node — chia đều tăm tắp: - oss-nvme-01: vol-4, vol-6 - oss-nvme-02: vol-1, vol-5 - oss-nvme-03: vol-2, vol-3

Nghĩa là khi chạy fio song song lên 6 volume, IO sẽ đổ đều primary lên cả 3 node — đúng mục tiêu ép trần cluster.
Không cần tạo thêm hay đổi gì, LightOS tự cân bằng tốt.

================================================================

PHẦN C — KẾT NỐI CLIENT (discovery-client)

================================================================

Theo docs thì discovery-client tốt hơn nvme connect bằng tay vì nó tự cập nhật path khi add/remove node — đúng nhu cầu add oss-nvme-06 làm node sau này.

C.1 Cài discovery-client (trên oss-nvme-06)

Như đã nói ở trên thì discovery-client có repo riêng, khác với repo lbcli.

# prereq
dnf install -y yum-utils

# import GPG key của discovery-client
rpm --import 'https://dl.lightbitslabs.com/public/discovery-client/gpg.014E5C7FAFD89AEE.key'

# repo file — el/9 cho AlmaLinux 9
curl -1sLf 'https://dl.lightbitslabs.com/public/discovery-client/config.rpm.txt?distro=el&codename=9' | tee /etc/yum.repos.d/lightbits-discovery-client.repo
dnf makecache -y --disablerepo='*' --enablerepo='lightbits-discovery-client'

# cài discovery-client
dnf install -y discovery-client

Nếu codename=9 không có gói, thử codename=8 (gói thường tương thích el8/el9).

C.2 Initial setup

# xác nhận nvme-tcp loaded + multipath=Y (đã làm ở A, có thể kiểm lại)
[root@oss-nvme-06 ~]# lsmod | grep nvme
nvme                   77824  0
nvme_tcp              102400  0
nvme_fabrics           49152  1 nvme_tcp
nvme_core             274432  3 nvme_tcp,nvme,nvme_fabrics
nvme_keyring           20480  3 nvme_tcp,nvme_core,nvme_fabrics
nvme_auth              36864  1 nvme_core

[root@oss-nvme-06 ~]# cat /sys/module/nvme_core/parameters/multipath
Y # PHẢI = Y

# khởi động service
[root@oss-nvme-06 ~]# systemctl enable --now discovery-client
Created symlink /etc/systemd/system/multi-user.target.wants/discovery-client.service → /etc/systemd/system/discovery-client.service.

C.2b Kết nối qua config file

Cần subsysNQN (B.2) + hostNQN (B.1). Tạo file config liệt kê cả 3 data IP.
Port discovery = 8009 (khác với port data 4420) với -n là SUBNQN và -q là HOSTNQN.

mkdir -p /etc/discovery-client/discovery.d/
SUBNQN="nqn.2016-01.com.lightbitslabs:uuid:9304cdd3-0716-4e2f-b37f-bf2f590d6c4d"
HOSTNQN=$(cat /etc/nvme/hostnqn)
cat > /etc/discovery-client/discovery.d/oss-cluster.conf << EOF
-t tcp -a 10.237.95.101 -s 8009 -q $HOSTNQN -n $SUBNQN
-t tcp -a 10.237.95.102 -s 8009 -q $HOSTNQN -n $SUBNQN
-t tcp -a 10.237.95.103 -s 8009 -q $HOSTNQN -n $SUBNQN
EOF
systemctl restart discovery-client

discovery-client tự nvme discover + nvme connect và tự cập nhật path khi add/remove node.

C.2c Connect-all bằng CLI

Nếu không muốn dùng config file thì dùng command này:

discovery-client connect-all -t tcp -a 10.237.95.101 -s 8009 -q $(cat /etc/nvme/hostnqn) -p

-p: giữ kết nối persistent để monitor thay đổi.

C.3 Nvme connect tay nếu discovery-client vướng

Cần subsysNQN và connect tới cả 3 node để có đa đường:

SUBNQN="nqn.2016-01.com.lightbitslabs:uuid:..."
HOSTNQN=$(cat /etc/nvme/hostnqn)
for ip in 10.237.95.101 10.237.95.102 10.237.95.103; do
  nvme connect -t tcp -a $ip -s 4420 -n "$SUBNQN" -q "$HOSTNQN" --ctrl-loss-tmo -1
done

--ctrl-loss-tmo -1: cho phép reconnect vô hạn, tránh timeout khi node fail (cần cho test failover).

C.4 Verify kết nối

# thấy các /dev/nvmeXnY = volume bench
[root@oss-nvme-06 ~]# systemctl enable --now discovery-client
Created symlink /etc/systemd/system/multi-user.target.wants/discovery-client.service → /etc/systemd/system/discovery-client.service.
[root@oss-nvme-06 ~]# nvme list
Node                  Generic               SN                   Model                                    Namespace  Usage                      Format           FW Rev
--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------
/dev/nvme0n1          /dev/ng0n1            S6VANN0L107093       SAMSUNG MZQL215THBLA-00A07               0x1          0.00   B /  15.36  TB      4 KiB +  0 B   GDD5302Q
/dev/nvme1n1          /dev/ng1n1            S6VANN0L105917       SAMSUNG MZQL215THBLA-00A07               0x1          0.00   B /  15.36  TB      4 KiB +  0 B   GDD5302Q
/dev/nvme2n1          /dev/ng2n1            S6VANN0L105522       SAMSUNG MZQL215THBLA-00A07               0x1          0.00   B /  15.36  TB      4 KiB +  0 B   GDD5302Q
/dev/nvme3n1          /dev/ng3n1            S6VANN0L107106       SAMSUNG MZQL215THBLA-00A07               0x1          0.00   B /  15.36  TB      4 KiB +  0 B   GDD5302Q
/dev/nvme4n1          /dev/ng4n1            S6VANN0L105513       SAMSUNG MZQL215THBLA-00A07               0x1          0.00   B /  15.36  TB      4 KiB +  0 B   GDD5302Q
/dev/nvme5n1          /dev/ng5n1            S6VANN0L105524       SAMSUNG MZQL215THBLA-00A07               0x1          0.00   B /  15.36  TB      4 KiB +  0 B   GDD5302Q
/dev/nvme6n1          /dev/ng6n1            S6VANN0L107103       SAMSUNG MZQL215THBLA-00A07               0x1          0.00   B /  15.36  TB      4 KiB +  0 B   GDD5302Q
/dev/nvme7n1          /dev/ng7n1            S6VANN0L107041       SAMSUNG MZQL215THBLA-00A07               0x1          0.00   B /  15.36  TB      4 KiB +  0 B   GDD5302Q
/dev/nvme8n1          /dev/ng8n1            f973dc215df7aebb     Lightbits LightOS                        0x2        214.75  GB / 214.75  GB      4 KiB +  0 B   3.18
/dev/nvme8n2          /dev/ng8n2            f973dc215df7aebb     Lightbits LightOS                        0x3        214.75  GB / 214.75  GB      4 KiB +  0 B   3.18
/dev/nvme8n3          /dev/ng8n3            f973dc215df7aebb     Lightbits LightOS                        0x5        214.75  GB / 214.75  GB      4 KiB +  0 B   3.18
/dev/nvme8n4          /dev/ng8n4            f973dc215df7aebb     Lightbits LightOS                        0x6        214.75  GB / 214.75  GB      4 KiB +  0 B   3.18
/dev/nvme8n5          /dev/ng8n5            f973dc215df7aebb     Lightbits LightOS                        0x1        214.75  GB / 214.75  GB      4 KiB +  0 B   3.18
/dev/nvme8n6          /dev/ng8n6            f973dc215df7aebb     Lightbits LightOS                        0x4        214.75  GB / 214.75  GB      4 KiB +  0 B   3.18

# thấy đa đường (3 path tới subsystem)
[root@oss-nvme-06 ~]# nvme list-subsys
nvme-subsys0 - NQN=nqn.1994-11.com.samsung:nvme:PM9A3:2.5-inch:S6VANN0L107093
               hostnqn=nqn.2014-08.org.nvmexpress:uuid:6f63c000-1464-11f2-8000-30560f906fe2
               iopolicy=numa
\
 +- nvme0 pcie 0000:81:00.0 live

nvme-subsys1 - NQN=nqn.1994-11.com.samsung:nvme:PM9A3:2.5-inch:S6VANN0L105917
               hostnqn=nqn.2014-08.org.nvmexpress:uuid:6f63c000-1464-11f2-8000-30560f906fe2
               iopolicy=numa
\
 +- nvme1 pcie 0000:c3:00.0 live

nvme-subsys8 - NQN=nqn.2016-01.com.lightbitslabs:uuid:a1bf3a50-1b6e-47aa-b5c2-3e8ae454f916
               hostnqn=nqn.2014-08.org.nvmexpress:uuid:6f63c000-1464-11f2-8000-30560f906fe2
               iopolicy=numa
\
 +- nvme10 tcp traddr=10.237.95.102,trsvcid=4420,src_addr=10.237.95.106 live
 +- nvme8 tcp traddr=10.237.95.103,trsvcid=4420,src_addr=10.237.95.106 live
 +- nvme9 tcp traddr=10.237.95.101,trsvcid=4420,src_addr=10.237.95.106 live

nvme-subsys2 - NQN=nqn.1994-11.com.samsung:nvme:PM9A3:2.5-inch:S6VANN0L105522
               hostnqn=nqn.2014-08.org.nvmexpress:uuid:6f63c000-1464-11f2-8000-30560f906fe2
               iopolicy=numa
\
 +- nvme2 pcie 0000:c4:00.0 live

nvme-subsys3 - NQN=nqn.1994-11.com.samsung:nvme:PM9A3:2.5-inch:S6VANN0L107106
               hostnqn=nqn.2014-08.org.nvmexpress:uuid:6f63c000-1464-11f2-8000-30560f906fe2
               iopolicy=numa
\
 +- nvme3 pcie 0000:82:00.0 live

nvme-subsys4 - NQN=nqn.1994-11.com.samsung:nvme:PM9A3:2.5-inch:S6VANN0L105513
               hostnqn=nqn.2014-08.org.nvmexpress:uuid:6f63c000-1464-11f2-8000-30560f906fe2
               iopolicy=numa
\
 +- nvme4 pcie 0000:83:00.0 live

nvme-subsys5 - NQN=nqn.1994-11.com.samsung:nvme:PM9A3:2.5-inch:S6VANN0L105524
               hostnqn=nqn.2014-08.org.nvmexpress:uuid:6f63c000-1464-11f2-8000-30560f906fe2
               iopolicy=numa
\
 +- nvme5 pcie 0000:84:00.0 live

nvme-subsys6 - NQN=nqn.1994-11.com.samsung:nvme:PM9A3:2.5-inch:S6VANN0L107103
               hostnqn=nqn.2014-08.org.nvmexpress:uuid:6f63c000-1464-11f2-8000-30560f906fe2
               iopolicy=numa
\
 +- nvme6 pcie 0000:05:00.0 live

nvme-subsys7 - NQN=nqn.1994-11.com.samsung:nvme:PM9A3:2.5-inch:S6VANN0L107041
               hostnqn=nqn.2014-08.org.nvmexpress:uuid:6f63c000-1464-11f2-8000-30560f906fe2
               iopolicy=numa
\
 +- nvme7 pcie 0000:06:00.0 live

# Các block device mới
[root@oss-nvme-06 ~]# lsblk | grep nvme
nvme7n1   259:0    0    14T  0 disk
nvme2n1   259:1    0    14T  0 disk
nvme1n1   259:2    0    14T  0 disk
nvme6n1   259:3    0    14T  0 disk
nvme3n1   259:4    0    14T  0 disk
nvme0n1   259:5    0    14T  0 disk
nvme4n1   259:6    0    14T  0 disk
nvme5n1   259:7    0    14T  0 disk
nvme8n1   259:9    0   200G  0 disk
nvme8n2   259:11   0   200G  0 disk
nvme8n6   259:17   0   200G  0 disk
nvme8n4   259:19   0   200G  0 disk
nvme8n5   259:21   0   200G  0 disk
nvme8n3   259:25   0   200G  0 disk

Mỗi volume nên hiện 1 device với nhiều path (live + inaccessible). Đó là multipath.

================================================================

PHẦN D — DỌN DẸP

================================================================

Sau khi bench xong, để dọn dẹp sạch oss-nvme-06, làm như sau:

# trên client dừng discovery-client trước (nếu không nó tự connect lại)
systemctl disable --now discovery-client
rm -f /etc/discovery-client/discovery.d/oss-cluster.conf

# ngắt mọi kết nối nvme
nvme disconnect-all

# trên controller xóa volume bench
for i in $(seq 1 6); do lbcli delete volume --name bench-vol-$i --project-name default; done

Thứ tự quan trọng: dừng discovery-client trước khi disconnect, nếu không nó sẽ tự kết nối lại volume (vì -p persistent monitor).

Personal Ops Runbook

Infra Notes

Lightbits Client

Thông tin

PHẦN A — CHUẨN BỊ CLIENT (trên oss-nvme-06)

A.0 Cài package, disable toàn bộ service cũ

[CLIENT] — Lấy bundle từ node ADMIN

[NEW] — Dựng repo local và cài đặt package

A.1 Kiểm tra kernel (cần >=5.3.5 cho nvme-tcp)

A.2 Kiểm tra data network tới 3 node

A.3 Cài gói cần thiết

A.4 Nạp module nvme-tcp

A.5 Bật NVMe native multipath

A.6 Reboot để áp dụng multipath

PHẦN B — LẤY HOSTNQN + TẠO VOLUME (ACL)

B.1 Lấy hostNQN của client (trên oss-nvme-06)

B.2 Lấy subsystem NQN của cluster (trên CONTROLLER vm-admin-01)

B.3 Tạo nhiều volume để bench MAX (trên CONTROLLER)

PHẦN C — KẾT NỐI CLIENT (discovery-client)

C.1 Cài discovery-client (trên oss-nvme-06)

C.2 Initial setup

C.2b Kết nối qua config file

C.2c Connect-all bằng CLI

C.3 Nvme connect tay nếu discovery-client vướng

C.4 Verify kết nối

PHẦN D — DỌN DẸP