Logo

Personal Ops Runbook

Personal runbook covering infrastructure operations for Cloud, Kubernetes, OpenStack, and Ceph environments. Includes deployment and teardown procedures, node management, cluster monitoring setup, and incident response workflows compiled from day-to-day operational work. Intended strictly for personal reference — configurations and scripts are environment-specific and not guaranteed to work as-is elsewhere.

BƯỚC 0 — Tắt repo online trên node

# Ensure node repos are OFF (force offline install)
for ip in 10.237.94.10{1..3}; do
ssh root@$ip 'cd /etc/yum.repos.d && for f in *.repo; do [ -e "$f" ] && mv "$f" "$f.disabled"; done; echo -n "$ip active repos: "; ls *.repo 2>/dev/null|wc -l'
done

BƯỚC 1 — Verify + bung kit

Verify kit integrity

cd ~
sha256sum -c lightos-offline-kit.tar.gz.sha256

Kỳ vọng lightos-offline-kit.tar.gz: OK

Extract kit

tar -xzf lightos-offline-kit.tar.gz
ls lightos-offline-kit/

BƯỚC 2 — Dựng bundle RPM và Ansible từ kit

RPM bundle -> ~/lightos-offline (source_rpms_dir points here)

KIT=~/lightos-offline-kit
mkdir -p ~/lightos-offline
cp -r $KIT/01-repo/target_rpms ~/lightos-offline/
cp -r $KIT/01-repo/deps ~/lightos-offline/
echo "bundle: target=$(ls ~/lightos-offline/target_rpms/*.rpm|wc -l) deps=$(ls ~/lightos-offline/deps/*.rpm|wc -l)"

Kỳ vọng bundle: target=130 deps=359

Original ansible deploy kit -> ~/light-app

rm -rf ~/light-app && mkdir -p ~/light-app
cp -r $KIT/02-ansible/* ~/light-app/
echo "light-app: $(ls ~/light-app/playbooks/*.yml|wc -l) playbooks, $(ls ~/light-app/roles/|wc -l) roles"

Kỳ vọng light-app: 11 playbooks, 9 roles

BƯỚC 3 — Tạo inventory clusterNew (01/02/03, 6+2 EC 4+2)

mkdir -p ~/light-app/ansible/inventories/clusterNew/{group_vars,host_vars}

cat > ~/light-app/ansible/inventories/clusterNew/hosts << 'EOF'
oss-nvme-01 ansible_host=10.237.94.101  ansible_connection=ssh  ansible_ssh_user=root   ansible_become_user=root
oss-nvme-02 ansible_host=10.237.94.102  ansible_connection=ssh  ansible_ssh_user=root   ansible_become_user=root
oss-nvme-03 ansible_host=10.237.94.103  ansible_connection=ssh  ansible_ssh_user=root   ansible_become_user=root

[duros_nodes]
oss-nvme-01
oss-nvme-02
oss-nvme-03

[etcd]
oss-nvme-01
oss-nvme-02
oss-nvme-03
EOF

cat > ~/light-app/ansible/inventories/clusterNew/group_vars/all.yml << 'EOF'
---
ipacl_explicit_allow: false
persistent_memory: false
start_discovery_service_retries: 5
chrony_enabled: true
certificates_directory: "{{ '~/lightos-certificates-clusterNew' | expanduser }}"
source_type: offline
source_rpms_dir: "/root/lightos-offline/target_rpms"
source_dependencies_rpms_dir: "/root/lightos-offline/deps"
dest_dir: "/tmp/rpms"
EOF

for n in 01 02 03; do
cat > ~/light-app/ansible/inventories/clusterNew/host_vars/oss-nvme-$n.yml << EOF
---
name: oss-nvme-$n
nodes:
- instanceID: 0
  data_ip: 10.237.95.1$n
  ec_enabled: true
  failure_domains:
  - oss-nvme-$n
  storageDeviceLayout:
    allowCrossNumaDevices: false
    deviceMatchers:
    - partition == false
    initialDeviceCount: 6
    maxDeviceCount: 6
  journalDeviceLayout:
    deviceMatchers:
    - partition == false
    numDevices: 2
EOF
done

BƯỚC 6 — Pre-check + xác nhận bundle còn

ls ~/lightos-offline/target_rpms/*.rpm 2>/dev/null | wc -l
cd ~/light-app
INV=ansible/inventories/clusterNew/hosts
echo "=== ping ==="; ansible -i $INV all -m ping | grep -E 'SUCCESS|UNREACHABLE'
echo "=== disks 8 + md126 NO ==="; ansible -i $INV all -m shell -a "echo -n disks:; lsblk|grep -cE 'nvme[0-7]n1'; echo -n ' md126:'; (grep -q md126 /proc/mdstat && echo YES||echo NO)"
echo "=== repo off ==="; ansible -i $INV all -m shell -a "ls /etc/yum.repos.d/*.repo 2>/dev/null|wc -l"
echo "=== no lightos pkg ==="; ansible -i $INV all -m shell -a "rpm -qa|grep -icE 'lightos|duros|node-manager'||echo 0"

Kỳ vọng 3 node SUCCESS, disks 8 + md126 NO, repo 0, pkg 0.

BƯỚC 7 - Deploy

cd ~/light-app
ansible-playbook -i ansible/inventories/clusterNew/hosts \
  playbooks/deploy-lightos.yml 2>&1 | tee ~/deploy-clusterNew.log

Giờ chạy verify với system jwt.

source ~/lightos-system-jwt
lbcli --endpoint https://10.237.94.101:443 -J "$LIGHTOS_JWT" list nodes

Xác nhận layout data/journal + EC.

source ~/lightos-system-jwt
EP="https://10.237.94.101:443"
echo "===== NVME DEVICES (ky vong 18 data + 6 journal) ====="
lbcli --endpoint $EP -J "$LIGHTOS_JWT" list nvme-devices 2>&1 | head -30
echo "===== CLUSTER INFO ====="
lbcli --endpoint $EP -J "$LIGHTOS_JWT" get cluster 2>&1 | head -20

Kỳ vọng mỗi node 6 data + 2 journal (tổng 18 data + 6 journal), EC enabled.