Logo

Personal Ops Runbook

Personal runbook covering infrastructure operations for Cloud, Kubernetes, OpenStack, and Ceph environments. Includes deployment and teardown procedures, node management, cluster monitoring setup, and incident response workflows compiled from day-to-day operational work. Intended strictly for personal reference — configurations and scripts are environment-specific and not guaranteed to work as-is elsewhere.

BUILD LIGHTOS OFFLINE KIT 3.18.3 — node AlmaLinux 9.0 (cần internet, chạy 1 lần)

Bản riêng cho môi trường AlmaLinux 9.0. Bản 9.8 xem [Phần 5] Offline KIT.

Khác bản 9.8: - Mọi RPM resolve theo AlmaLinux 9.0 vault (--disablerepo=*), không lấy từ repo đang chạy. Repo đang chạy trả về el9_8 → kit chứa glibc/openssl-libs 3.5 el9_8 nhưng không có openssh → cài lên node 9.0 sẽ nâng openssl nửa vời, sshd chết (đã gặp thật trên máy build). - klight_utils 3.18.3 cần ipmctl → chỉ có ở EPEL 9. - Repo 3.18.3 không có gftl; repo 9.0 không có drpm. - management-lbcli 3.18.3 lấy từ repo private (cùng version cluster), không dùng repo public. - Monitoring image lấy từ RPM lightos-monitoring-images (đã chứa sẵn prometheus v2.42.0 + grafana 9.3.6) → không cần docker pull, không cài docker lên máy build. - Collections lấy từ bản bundle trong ansible==9.13.0 (tương thích core 2.16).

Môi trường build đã chạy: 172.16.102.2 (AlmaLinux 9.0, đi internet thẳng, không proxy). Máy này cũng là controller.

Chuẩn bị máy build

⚠️ Máy 9.0 + repo hiện tại (el9_8): cài python3.12 sẽ kéo openssl-libs 3.5 và cần expat mới. Phải nâng openssh* + expat cùng lúc, nếu không sshd không nhận kết nối mới và pip3.12 lỗi pyexpat.

dnf install -y tar createrepo_c python3.12 python3.12-pip dnf-plugins-core
dnf upgrade -y 'openssh*' expat && systemctl restart sshd

Kiểm tra trước khi đóng session SSH hiện tại (mở session mới thử):

rpm -q openssl-libs openssh-server expat python3.12
ssh -o ConnectTimeout=5 localhost true && echo "sshd OK"

Tạo các thư mục + repo

KIT=~/lightos-offline-kit
mkdir -p $KIT/{01-repo,02-ansible,03-controller,04-docs}
mkdir -p $KIT/03-controller/{ansible,lbcli,docker,monitoring,node-exporter}

# Repo Lightbits 3.18.3 (thay <USERKEY> that, bo ca dau <>)
cat > /etc/yum.repos.d/lightos.repo << 'EOF'
[lightos]
name=Lightbits LightOS
baseurl=https://dl.lightbitslabs.com/<USERKEY>/lightos-3-18-3-rhl-9/rpm/el/9/$basearch
enabled=1
gpgcheck=0
EOF

# EPEL chi dung luc tai (enabled=0, khong doi hanh vi dnf cua may)
cat > /etc/yum.repos.d/epel-kit.repo << 'EOF'
[epel-kit]
name=EPEL 9 (kit build only)
baseurl=https://dl.fedoraproject.org/pub/epel/9/Everything/$basearch/
enabled=0
gpgcheck=1
gpgkey=https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-9
EOF

# Docker repo (cho 03-controller/docker)
dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
dnf makecache

# Bo option resolve theo AlmaLinux 9.0 vault (dung cho moi lenh download ben duoi)
V=https://vault.almalinux.org/9.0
R90="--disablerepo=* --repofrompath=v90base,$V/BaseOS/x86_64/os/ --repofrompath=v90app,$V/AppStream/x86_64/os/ --repofrompath=v90crb,$V/CRB/x86_64/os/ --setopt=v90base.gpgcheck=0 --setopt=v90app.gpgcheck=0 --setopt=v90crb.gpgcheck=0"

====== 01-repo : RPM Lightbits + deps ======

mkdir -p $KIT/01-repo/target_rpms $KIT/01-repo/deps

# Goi Lightbits + toan bo deps (khong co gftl o 3.18.3; ipmctl tu EPEL)
dnf $R90 --enablerepo=lightos,epel-kit download --resolve --alldeps \
  --destdir=$KIT/01-repo/target_rpms \
  lightos duroslight management-node-manager management-cluster-manager \
  management-api-service management-profile-generator discovery-service \
  discovery-client management-lightbox-exporter management-upgrade-manager \
  etcd userlbe klight_utils nvme-cli

# Deps OS (role offline cai createrepo_c*.rpm truc tiep -> bat buoc ban 9.0; 9.0 khong co drpm)
dnf $R90 download --resolve --alldeps --destdir=$KIT/01-repo/deps \
  chrony mdadm createrepo_c

createrepo_c $KIT/01-repo/target_rpms
createrepo_c $KIT/01-repo/deps
echo "target=$(ls $KIT/01-repo/target_rpms/*.rpm|wc -l) deps=$(ls $KIT/01-repo/deps/*.rpm|wc -l)"

→ Lần build 2026-10-11: target=130, deps=123.

====== 02-ansible : bo deploy GOC Lightbits ======

# Tai tgz goc (thay <USERKEY>)
cd ~
curl -1 'https://dl.lightbitslabs.com/<USERKEY>/lightos-3-18-3-rhl-9/raw/files/light-app-install-environment-v3.18.3b32060372798.tgz?accept_eula=1' \
  -o light-app-install-environment-v3.18.3b32060372798.tgz
tar -xzf ~/light-app-install-environment-v3.18.3*.tgz -C $KIT/02-ansible
ls $KIT/02-ansible/   # phai co: ansible ansible.cfg playbooks plugins roles

====== 03-controller/ansible : ansible offline ======

pip3.12 download \
  'ansible==9.13.0' 'ansible-core==2.16.19' 'jinja2==3.1.6' 'PyYAML==6.0.3' \
  'cryptography==49.0.0' 'netaddr==1.3.0' 'packaging==26.2' 'resolvelib==1.0.1' \
  'PyJWT==2.15.1' \
  -d $KIT/03-controller/ansible/pip-wheels
# PyJWT BAT BUOC: plugins/modules/jwt.py (import jwt) tao JWT system:root tren controller.
# Thieu -> deploy fail o buoc "Create a JWT token". Module chi goi jwt.encode() -> tuong thich PyJWT 2.x.

# Collections playbook can: community.crypto, community.general, ansible.posix, grafana.grafana
# Lay tu ban bundle trong ansible 9.13.0 (tuong thich core 2.16)
python3.12 -m venv /root/kit-venv
/root/kit-venv/bin/pip install --no-index --find-links $KIT/03-controller/ansible/pip-wheels ansible
SP=$(/root/kit-venv/bin/python -c "import ansible_collections,os;print(os.path.dirname(ansible_collections.__path__[0]))")
DST=~/.ansible/collections/ansible_collections
mkdir -p $DST/community $DST/ansible $DST/grafana
cp -a $SP/ansible_collections/community/crypto $SP/ansible_collections/community/general $DST/community/
cp -a $SP/ansible_collections/ansible/posix $DST/ansible/
cp -a $SP/ansible_collections/grafana/grafana $DST/grafana/
tar -czf $KIT/03-controller/ansible/ansible-collections.tar.gz -C ~/.ansible collections

====== 03-controller/lbcli : lbcli offline ======

# lbcli 3.18.3 co trong repo private -> cung version cluster
dnf $R90 --enablerepo=lightos download --resolve --alldeps \
  --destdir=$KIT/03-controller/lbcli management-lbcli

====== 03-controller/docker : docker-ce offline ======

dnf $R90 --enablerepo=docker-ce-stable download --resolve --alldeps \
  --destdir=$KIT/03-controller/docker \
  docker-ce docker-ce-cli containerd.io docker-ce-rootless-extras

====== 03-controller/monitoring : RPM + image tar ======

dnf $R90 --enablerepo=lightos download --resolve --alldeps \
  --destdir=$KIT/03-controller/monitoring \
  lightos-monitoring-images lightos-monitoring-clustering

# Image da nam san trong RPM (dinh dang docker save) -> gop thanh 1 file tar cho [Phan 12.1]
M=/root/img-merge; rm -rf $M /root/mon-x; mkdir -p $M/out /root/mon-x
cd /root/mon-x && rpm2cpio $KIT/03-controller/monitoring/lightos-monitoring-images-*.rpm | cpio -idm --quiet
cd /root/mon-x/var/lib/monitoring-images
for f in prometheus-v2.42.0 grafana-9.3.6; do mkdir -p $M/$f && tar -xzf $f.tar.gz -C $M/$f; done
python3 -I - "$M" << 'PY'
import json, os, shutil, sys
M = sys.argv[1]; out = os.path.join(M, "out")
manifest, repos, index = [], {}, None
for src in ("prometheus-v2.42.0", "grafana-9.3.6"):
    d = os.path.join(M, src)
    for root, _, files in os.walk(os.path.join(d, "blobs")):
        for fn in files:
            rel = os.path.relpath(os.path.join(root, fn), d)
            dst = os.path.join(out, rel); os.makedirs(os.path.dirname(dst), exist_ok=True)
            if not os.path.exists(dst): shutil.copy2(os.path.join(root, fn), dst)
    manifest += json.load(open(os.path.join(d, "manifest.json")))
    for k, v in json.load(open(os.path.join(d, "repositories"))).items(): repos.setdefault(k, {}).update(v)
    idx = json.load(open(os.path.join(d, "index.json")))
    if index is None: index = idx
    else: index["manifests"] += idx["manifests"]
    shutil.copy2(os.path.join(d, "oci-layout"), os.path.join(out, "oci-layout"))
json.dump(manifest, open(os.path.join(out, "manifest.json"), "w"))
json.dump(repos, open(os.path.join(out, "repositories"), "w"))
json.dump(index, open(os.path.join(out, "index.json"), "w"))
for m in manifest:
    for p in [m["Config"]] + m["Layers"]:
        assert os.path.exists(os.path.join(out, p)), p
print("tags:", [t for m in manifest for t in m["RepoTags"]])
PY
tar -cf $KIT/03-controller/monitoring/monitoring-docker-images.tar -C $M/out .
cd ~ && rm -rf $M /root/mon-x

→ In ra tags: ['prom/prometheus:v2.42.0', 'grafana/grafana:9.3.6']. File tar khoảng 532M.

====== 03-controller/node-exporter : binary ======

VER=1.8.2
mkdir -p /root/ne-dl && cd /root/ne-dl
curl -LO https://github.com/prometheus/node_exporter/releases/download/v${VER}/node_exporter-${VER}.linux-amd64.tar.gz
curl -sL https://github.com/prometheus/node_exporter/releases/download/v${VER}/sha256sums.txt | grep linux-amd64.tar.gz | sha256sum -c -
tar xzf node_exporter-${VER}.linux-amd64.tar.gz
cp node_exporter-${VER}.linux-amd64/node_exporter $KIT/03-controller/node-exporter/
cat > $KIT/03-controller/node-exporter/node_exporter.service << 'EOF'
[Unit]
Description=Prometheus Node Exporter
After=network.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter --web.listen-address=:9100
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
cd ~ && rm -rf /root/ne-dl

====== 04-docs : README ======

cat > $KIT/04-docs/README.md << 'EOF'
# LightOS Offline Kit - AlmaLinux 9.0 x86_64 + Lightbits 3.18.3 (3.18.3b32060372798)

## Structure
- 01-repo/       : Lightbits RPM + deps (deploy cluster node). All OS deps resolved against AlmaLinux 9.0 vault (el9_0).
                   Includes ipmctl/libipmctl from EPEL 9 (required by klight_utils).
- 02-ansible/    : original Lightbits deploy kit v3.18.3 (playbooks/roles), ansible-core >= 2.16
- 03-controller/ : lbcli 3.18.3 + docker-ce + monitoring (RPM + merged image tar) + node_exporter 1.8.2 + ansible wheels/collections
                   (RPMs resolved against AlmaLinux 9.0 vault)
- 04-docs/       : this file

## Versions
- ansible 9.13.0 / ansible-core 2.16.19 (python3.12 wheels, cp312)
- collections: community.crypto 2.22.3, community.general 8.6.8, ansible.posix 1.6.2, grafana.grafana 2.2.5
- monitoring images: prom/prometheus:v2.42.0, grafana/grafana:9.3.6 (taken from lightos-monitoring-images RPM)

## Assumptions
- Storage nodes: AlmaLinux 9.0 (kit contains only el9_0 OS packages -> no partial upgrade of glibc/openssl).
- Controller: needs python3.12 (NOT in AlmaLinux 9.0 repos).

## Deploy = fully offline (see 1-Deploy.md)
## Destroy = see 0-Destroy.md
EOF

====== Dong goi + checksum ======

cd ~
tar -czf lightos-offline-kit.tar.gz -C ~ lightos-offline-kit
sha256sum lightos-offline-kit.tar.gz > lightos-offline-kit.tar.gz.sha256
ls -lh lightos-offline-kit.tar.gz*

→ Lần build 2026-10-11 (đã có PyJWT): 1010M, sha256 02692f11f76a54ec4f2594816c9b13d6e7b170129b35c12691f6a18839e91633.

====== Verify kit truoc khi dem di ======

sha256sum -c ~/lightos-offline-kit.tar.gz.sha256
mkdir -p /tmp/vk && tar -xzf ~/lightos-offline-kit.tar.gz -C /tmp/vk
K=/tmp/vk/lightos-offline-kit
echo "repo: $(ls $K/01-repo/target_rpms/*.rpm|wc -l)+$(ls $K/01-repo/deps/*.rpm|wc -l) repodata: $(ls $K/01-repo/*/repodata/repomd.xml|wc -l)"
echo "ansible: $(ls $K/02-ansible/playbooks/*.yml|wc -l) playbooks, $(ls $K/02-ansible/roles/|wc -l) roles"
echo "lbcli: $(ls $K/03-controller/lbcli/*.rpm|wc -l) docker: $(ls $K/03-controller/docker/*.rpm|wc -l) monitoring: $(ls $K/03-controller/monitoring/*.rpm|wc -l)"
# Bat buoc = 0: khong duoc co goi OS ban 9.1..9.8
echo "el9_1..8 rpm: $(find $K -name '*.rpm' | grep -cE 'el9_[1-9]')"
# Image tar: digest tung blob phai khop ten file
mkdir -p /tmp/vk/img && tar -xf $K/03-controller/monitoring/monitoring-docker-images.tar -C /tmp/vk/img
cd /tmp/vk/img/blobs/sha256 && bad=0; for b in *; do [ "$(sha256sum $b | cut -c1-64)" = "$b" ] || bad=$((bad+1)); done; echo "image blobs: $(ls | wc -l), mismatch: $bad"; cd ~
# Ansible cai offline trong venv sach (khong nuot loi)
python3.12 -m venv /tmp/tv && /tmp/tv/bin/pip install -q --no-index \
  --find-links $K/03-controller/ansible/pip-wheels ansible PyJWT \
  && /tmp/tv/bin/ansible --version | head -1 \
  && /tmp/tv/bin/python -c "import jwt; print('PyJWT', jwt.__version__)"
rm -rf /tmp/vk /tmp/tv

→ Lần build 2026-10-11: repo: 130+123 repodata: 2, 11 playbooks, 9 roles, lbcli: 21 docker: 164 monitoring: 116, el9_1..8 rpm: 0, mismatch: 0, ansible [core 2.16.19], PyJWT 2.15.1, 12 wheels.

Chưa test: docker load image tar thật và cài offline lên node 9.0 thật — kiểm ở bước deploy / [Phần 12.1].

Điều kiện ngoài kit khi deploy offline (đã rà playbook 3.18.3)

  • Node: tắt hết repo online (/etc/yum.repos.d/*.repo → .disabled). Task chrony (package:) và install_lbcli (yum chỉ enablerepo, không disablerepo) vẫn đọc mọi repo đang bật → repo không reach được thì dnf fail. Playbook tự thêm repo lightos = file://{{ dest_dir }}.
  • all.yml: source_type: offline, source_rpms_dir: <kit>/01-repo/target_rpms, source_dependencies_rpms_dir: <kit>/01-repo/deps, dest_dir: /tmp/rpms — đường dẫn trên controller (role copy từ controller sang node).
  • Controller: cài ansible + PyJWT từ wheels vào venv python3.12, chạy ansible-playbook từ venv đó (JWT module chạy delegate_to: localhost bằng python của venv). Cần openssl CLI (tls-setup gọi openssl req).
  • etcd cài từ RPM (install_from_rpm.yml, nhánh RedHat) → không cần source_etcd_binary. Nhánh GitHub binary chỉ chạy khi run_in_container.
  • deploy-lightos.yml 3.18.3 không hardcode proxy (khác bản 3.18.2 đã sửa tay trước đây).
  • Collection ansible.netcommon (filter ipwrap trong role etcd) có sẵn trong gói ansible 9.13.0 cài từ wheel.