Overview

This document provides a step-by-step guide to setting up the admin.collector application. It includes instructions for installing required packages, creating and activating a virtual environment, installing necessary modules, setting up system services, and configuring RabbitMQ. The guide also covers commands for checking service status, debugging, and updating the application code. Additionally, it provides information on licensing and contact details for the author.

Install the required packages

apt update
apt install python3-pip python3-venv -y

Creating the application directory.

mkdir -p /etc/admin.collector

Create the virtual environment

python3 -m venv /etc/admin.collector/venv

Activate the virtual environment

source /etc/admin.collector/venv/bin/activate

Install the modules.

Ubuntu 22.04 Operating System

shell> lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.4 LTS
Release:        22.04
Codename:       jammy

Debian (Proxmox) Operating System

shell> lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 12 (bookworm)
Release:        12
Codename:       bookworm

Python version 3.10.x

shell> python3 --version
Python 3.10.12

Python version 3.11.x

shell> python3 --version
Python 3.11.2

Install git

apt update
apt install git -y

Clone plugins

git clone https://github.com/hoanghd164/admin.collector.git /opt/admin.collector
ln -s /opt/admin.collector /etc/admin.collector/source

Create the requirements file

cat > /etc/admin.collector/source/requirements.txt << 'OEF'
mysql-connector-python==8.0.23
protobuf==3.19.6
setuptools_rust
prometheus_client
jsonpath_ng
pyopenssl
requests
pyyaml
docker
wheel
pandas
unidecode
OEF

Install the required packages

/etc/admin.collector/venv/bin/python -m pip install -r /etc/admin.collector/source/requirements.txt

Install SNMP if you are monitoring Cisco or Fortigate devices

apt update
apt install snmp -y

Install lm-sensors if you want to monitor server temperature

apt update
apt install lm-sensors -y

Install sysstat if you want to monitor disk IOPS

apt-get update
apt-get install sysstat -y

Test script

project='config'
/etc/admin.collector/venv/bin/python /etc/admin.collector/source/plugins/run.py --project ${project}

Create a systemd service

Perform this step if you want to use the script as an endpoint for Prometheus to collect metrics. If you want to use the script running by Telegraf, skip this step.

cat > /etc/systemd/system/admin_collector.service << OEF
[Unit]
Description=Exporter Custom Metrics Service
After=network.target

[Service]
WorkingDirectory=/etc/admin.collector/source/plugins
ExecStart=/etc/admin.collector/venv/bin/python /etc/admin.collector/source/plugins/run.py --project ${project}
Restart=always
User=root
Group=root
Environment=PYTHONUNBUFFERED=1

[Install]
WantedBy=multi-user.target
OEF

Enable and start the service

systemctl daemon-reload
systemctl enable admin_collector.service
systemctl restart admin_collector.service

Check status service

Command

systemctl status admin_collector.service

Output tham khảo

● admin_collector.service - Exporter Custom Metrics Service
     Loaded: loaded (/etc/systemd/system/admin_collector.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2024-12-31 06:05:24 +07; 38s ago
   Main PID: 362451 (python)
      Tasks: 2 (limit: 38391)
     Memory: 18.2M
        CPU: 6.736s
     CGroup: /system.slice/admin_collector.service
             └─362451 /etc/admin.collector/venv/bin/python /etc/admin.collector/source/plugins/run.py --project zenhub

Dec 31 06:05:29 adminserver-251 python[362451]: Processed 7-mysql.sensor.py and saved output to /etc/admin.collector/source/plugins/data/7-mysql.sensor.json
Dec 31 06:05:30 adminserver-251 python[362451]: Processed 5-host.status.sensor.py and saved output to /etc/admin.collector/source/plugins/data/5-host.status.sensor.json
Dec 31 06:05:31 adminserver-251 python[362451]: Processed 9-pve.sensor.py and saved output to /etc/admin.collector/source/plugins/data/9-pve.sensor.json
Dec 31 06:05:31 adminserver-251 python[362451]: Processed 2-common.sensor.py and saved output to /etc/admin.collector/source/plugins/data/2-common.sensor.json
Dec 31 06:05:32 adminserver-251 python[362451]: Processed 11-sslfolder.sensor.py and saved output to /etc/admin.collector/source/plugins/data/11-sslfolder.sensor.json
Dec 31 06:05:32 adminserver-251 python[362451]: Processed 4-ftg.sensor.py and saved output to /etc/admin.collector/source/plugins/data/4-ftg.sensor.json
Dec 31 06:05:32 adminserver-251 python[362451]: Processed 13-wordpress.sensor.py and saved output to /etc/admin.collector/source/plugins/data/13-wordpress.sensor.json
Dec 31 06:05:33 adminserver-251 python[362451]: Processed 12-sslstatic.sensor.py and saved output to /etc/admin.collector/source/plugins/data/12-sslstatic.sensor.json
Dec 31 06:05:33 adminserver-251 python[362451]: Processed 8-process.sensor.py and saved output to /etc/admin.collector/source/plugins/data/8-process.sensor.json
Dec 31 06:05:33 adminserver-251 python[362451]: Processed 6-http.status.sensor.py and saved output to /etc/admin.collector/source/plugins/data/6-http.status.sensor.json

Debug command

Command

journalctl -u admin_collector -n 20 -f

Output example

Dec 31 07:58:35 adminserver-251 python[364024]: Processed 14-zfs.sensor.py and saved output to /etc/admin.collector/source/plugins/data/14-zfs.sensor.json
Dec 31 07:58:36 adminserver-251 python[364024]: Processed 3-docker.sensor.py and saved output to /etc/admin.collector/source/plugins/data/3-docker.sensor.json
Dec 31 07:58:39 adminserver-251 python[364024]: Processed 1-cisco.sensor.py and saved output to /etc/admin.collector/source/plugins/data/1-cisco.sensor.json
Dec 31 07:58:39 adminserver-251 python[364024]: Processed 10-rabbitmq.sensor.py and saved output to /etc/admin.collector/source/plugins/data/10-rabbitmq.sensor.json
Dec 31 07:58:39 adminserver-251 python[364024]: Processed 7-mysql.sensor.py and saved output to /etc/admin.collector/source/plugins/data/7-mysql.sensor.json
Dec 31 07:58:41 adminserver-251 python[364024]: Processed 5-host.status.sensor.py and saved output to /etc/admin.collector/source/plugins/data/5-host.status.sensor.json
Dec 31 07:58:41 adminserver-251 python[364024]: Processed 9-pve.sensor.py and saved output to /etc/admin.collector/source/plugins/data/9-pve.sensor.json
Dec 31 07:58:42 adminserver-251 python[364024]: Processed 2-common.sensor.py and saved output to /etc/admin.collector/source/plugins/data/2-common.sensor.json
Dec 31 07:58:42 adminserver-251 python[364024]: Processed 11-sslfolder.sensor.py and saved output to /etc/admin.collector/source/plugins/data/11-sslfolder.sensor.json
Dec 31 07:58:42 adminserver-251 python[364024]: Processed 4-ftg.sensor.py and saved output to /etc/admin.collector/source/plugins/data/4-ftg.sensor.json
Dec 31 07:58:43 adminserver-251 python[364024]: Processed 13-wordpress.sensor.py and saved output to /etc/admin.collector/source/plugins/data/13-wordpress.sensor.json
Dec 31 07:58:43 adminserver-251 python[364024]: Processed 12-sslstatic.sensor.py and saved output to /etc/admin.collector/source/plugins/data/12-sslstatic.sensor.json
Dec 31 07:58:44 adminserver-251 python[364024]: Processed 8-process.sensor.py and saved output to /etc/admin.collector/source/plugins/data/8-process.sensor.json
Dec 31 07:58:44 adminserver-251 python[364024]: Processed 6-http.status.sensor.py and saved output to /etc/admin.collector/source/plugins/data/6-http.status.sensor.json
Dec 31 07:59:14 adminserver-251 python[364024]: Processed 14-zfs.sensor.py and saved output to /etc/admin.collector/source/plugins/data/14-zfs.sensor.json
Dec 31 07:59:15 adminserver-251 python[364024]: Processed 3-docker.sensor.py and saved output to /etc/admin.collector/source/plugins/data/3-docker.sensor.json
Dec 31 07:59:17 adminserver-251 python[364024]: Processed 1-cisco.sensor.py and saved output to /etc/admin.collector/source/plugins/data/1-cisco.sensor.json

Stop service

systemctl stop admin_collector.service

Setup RabbitMQ

Install RabbitMQ

apt update
apt install -y rabbitmq-server

Khởi động RabbitMQ và enable RabbitMQ

systemctl restart rabbitmq-server
systemctl enable rabbitmq-server

Check status RabbitMQ.

systemctl status rabbitmq-server

Stop RabbitMQ

systemctl stop rabbitmq-server

Delete all message queue

rabbitmqctl purge_queue <queue_name>

Delete topic

rabbitmqctl delete_queue <queue_name>

Create username and password

rabbitmqctl add_user hoanghd rabbitmq_password
rabbitmqctl set_user_tags hoanghd administrator
rabbitmqctl set_permissions -p / hoanghd ".*" ".*" ".*"

Verify the created username

Command verify

rabbitmqctl list_users

This is output example

Listing users ...
user    tags
admin   [administrator]
hoanghd [administrator]
guest   [administrator]
rabbitmq_management [administrator]

Enable plugins rabbitmq management

rabbitmq-plugins enable rabbitmq_management
rabbitmqctl add_user rabbitmq_management rabbitmq_password
rabbitmqctl set_user_tags rabbitmq_management administrator

Change password for user admin

rabbitmqctl change_password rabbitmq_management rabbitmq_password

Check API response

shell> curl -i -u rabbitmq_management:rabbitmq_password http://localhost:15672/api/queues
HTTP/1.1 200 OK
cache-control: no-cache
content-length: 2
content-security-policy: script-src 'self' 'unsafe-eval' 'unsafe-inline'; object-src 'self'
content-type: application/json
date: Tue, 31 Dec 2024 18:00:37 GMT
server: Cowboy
vary: accept, accept-encoding, origin

Update code

Stop services

systemctl stop admin_collector.service

Update code from Git repo.

cd /etc/admin.collector/source
git fetch
git reset --hard origin/main

Restart services

systemctl restart admin_collector.service

Check services status

systemctl status admin_collector.service

Debug command command

journalctl -u admin_collector -n 100 -f

Using Telegraf

Run script /etc/admin.collector/venv/bin/python /etc/admin.collector/source/plugins/run.py --project ${project} to get Json data in folder /etc/admin.collector/source/plugins/data

project='config'
/etc/admin.collector/venv/bin/python /etc/admin.collector/source/plugins/run.py --project ${project}

Verify folder /etc/admin.collector/source/plugins/data

shell> tree /etc/admin.collector/source/plugins/data
/etc/admin.collector/source/plugins/data
├── 10-rabbitmq.sensor.json
├── 11-sslfolder.sensor.json
├── 12-sslstatic.sensor.json
├── 13-wordpress.sensor.json
├── 14-zfs.sensor.json
├── 15-k8s.sensor.json
├── 1-cisco.sensor.json
├── 2-common.sensor.json
├── 3-docker.sensor.json
├── 4-ftg.sensor.json
├── 5-host.status.sensor.json
├── 6-http.status.sensor.json
├── 7-mysql.sensor.json
├── 8-process.sensor.json
└── 9-pve.sensor.json

0 directories, 15 files

Run /etc/admin.collector/source/plugins/tools/telegraf/telegraf.gen.config.py to get Telegraf config in folder /etc/admin.collector/source/plugins/tools/telegraf/telegraf.d/

python3 /etc/admin.collector/source/plugins/tools/telegraf/telegraf.gen.config.py

Verify folder /etc/admin.collector/source/plugins/tools/telegraf/telegraf.d/

shell> tree /etc/admin.collector/source/plugins/tools/telegraf/telegraf.d/
├── 10-rabbitmq.sensor.conf
├── 11-sslfolder.sensor.conf
├── 12-sslstatic.sensor.conf
├── 13-wordpress.sensor.conf
├── 14-zfs.sensor.conf
├── 15-k8s.sensor.conf
├── 1-cisco.sensor.conf
├── 2-common.sensor.conf
├── 3-docker.sensor.conf
├── 4-ftg.sensor.conf
├── 5-host.status.sensor.conf
├── 6-http.status.sensor.conf
├── 7-mysql.sensor.conf
├── 8-process.sensor.conf
└── 9-pve.sensor.conf

0 directories, 15 files

Copy config file /etc/admin.collector/source/plugins/tools/telegraf/telegraf.d/* to /etc/telegraf/telegraf.d/

cp /etc/admin.collector/source/plugins/tools/telegraf/telegraf.d/* /etc/telegraf/telegraf.d/

Test Telegraf config file

telegraf --config-directory /etc/telegraf/telegraf.d/ --test
telegraf --config-directory /etc/telegraf/telegraf.d/<filename>.conf --test

Example check file 15-k8s.sensor.conf

shell> 2025-02-18T08:08:37Z I! Loading config: /etc/telegraf/telegraf.d/15-k8s.sensor.conf
2025-02-18T08:08:37Z I! Starting Telegraf 1.32.1 brought to you by InfluxData the makers of InfluxDB
2025-02-18T08:08:37Z I! Available plugins: 235 inputs, 9 aggregators, 32 processors, 26 parsers, 62 outputs, 6 secret-stores
2025-02-18T08:08:37Z I! Loaded inputs: exec
2025-02-18T08:08:37Z I! Loaded aggregators:
2025-02-18T08:08:37Z I! Loaded processors:
2025-02-18T08:08:37Z I! Loaded secretstores:
2025-02-18T08:08:37Z W! Outputs are not used in testing mode!
2025-02-18T08:08:37Z I! Tags enabled: host=TO-STOR-MSG-K3S-MASTER-071
> metrics_report,description=Sensor\ not\ found\ in\ the\ config\ file,exporter_ipaddr=10.10.45.71,host=TO-STOR-MSG-K3S-MASTER-071,metric_name=kubernetes_exporter,role=report value=1 1739866118000000000
> kube_node_status,host=TO-STOR-MSG-K3S-MASTER-071,node=soc-stor-msg-k3s-worker-074-f911ff2b,node_ip=10.10.45.74,node_roles=worker,os_image=Ubuntu\ 20.04.6\ LTS,role=node_status,version=v1.25.3+k3s1 value=1 1739866118000000000
> kube_node_cpu_limit,host=TO-STOR-MSG-K3S-MASTER-071,node=soc-stor-msg-k3s-worker-074-f911ff2b,node_ip=10.10.45.74,node_roles=worker,os_image=Ubuntu\ 20.04.6\ LTS,role=node_cpu,version=v1.25.3+k3s1 value=32 1739866118000000000
> kube_node_memory_limit,host=TO-STOR-MSG-K3S-MASTER-071,node=soc-stor-msg-k3s-worker-074-f911ff2b,node_ip=10.10.45.74,node_roles=worker,os_image=Ubuntu\ 20.04.6\ LTS,role=node_memory,version=v1.25.3+k3s1 value=270366121984 1739866118000000000
> kube_node_max_pods,host=TO-STOR-MSG-K3S-MASTER-071,node=soc-stor-msg-k3s-worker-074-f911ff2b,node_ip=10.10.45.74,node_roles=worker,os_image=Ubuntu\ 20.04.6\ LTS,role=node_pods,version=v1.25.3+k3s1 value=110 1739866118000000000
> kube_node_status,host=TO-STOR-MSG-K3S-MASTER-071,node=soc-stor-msg-k3s-worker-075-fca3f6ce,node_ip=10.10.45.75,node_roles=worker,os_image=Ubuntu\ 20.04.6\ LTS,role=node_status,version=v1.25.3+k3s1 value=1 1739866118000000000
> kube_node_cpu_limit,host=TO-STOR-MSG-K3S-MASTER-071,node=soc-stor-msg-k3s-worker-075-fca3f6ce,node_ip=10.10.45.75,node_roles=worker,os_image=Ubuntu\ 20.04.6\ LTS,role=node_cpu,version=v1.25.3+k3s1 value=56 1739866118000000000
> kube_node_memory_limit,host=TO-STOR-MSG-K3S-MASTER-071,node=soc-stor-msg-k3s-worker-075-fca3f6ce,node_ip=10.10.45.75,node_roles=worker,os_image=Ubuntu\ 20.04.6\ LTS,role=node_memory,version=v1.25.3+k3s1 value=270372302848 1739866118000000000
> kube_node_max_pods,host=TO-STOR-MSG-K3S-MASTER-071,node=soc-stor-msg-k3s-worker-075-fca3f6ce,node_ip=10.10.45.75,node_roles=worker,os_image=Ubuntu\ 20.04.6\ LTS,role=node_pods,version=v1.25.3+k3s1 value=110 1739866118000000000
> kube_node_status,host=TO-STOR-MSG-K3S-MASTER-071,node=soc-stor-msg-k3s-worker-076-6d086f66,node_ip=10.10.45.76,node_roles=worker,os_image=Ubuntu\ 20.04.6\ LTS,role=node_status,version=v1.25.3+k3s1 value=1 1739866118000000000

Restart telegraf.service

systemctl restart telegraf.service

Check status telegraf.service

systemctl status telegraf.service

Check log

tail -f /var/log/telegraf/telegraf.log

If you encounter an error where Python files cannot be executed when checking the log at /var/log/telegraf/telegraf.log, follow these steps: Go to /usr/lib/systemd/system/telegraf.service and change User=telegraf to User=root. Run the command systemctl daemon-reload and restart Telegraf.

[Unit]
Description=Telegraf
Documentation=https://github.com/influxdata/telegraf
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
NotifyAccess=all
EnvironmentFile=-/etc/default/telegraf
#User=telegraf
User=root
ImportCredential=telegraf.*
ExecStart=/usr/bin/telegraf -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d $TELEGRAF_OPTS
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartForceExitStatus=SIGPIPE
KillMode=mixed
LimitMEMLOCK=8M:8M
PrivateMounts=true

[Install]
WantedBy=multi-user.target

Personal Ops Runbook

Infra Notes

Overview

Install the required packages

Creating the application directory.

Create the virtual environment

Activate the virtual environment

Install the modules.

Ubuntu 22.04 Operating System

Debian (Proxmox) Operating System

Python version 3.10.x

Python version 3.11.x

Install git

Clone plugins

Create the requirements file

Install the required packages

Install SNMP if you are monitoring Cisco or Fortigate devices

Install lm-sensors if you want to monitor server temperature

Install sysstat if you want to monitor disk IOPS

Test script

Create a systemd service

Enable and start the service

Check status service

Command

Output tham khảo

Debug command

Command

Output example

Stop service

Setup RabbitMQ

Install RabbitMQ

Khởi động RabbitMQ và enable RabbitMQ

Check status RabbitMQ.

Stop RabbitMQ

Delete all message queue

Delete topic

Create username and password

Verify the created username

Enable plugins rabbitmq management

Change password for user admin

Check API response

Update code

Stop services

Update code from Git repo.

Restart services

Check services status

Debug command command

Using Telegraf