1. PrometheusAlertManager overview

PrometheusAlertManager is a CLI tool designed to manage and monitor your projects with ease. It integrates with Prometheus to fetch metrics, evaluate rules, and send alerts via Telegram. The tool supports multiple configurations and can be extended to use different databases like MongoDB or File.

2. Features

• Load and parse configuration files
• Monitor metrics and send alerts
• Log information and errors
• Send notifications to Telegram
• Support for multiple database backends (file, MongoDB)

3. Project Structure

├── README.md
├── cli.py
├── config
│   ├── prod
│   │   ├── cisco.yml
│   │   ├── ceph.yml
│   │   ├── influxdb.yml
│   │   ├── main.yml
│   │   └── prometheus.yml
│   └── stg
│       ├── ceph.yml
│       ├── cisco.yml
│       ├── main.yml
│       └── proxmox.yml
├── docs
│   ├── deploy.container.prod.md
│   ├── deploy.container.staging.md
│   ├── guild.influxdb.md
│   ├── install.docker.md
│   ├── install.telegraf.md
│   └── telegram.bot.md
├── run.py
├── sendmsg.py
└── update.code.sh

4. Getting Started

4.1. Prerequisites

Services

• Prometheus
• RabbitMQ
• MongoDB
• Git

OS version

• Proxmox 7.x, 8.x

Python version

• Python 3.x Ex: (3.10.12, 3.11.2,...)

4.2. Installation

Install the required packages

apt update
apt install python3-pip python3-venv -y

Create application directory.

mkdir -p /etc/alertmanager

Create the virtual environment

python3 -m venv /etc/alertmanager/venv

Install git

apt update
apt install git -y

7. RabbitMQ

Install RabbitMQ

apt update
apt install -y rabbitmq-server

Start RabbitMQ and enable RabbitMQ

systemctl restart rabbitmq-server
systemctl enable rabbitmq-server

Check status RabbitMQ.

systemctl status rabbitmq-server

Stop RabbitMQ

systemctl stop rabbitmq-server

Purge all messages in Queue

rabbitmqctl purge_queue <queue_name>

Delete topic

rabbitmqctl delete_queue <queue_name>

Create username and password

rabbitmqctl add_user hoanghd rabbitmq_password
rabbitmqctl set_user_tags hoanghd administrator
rabbitmqctl set_permissions -p / hoanghd ".*" ".*" ".*"

Verify created username

The following command lists all the users in RabbitMQ along with their tags:

rabbitmqctl list_users

Here is a sample output

Listing users ...
user    tags
admin   [administrator]
hoanghd [administrator]
guest   [administrator]
rabbitmq_management [administrator]

Enable plugins rabbitmq management

rabbitmq-plugins enable rabbitmq_management
rabbitmqctl add_user rabbitmq_management rabbitmq_password
rabbitmqctl set_user_tags rabbitmq_management administrator

Change password for user admin

rabbitmqctl change_password rabbitmq_management rabbitmq_password

Check API response

shell> curl -i -u rabbitmq_management:rabbitmq_password http://localhost:15672/api/queues
HTTP/1.1 200 OK
cache-control: no-cache
content-length: 2
content-security-policy: script-src 'self' 'unsafe-eval' 'unsafe-inline'; object-src 'self'
content-type: application/json
date: Tue, 31 Dec 2024 18:00:37 GMT
server: Cowboy
vary: accept, accept-encoding, origin

Clone plugins

git clone https://github.com/hoanghd164/PrometheusAlertManager.git /opt/alertmanager

Create symbolic

ln -s /opt/alertmanager /etc/alertmanager/source

Create the requirements file

cat > /etc/alertmanager/source/requirements.txt << 'OEF'
requests
pyyaml
pika
prettytable
jinja2
pymongo
influxdb_client
prometheus_client
influxdb
ping3
OEF

Install the required packages

/etc/alertmanager/venv/bin/python -m pip install -r /etc/alertmanager/source/requirements.txt

5. Test script.

For example, the project_name is stg.

project_name='stg'

Test the script using the command below

cd /etc/alertmanager/source
source /etc/alertmanager/venv/bin/activate
python3 run.py --project ${project_name}

Or you can also test with the command.

/etc/alertmanager/venv/bin/python /etc/alertmanager/source/run.py --project ${project_name}

6. Create a systemd service

Create service file

For the service alertmanager.service

cat > /etc/systemd/system/alertmanager.service << OEF
[Unit]
Description=Exporter Custom Metrics Service
After=network.target

[Service]
WorkingDirectory=/etc/alertmanager/source
ExecStart=/etc/alertmanager/venv/bin/python /etc/alertmanager/source/run.py --project ${project_name}
Restart=always
User=root
Group=root
Environment=PYTHONUNBUFFERED=1

[Install]
WantedBy=multi-user.target
OEF

For the sendmsg.service

cat > /etc/systemd/system/sendmsg.service << OEF
[Unit]
Description=Send Message Service
After=network.target

[Service]
WorkingDirectory=/etc/alertmanager/source
ExecStart=/etc/alertmanager/venv/bin/python /etc/alertmanager/source/sendmsg.py --project ${project_name}
Restart=always
User=root
Group=root
Environment=PYTHONUNBUFFERED=1

[Install]
WantedBy=multi-user.target
OEF

Reload systemctl daemon

systemctl daemon-reload

Enable and start the alertmanager.service

systemctl enable alertmanager.service
systemctl restart alertmanager.service

Enable and start the sendmsg.service

systemctl enable sendmsg.service
systemctl restart sendmsg.service

Check status service

For the alertmanager.service

systemctl status alertmanager.service

Output

● alertmanager.service - Exporter Custom Metrics Service
     Loaded: loaded (/etc/systemd/system/alertmanager.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2025-01-01 00:52:23 +07; 4s ago
   Main PID: 1404404 (python)
      Tasks: 1 (limit: 77110)
     Memory: 28.4M
        CPU: 277ms
     CGroup: /system.slice/alertmanager.service
             └─1404404 /etc/alertmanager/venv/bin/python /etc/alertmanager/source/run.py --project stg

Jan 01 00:52:24 monitoring-100 python[1404404]:         - Using project: stg
Jan 01 00:52:24 monitoring-100 python[1404404]:         - Using database file: ./db/db.stg.json
Jan 01 00:52:24 monitoring-100 python[1404404]:         - Using configuration file: ./config/config.stg.yml

For the sendmsg.service

Command

systemctl status sendmsg.service

Output

● sendmsg.service - Send Message Service
     Loaded: loaded (/etc/systemd/system/sendmsg.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2025-01-01 08:06:12 +07; 10s ago
   Main PID: 1583787 (python)
      Tasks: 1 (limit: 77110)
     Memory: 19.0M
        CPU: 244ms
     CGroup: /system.slice/sendmsg.service
             └─1583787 /etc/alertmanager/venv/bin/python /etc/alertmanager/source/sendmsg.py --project ntjc

Jan 01 08:06:12 monitoring-100 systemd[1]: Started Send Message Service.
Jan 01 08:06:12 monitoring-100 python[1583789]: TERM environment variable not set.
Jan 01 08:06:12 monitoring-100 python[1583787]: Waiting for messages. To exit press CTRL+C

Debug command

For the alertmanager.service

shell> journalctl -u alertmanager -n 30 -f
Jan 01 00:52:23 monitoring-100 systemd[1]: Started Exporter Custom Metrics Service.
Jan 01 00:52:24 monitoring-100 python[1404406]: TERM environment variable not set.
Jan 01 00:52:24 monitoring-100 python[1404404]:         **************************************************
Jan 01 00:52:24 monitoring-100 python[1404404]:         *                                                *
Jan 01 00:52:24 monitoring-100 python[1404404]:         *              WELCOME TO THE CLI TOOL           *
Jan 01 00:52:24 monitoring-100 python[1404404]:         *                                                *
Jan 01 00:52:24 monitoring-100 python[1404404]:         **************************************************
Jan 01 00:52:24 monitoring-100 python[1404404]:         *                                                *
Jan 01 00:52:24 monitoring-100 python[1404404]:         *  This tool helps you manage and monitor your   *
Jan 01 00:52:24 monitoring-100 python[1404404]:         *  projects with ease.                           *
Jan 01 00:52:24 monitoring-100 python[1404404]:         *                                                *
Jan 01 00:52:24 monitoring-100 python[1404404]:         *  Features:                                     *
Jan 01 00:52:24 monitoring-100 python[1404404]:         *  - Load and parse configuration files          *
Jan 01 00:52:24 monitoring-100 python[1404404]:         *  - Monitor metrics and send alerts             *
Jan 01 00:52:24 monitoring-100 python[1404404]:         *  - Log information and errors                  *
Jan 01 00:52:24 monitoring-100 python[1404404]:         *  - Send notifications to Telegram              *
Jan 01 00:52:24 monitoring-100 python[1404404]:         *                                                *
Jan 01 00:52:24 monitoring-100 python[1404404]:         *  Contact:                                      *
Jan 01 00:52:24 monitoring-100 python[1404404]:         *  - Author: Hà Đăng Hoàng                       *
Jan 01 00:52:24 monitoring-100 python[1404404]:         *  - Email: hoanghd3@vng.com.vn                  *
Jan 01 00:52:24 monitoring-100 python[1404404]:         *  - Mobile: 0962277556                          *
Jan 01 00:52:24 monitoring-100 python[1404404]:         *                                                *
Jan 01 00:52:24 monitoring-100 python[1404404]:         *  Please wait while we load your configuration. *
Jan 01 00:52:24 monitoring-100 python[1404404]:         *                                                *
Jan 01 00:52:24 monitoring-100 python[1404404]:         **************************************************
Jan 01 00:52:24 monitoring-100 python[1404404]:         Proect information:
Jan 01 00:52:24 monitoring-100 python[1404404]:         - Using project: stg
Jan 01 00:52:24 monitoring-100 python[1404404]:         - Using database file: ./db/db.stg.json
Jan 01 00:52:24 monitoring-100 python[1404404]:         - Using configuration file: ./config/config.stg.yml
Jan 01 00:52:24 monitoring-100 python[1404404]: --------------------> Start running the script <--------------------
Jan 01 00:52:24 monitoring-100 python[1404404]: Jan 01 00:52:24 monitoring-100 python[1404404]: demo_metric returned no results
Jan 01 00:52:24 monitoring-100 python[1404404]: [./db/db.stg.json] Finished for the 1th (0.006888628005981445 seconds)

For the sendmsg.service

shell> journalctl -u sendmsg -n 30 -f
Jan 01 08:06:12 monitoring-100 systemd[1]: Started Send Message Service.
Jan 01 08:06:12 monitoring-100 python[1583789]: TERM environment variable not set.
Jan 01 08:06:12 monitoring-100 python[1583787]: Waiting for messages. To exit press CTRL+C

Stop service

systemctl stop alertmanager.service
systemctl stop sendmsg.service

9. Install MongoDB

Create docker-compose.yml.

cat > /etc/alertmanager/source/mongodb/docker-compose.yml << 'OEF'
version: '3.8'

services:
  mongodb:
    image: mongo:latest
    container_name: mongodb
    network_mode: host
    environment:
      MONGO_INITDB_ROOT_USERNAME: root
      MONGO_INITDB_ROOT_PASSWORD: mongodb_password
    volumes:
      - ./db:/data/db
OEF

Run stack

docker-compose -f /etc/alertmanager/source/mongodb/docker-compose.yml up -d

Edit config use type database is mongodb

  database:
    prefix: alert_
    type: mongodb # mongodb or file
    host: localhost
    port: 27017
    username: root
    password: mongodb_password

How to access MongoDB using shell

I use the MongoDB container to avoid installing additional tools. Access the MongoDB container with the command below.

docker exec -it mongodb bash

Access the MongoDB CLI.

mongosh --host localhost --port 27017 -u root -p mongodb_password

Sample output of the mongosh command

Current Mongosh Log ID: 677570a0e68e6413a6fe6910
Connecting to:          mongodb://<credentials>@localhost:27017/?directConnection=true&serverSelectionTimeoutMS=2000&appName=mongosh+2.3.2
Using MongoDB:          8.0.3
Using Mongosh:          2.3.2

For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/


To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy).
You can opt-out by running the disableTelemetry() command.

------
   The server generated these startup warnings when booting
   2025-01-01T16:22:09.858+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
   2025-01-01T16:22:11.090+00:00: You are running on a NUMA machine. We suggest launching mongod like this to avoid performance problems: numactl --interleave=all mongod [other options]
   2025-01-01T16:22:11.090+00:00: For customers running the current memory allocator, we suggest changing the contents of the following sysfsFile
   2025-01-01T16:22:11.090+00:00: For customers running the current memory allocator, we suggest changing the contents of the following sysfsFile
   2025-01-01T16:22:11.090+00:00: We suggest setting the contents of sysfsFile to 0.
   2025-01-01T16:22:11.090+00:00: Your system has glibc support for rseq built in, which is not yet supported by tcmalloc-google and has critical performance implications. Please set the environment variable GLIBC_TUNABLES=glibc.pthread.rseq=0
   2025-01-01T16:22:11.090+00:00: vm.max_map_count is too low
   2025-01-01T16:22:11.090+00:00: We suggest setting swappiness to 0 or 1, as swapping can cause performance problems.
------

test>

Command to check MongoDB version.

prod> db.version()
8.0.3

To display the list of databases in MongoDB, use the following command in the mongosh shell:

test> show databases
admin   100.00 KiB
config   12.00 KiB
local    72.00 KiB
prod     56.00 KiB
test>

Hoặc

test> show dbs
admin   100.00 KiB
config   12.00 KiB
local    72.00 KiB
prod     56.00 KiB

Select database.

test> use prod
switched to db prod

List collections in the database

prod> show collections
prod

View the contents of a collection, for example, the prod collection.

db.prod.find()

Or to display data in a prettier format.

db.users.find().pretty()

To limit the number of elements in the data array, you need to use projection or aggregation pipeline.

prod> db.prod.find({}, { data: { $slice: 1 } })
[
  {
    _id: 'prod',
    data: [
      {
        key: 'TFMwdExTQkRaWEJvSUUxaGJtRm5aWElnVTNSaGRIVnpJQzB0TFMwS1VYVmxjbms2SUdOdmRXNTBLR05sY0doZmJXZHlYMjFsZEdGa1lYUmhLUW89NQ==',
        id: 'LS0tLSBDZXBoIE1hbmFnZXIgU3RhdHVzIC0tLS0KUXVlcnk6IGNvdW50KGNlcGhfbWdyX21ldGFkYXRhKQo=',
        verify: 26,
        resend: 25,
        value: 5,
        old_value: [ 5 ],
        change: false,
        compare: true,
        alert: true,
        count: 26,
        timestamp: 1735750140.4607637
      }
    ]
  }
]

To view detailed information (number of documents, indexes, etc.) of the collection, use:

db.prod.stats()

The limit(1) command limits the number of documents returned from the query, without limiting the number of elements in the data array within a document.

db.prod.find().limit(1)

This command will delete the prod database, all collections, and data within this database:

use prod
db.dropDatabase()

If you only want to delete data in a collection (e.g., the prod collection). The result of this command is that the prod collection still exists, but all data in the collection will be deleted.

use prod
db.users.deleteMany({})

If you want to delete data in all collections but keep the database structure:

use prod
db.getCollectionNames().forEach(function(collection) {
  db[collection].deleteMany({})
})

If you only want to delete some documents in the prod collection, you can use the command with a condition (this command only deletes documents that match the condition in the users collection):

use prod
db.prod.deleteMany({ key: "some_condition" })

Relationship between Database, Collection, and Document

• A Database contains multiple Collections.
• A Collection contains multiple Documents.
• A Document is the smallest entity, storing data in JSON format.
• Example:

Database: prod
    ├── Collection: users
    │       ├── Document: { "name": "Alice", "age": 25 }
    │       └── Document: { "name": "Bob", "age": 30 }
    ├── Collection: orders
    │       ├── Document: { "order_id": 1, "amount": 100 }
    │       └── Document: { "order_id": 2, "amount": 200 }

10. Update code

Stop services

systemctl stop alertmanager.service
systemctl stop sendmsg.service

Update code from Git repo.

cd /etc/alertmanager/source
git fetch
git reset --hard origin/main

Clean database (Option)

rm -rf /etc/alertmanager/source/db/*

Restart services

systemctl restart alertmanager.service
systemctl restart sendmsg.service

Check services status

systemctl status alertmanager.service
systemctl status sendmsg.service

Using script

systemctl stop alertmanager.service
systemctl stop sendmsg.service
cd /etc/alertmanager/source
git fetch
git reset --hard origin/main
rm -rf /etc/alertmanager/source/db/*
systemctl restart alertmanager.service
systemctl restart sendmsg.service
systemctl status alertmanager.service
systemctl status sendmsg.service

Debug command

journalctl -u alertmanager.service -n 100 -f
journalctl -u sendmsg -n 100 -f

Remove alertmanager

rm -rf /etc/alertmanager
rm -rf /opt/alertmanager
systemctl stop alertmanager.service
systemctl disable alertmanager.service
systemctl stop alertmanager.service
systemctl disable alertmanager.service
rm /etc/systemd/system/sendmsg.service
rm /etc/systemd/system/alertmanager.service
systemctl daemon-reload

11. License

This project is licensed under the MIT License.

12. Contact

Author: Hà Đăng Hoàng \ Email: hoanghd@gmail.com \ Website: https://wiki.hoanghd.com \ Mobile: +84 962 277 556