Logo

Personal Ops Runbook

Personal runbook covering infrastructure operations for Cloud, Kubernetes, OpenStack, and Ceph environments. Includes deployment and teardown procedures, node management, cluster monitoring setup, and incident response workflows compiled from day-to-day operational work. Intended strictly for personal reference — configurations and scripts are environment-specific and not guaranteed to work as-is elsewhere.

OpenStack command-line interface cheat sheet


Identity (keystone).

List all users.

shell> openstack user list
+----------------------------------+-------+
| ID                               | Name  |
+----------------------------------+-------+
| 0877790a087b428193d1f4b00feaed5c | admin |
| 7659f798cdca4790ac7c900e140c3e89 | swift |
+----------------------------------+-------+

List Identity service catalog

shell> openstack catalog list
+----------+--------------+------------------------------------------------------------------------------+
| Name     | Type         | Endpoints                                                                    |
+----------+--------------+------------------------------------------------------------------------------+
| keystone | identity     | RegionOne                                                                    |
|          |              |   internal: http://swift-proxy:5000/v3/                                      |
|          |              | RegionOne                                                                    |
|          |              |   public: http://swift-proxy:5000/v3/                                        |
|          |              | RegionOne                                                                    |
|          |              |   admin: http://swift-proxy:5000/v3/                                         |
|          |              |                                                                              |
| swift    | object-store | RegionOne                                                                    |
|          |              |   public: http://swift-proxy:8080/v1/AUTH_99ebdac9f0bf4018a826f43e2b54f8fe   |
|          |              | RegionOne                                                                    |
|          |              |   admin: http://swift-proxy:8080/v1                                          |
|          |              | RegionOne                                                                    |
|          |              |   internal: http://swift-proxy:8080/v1/AUTH_99ebdac9f0bf4018a826f43e2b54f8fe |
|          |              |                                                                              |
+----------+--------------+------------------------------------------------------------------------------+

Object Storage (swift)

Create the container.

openstack container create container1
openstack container create container2
openstack container create container3

Output example.

shell> openstack container create container3
+---------------------------------------+------------+------------------------------------+
| account                               | container  | x-trans-id                         |
+---------------------------------------+------------+------------------------------------+
| AUTH_99ebdac9f0bf4018a826f43e2b54f8fe | container3 | tx0cb6e293cfee4ad0a68cd-0066817ce7 |
+---------------------------------------+------------+------------------------------------+

Delete a container.

openstack container delete container3

Nếu bạn gặp lỗi Conflict (HTTP 409) khi bạn cố gắng xóa một container trong OpenStack thường xảy ra do một trong các lý do sau:

  • Container không rỗng: Container đang chứa các objects và cần phải được làm trống trước khi xóa.
  • Container đang được sử dụng: Container có thể đang được sử dụng bởi một tiến trình khác hoặc có các khóa (locks) đang hoạt động.
shell> openstack container delete container1
Conflict (HTTP 409) (Request-ID: txeb786d04a5b249b4ae92e-0066af29de)

Các bước khắc phục:

Kiểm tra và xóa các đối tượng trong container:

Liệt kê các object trong container:

shell> openstack object list container1
+------------+
| Name       |
+------------+
| Object-132 |
+------------+

Xóa từng object trong container:

openstack object delete container1 Object-132

Hoặc xóa tất cả các đối tượng trong container bằng một lệnh vòng lặp:

for obj in $(openstack object list container1 -f value -c Name); do
    openstack object delete container1 "$obj"
done

Xóa lại container sau khi đã làm trống:

openstack container delete container1

List container.

shell> openstack container list
+---------------------+
| Name                |
+---------------------+
| container1          |
| container2          |
+---------------------+

Upload a file named to a container named in the OpenStack Object Storage service (Swift).

shell> openstack object create container1 testfile
+----------+------------+----------------------------------+
| object   | container  | etag                             |
+----------+------------+----------------------------------+
| testfile | container1 | 3566de3a97906edb98d004d6b947ae9b |
+----------+------------+----------------------------------+

List object in the container.

shell> openstack object list container1
+--------------------+
| Name               |
+--------------------+
| admin-openrc       |
| demo-openrc        |
| mariadb_repo_setup |
| sh.sh              |
| testfile           |
+--------------------+

View object size.

shell> openstack object show container1 testfile
+----------------+---------------------------------------+
| Field          | Value                                 |
+----------------+---------------------------------------+
| account        | AUTH_99ebdac9f0bf4018a826f43e2b54f8fe |
| container      | container1                            |
| content-length | 209715200                             |
| content-type   | application/octet-stream              |
| etag           | 3566de3a97906edb98d004d6b947ae9b      |
| last-modified  | Sun, 30 Jun 2024 15:47:49 GMT         |
| object         | testfile                              |
+----------------+---------------------------------------+

Or

shell> openstack object show container1 testfile -c content-length -f value
209715200

Display statistics for a Swift account.

root@swift-proxy1:~# swift stat
                        Account: AUTH_99ebdac9f0bf4018a826f43e2b54f8fe
                     Containers: 3
                        Objects: 5
                          Bytes: 209758992
Containers in policy "policy-0": 3
   Objects in policy "policy-0": 5
     Bytes in policy "policy-0": 209758992
                         Server: nginx/1.18.0 (Ubuntu)
                   Content-Type: text/plain; charset=utf-8
                     Connection: keep-alive
                    X-Timestamp: 1719686215.33484
                  Accept-Ranges: bytes
                           Vary: Accept
                     X-Trans-Id: tx594d2976586546deb97a7-0066817b64
         X-Openstack-Request-Id: tx594d2976586546deb97a7-0066817b64
        Content-Security-Policy: upgrade-insecure-requests

Swift list will display a list of all containers in the Swift account.

shell> swift list
container1
container2

Used to display statistics and metadata for a specific container named within the Swift object storage system.

shell> swift stat container1
                Account: AUTH_99ebdac9f0bf4018a826f43e2b54f8fe
              Container: container1
                Objects: 5
                  Bytes: 209758992
               Read ACL:
              Write ACL:
                Sync To:
               Sync Key:
                 Server: nginx/1.18.0 (Ubuntu)
           Content-Type: text/plain; charset=utf-8
             Connection: keep-alive
            X-Timestamp: 1719686215.49936
          Last-Modified: Sat, 29 Jun 2024 18:51:49 GMT
          Accept-Ranges: bytes
       X-Storage-Policy: Policy-0
                   Vary: Accept
             X-Trans-Id: tx7876ba311bf24efcbea47-0066817c22
 X-Openstack-Request-Id: tx7876ba311bf24efcbea47-0066817c22
Content-Security-Policy: upgrade-insecure-requests

Performs a comprehensive health check and gathers metrics across all available categories from the Swift cluster.

swift-recon --all

The --diskusage option is used to report on the disk usage across all storage nodes in the Swift cluster.

shell> swift-recon --diskusage
===============================================================================
--> Starting reconnaissance on 4 hosts (object)
===============================================================================
[2024-06-30 15:59:36] Checking disk usage now
Distribution Graph:
  0%   18 *********************************************************************
  1%    2 *******
Disk usage: space used: 5587931136 of 643930521600
Disk usage: space free: 638342590464 of 643930521600
Disk usage: lowest: 0.8%, highest: 1.45%, avg: 0.8677847917684416%
===============================================================================

Upload folder.

find /home/hoanghd/resources -type f -print0 | xargs -0 -I {} openstack object create container1 {} --name '{}/{}'

Create Access and Secret.

shell> pip install python-openstackclient
shell> openstack ec2 credentials create
shell> openstack ec2 credentials list
+----------------------------------+----------------------------------+----------------------------------+----------------------------------+
| Access                           | Secret                           | Project ID                       | User ID                          |
+----------------------------------+----------------------------------+----------------------------------+----------------------------------+
| 7740ac01e943412f9aca2067fe37bbc6 | 66d34772e2b24ecebb6451244b6f8f88 | 99ebdac9f0bf4018a826f43e2b54f8fe | 0877790a087b428193d1f4b00feaed5c |
+----------------------------------+----------------------------------+----------------------------------+----------------------------------+

Display detailed information about a specific project in OpenStack

shell> openstack project show 99ebdac9f0bf4018a826f43e2b54f8fe
+-------------+-----------------------------------------------+
| Field       | Value                                         |
+-------------+-----------------------------------------------+
| description | Bootstrap project for initializing the cloud. |
| domain_id   | default                                       |
| enabled     | True                                          |
| id          | 99ebdac9f0bf4018a826f43e2b54f8fe              |
| is_domain   | False                                         |
| name        | admin                                         |
| options     | {}                                            |
| parent_id   | default                                       |
| tags        | []                                            |
+-------------+-----------------------------------------------+

Display detailed information about a specific user in OpenStack

shell> openstack user show 0877790a087b428193d1f4b00feaed5c
+---------------------+----------------------------------+
| Field               | Value                            |
+---------------------+----------------------------------+
| domain_id           | default                          |
| enabled             | True                             |
| id                  | 0877790a087b428193d1f4b00feaed5c |
| name                | admin                            |
| options             | {}                               |
| password_expires_at | None                             |
+---------------------+----------------------------------+

List the containers in the Swift account.

curl http://swift-proxy:8080/v1/AUTH_99ebdac9f0bf4018a826f43e2b54f8fe
container1
container2

Sync config object server

for device in {1..4}; do
  scp /etc/swift/swift.conf root@swift-object${device}:/etc/swift
done

This script is used to format disk for object server and mount it to /srv/node/sd{b..f} directory.

mkfs.xfs /dev/sd{b..f}
mkdir -p /srv/node/sd{b..f}

for drive in {b..f}; do
    UUID=$(blkid -s UUID -o value /dev/sd${drive})
    echo "UUID=${UUID} /srv/node/sd${drive} xfs noatime 0 2" >> /etc/fstab
done

mount -a