Monitoring

Data Types

Monitoring has 3 general catergories:

1. Real-time metrics

   metric = {
       "cpu_pct": 87.2,
       "disk_full": False,
   }

2. Events

   event  = {
       "timestamp": 1721692801,
       "msg": "auth failed",
       "user": "amy",
   }

3. Aggregated and summarized historic trends

   trend  = [
       (1721692800, 87.2),
       (1721692860, 91.0),
   ]

System Monitoring

Since the kernel controls a system’s CPU, memory, I/O, and devices, most of the interesting system-level state information you might want to monitor lives somewhere inside the kernel.

It’s generally easiest and most efficient to access values directly from the kernel (through sysctl or /proc) if you can. Example:

cat /proc/loadavg
0.00 0.00 0.00 1/162 67339

Some more commands to derive metrics

df      # free/used disk space + inodes
du      # directory sizes
free    # free/used/swap memory
iostat  # disk performance + throughput
lsof    # open files + network ports
mpstat  # per-CPU utilization (multiprocessor)
vmstat  # process, CPU, memory stats
sar     # system activity report

System integrety verification

System integrity verification is the validation of the current state of the system against a known-good baseline.

• Compare the contents of the system files (kernel, executable commands, config files) with checksum.
• If the checksum value of the file in the running system is different from that of the baseline version, a sysadmin is notified.

Application Monitoring

Log Monitoring

Log monitoring involves:

• grepping through log files to find interesting data you’d like to monitor
• pulling out that data
• processing it into a form that’s usable for analysis, display, and alerting

Process Monitoring

Supervisor

Supervisor and its server process supervisord help you monitor processes and generate events or notifications when the processes exit or throw an exception. Similar to how systemd monitors processes.