Systemd

What is systemd?

Pithy
systemd is a Linux init system and service manager that starts services and keeps them alive under a unified dependency and logging framework.

Overview

Question
What’s the point, why not just use shell scripts to start everything?

Answer
Shell scripts don’t scale, there’s no standard way to handle logging, restarts, dependencies, or resource limits across every service.
systemd defines the schema (vocabulary) - what keys are valid, what values they accept, what they do. e.g. SuccessExitStatus=, Restart=
the unit file author fills it in - picking only the keys relevant to their service, leaving the rest at their defaults. e.g. SuccessExitStatus=DATAERR, Restart=on-failure
it also allows services to be managed through one interface systemctl
This way each unit has a standardized way to control their service without having to rewrite the same logic in shell - things like logging, restart behavior, dependency ordering, resource limits and exit code handling are all handled by systemd uniformly.
Instead of every service author writing their own daemonizing and logging code, they just declare Type=forking or StandardOutput=journal and systemd handles it.

Question
Daemons?

Answer
Originally: from Greek/Latin concept “a supernatural being of a nature intermediate between that of gods and men or a guiding spirit”.
Then: borrowed by physicist James Maxwell for a thought experiment: an imaginary agent silently sorting gas molecules by speed.
Now: it means an invisible background processes.

Unit files

Unit and service definition

Pithy
Unit: a declarative description of a managed thing
Service: the thing that is running
A similar relationship/abstraction to a program and a process

Unit file locations

# Main 
/usr/lib/systemd/system
/lib/systemd/system # some systems
# Custom unit files
/etc/systemd/system
/run/systemd/system

Unit file format

custom.service
# /etc/systemd/system/custom.service

# Unit section optional for metadata
[Unit]
Description=Test Service

# Service section is the actual config for what the service does
[Service]
# How the service tells the manager that startup has finished
Type=oneshot
# Actual command to run
ExecStart=/bin/echo "hello from custom.service"

# Install section says what to do if the service is "enabled" (auto-start)
[Install]  
# Creates a symlink in multi-user.target.wants/ that points to my service
# multi-user.target is like a checkpoint that certain services have been started
# so once it hits that checkpoint my service is started (if enabled).
# This doesn't affect if the service is run manually with "start".
WantedBy=multi-user.target

Unit file examples

sshd

sshd is poorly packaged in debian/ubuntu as ssh - try to ignore this as it means sshd and is separate from the ssh binary.

I will omit the less interesting parts of the unit. See the above man pages (man 5 systemd.unit)

ssh.service
#/etc/lib/systemd/system/ssh.service
[Unit]
# Once network manager is started (not connected) https://systemd.io/NETWORK_ONLINE/
# and audit logging deamon is up
After=network.target auditd.service
# if you create the following file: /etc/ssh/sshd_not_to_be_run
# ssh service will refuse to start with any of the systemctl start commands:
#   start, restart, enable --now
# This is like the WannaCry dns kill switch
ConditionPathExists=!/etc/ssh/sshd_not_to_be_run

# TODO! Continue 
[Service]
EnvironmentFile=-/etc/default/ssh
ExecStartPre=/usr/sbin/sshd -t
ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
ExecReload=/usr/sbin/sshd -t
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartPreventExitStatus=255
Type=notify
RuntimeDirectory=sshd
RuntimeDirectoryMode=0755

[Install]
WantedBy=multi-user.target
# Aliased to sshd to ease of administration
Alias=sshd.service

Documentation

Bash
man 5 systemd.unit              # [Unit] and [Install] sections — common to all unit types
man 5 systemd.service           # [Service] section
man 5 systemd.exec              # execution environment (User=, WorkingDirectory=, Environment=, etc.)
man 5 systemd.kill              # how processes are killed (KillMode=, KillSignal=, etc.)
man 5 systemd.resource-control  # resource limits (CPUQuota=, MemoryLimit=, etc.)
man 7 systemd.syntax            # the unit file syntax/format rules
man 1 systemctl                 # the CLI to manage units

Systemctl

Systemctl is the cli to interface with systemd.

Commands

Bash
sudo systemctl daemon-reload         # Systemd will recreate the dencancy tree and add/update services
sudo systemctl start custom.service  # Run the service
sudo systemctl status custom.service # Check the status of the service
sudo systemctl edit kubeview.service         # edit overrides (avoid changing the pure service)
sudo systemctl edit --full kubeview.service  # edit the service directly
sudo systemctl list-units --type=service --state=running # list all runninng
systemd-cgls # tree group by unit (need to looking to cgroups)

Journalctl

Journalctl is the cli to interface with the journald, which is the canonical logging system for systemd.

Commands

Shell
journalctl | vim -             # pipe into vim
journalctl --since=today   
journalctl --unit unit          
journalctl -f                  # follow live (like tail -f)
journalctl _EXE=/usr/bin/sudo  # specific binay
journalctl --list-boots        # to see all the boot logs

journalctl --priority err      # errors and above only
# priority levels:
# 0=emerg 1=alert 2=crit 3=err 4=warning 5=notice 6=info 7=debug

Example output

sudo systemctl status custom.service
○ custom.service - Test Service
     Loaded: loaded (/etc/systemd/system/custom.service; disabled; preset: enabled)
     Active: inactive (dead)

May 16 12:21:13 fatpc systemd[1]: Starting custom.service - Test Service...
May 16 12:21:13 fatpc echo[52743]: hello from custom.service
May 16 12:21:13 fatpc systemd[1]: custom.service: Deactivated successfully.
May 16 12:21:13 fatpc systemd[1]: Finished custom.service - Test Service.

Edit this page