Process Control

Lifecycle of a Process

Normal lifecycle

parent forks - child runs - child exits - child becomes zombie
                                        - parent calls wait()
                                        - zombie cleaned up ✓

When parent dies first

parent dies - kernel scans all processes
            - finds children whose parent PID is now dead
            - reparents them to PID 1 (init/systemd)
            - children keep running normally
            - when they eventually die, systemd calls wait()
            - cleaned up
# note: a child can opt in to dying with its parent

Starting background processes from the Shell

As show above starting a process in the shell is terminated when it’s parent dies.

&

When not told otherwise commands take the foreground. You can only have one “foreground” process running in a single shell session.

  • & tells shells to run the command in a background process under the parent shell.
  • By default & gives the child process the same fds (fd 0, 1, 2 -> parent terminal). So stdout and stderr will send it to the terminal.
  • When the terminal closes, the terminal sends HUP to the foreground process group. The shell may then forward this to background jobs in its job table as it exits
$ ./program &
LOG [INFO] program is runnning    # stdout
LOG [ERROR] program had an error  # stderr
$                                 
# you will still be able to control the shell but the program will continue to use the fds of the main process

# This redirects stdout to null and stderr to stdout (null)
$ ./program > /dev/null 2>&1 &
# But still this isn't safe from a HUP signal from the parent shell

# You can use plain ps to see what child processes you are running 
$ ps

nohup

The main failing of & is the vulnerability to a HUP signal. The nohup command adds a signal handlers before running your program to tell the kernel to ignore the HUP signal.
Your program doesn’t know anything about it and nohup doesn’t know anything about your program

nohup ./program &
# The same as above but safeguards it from the HUP signal

# A full combination to run in the background safely and quietly
nohup ./program > /dev/null 2>&1 &

# Or log to a file
nohup ./program >> /var/log/myprogram.log 2>&1 &
# Or log to syslog
nohup ./program 2>&1 | logger -t myprogram &

The process is still a child of your terminal, until the terminal closes which the process ignores and the os reparents to PID 1 Systemd.

Stack Overflow


/proc

  • The /proc directory, a pseudo-filesystem in which the kernel exposes a variety of interesting information about the system’s state.
  • Despite the name /proc (and the name of the underlying filesystem type, “proc”), the information is not limited to process information a variety of status information and statistics generated by the kernel are represented here
FileContents
cgroupThe control groups to which the process belongs
cmdCommand or program the process is executing
cmdlineComplete command line of the process (null-separated)
cwdSymbolic link to the process’s current directory
environThe process’s environment variables (null-separated)
exeSymbolic link to the file being executed
fdSubdirectory containing links for each open file descriptor
fdinfoSubdirectory containing further info for each open file descriptor
mapsMemory mapping information (shared segments, libraries, etc.)
nsSubdirectory with links to each namespace used by the process
rootSymbolic link to the process’s root directory (set with chroot)
statGeneral process status information (best decoded with ps)
statmMemory usage information

Inspecting a /proc process

Let’s check the directories - /environ /fd from above

# environ, reading env vars of a process
cat environ | tr "\000" "\n"
IRIS_APIKEY=DAS3eML1bOqjsrfYWseXNEyEqgAnK3g31hq9zJ93TOx6jX48
EXPORTER_PORT=10043

# fds, are connecting to pipes and null input
ls -l fd
lrwx------    1 alpine   alpine          64 May 24 18:36 0 -> /dev/null
l-wx------    1 alpine   alpine          64 May 24 18:36 1 -> pipe:[269563]
l-wx------    1 alpine   alpine          64 May 24 18:36 2 -> pipe:[269564]

strace - system call trace

snoop on processes’ syscalls

Commands

strace -p 8948 # attach to a running process
strace -e trace=file vim # start a process and trace, filter for file syscalls only

# Super useful to see where it looks for config files
strace -e trace=file -o vim_trace.txt vim # send to a file
# ./vim_trace.txt
stat("/home/alpine/.vimrc", 0x7ffd5cbbc7c0) = -1 ENOENT (No such file or directory)
open("/home/alpine/.vimrc", O_RDONLY|O_NONBLOCK|O_LARGEFILE) = -1 ENOENT (No such file or directory)
stat("/home/alpine/.vim/vimrc", 0x7ffd5cbbc7c0) = -1 ENOENT (No such file or directory)
open("/home/alpine/.vim/vimrc", O_RDONLY|O_NONBLOCK|O_LARGEFILE) = -1 ENOENT (No such file or directory)
stat("/home/alpine/.config/vim/vimrc", 0x7ffd5cbbc7c0) = -1 ENOENT (No such file or directory)
open("/home/alpine/.config/vim/vimrc", O_RDONLY|O_NONBLOCK|O_LARGEFILE) = -1 ENOENT (No such file or directory)

ps - process status

ps has historical baggage and trees positional arguments as flags, but really they’re all just flags. aux is the same as -a -u -x.

# grep ps, hide grep
ps aux | grep -v grep | grep top

ps -aux # every process different format
    a - all users
    u - user oriented format 
    x - include processes with no controlling term (daemons)