Service managers

Michal Sekletár 

Lukáš Nykrýn

<msekleta@redhat.com>

<lnykryn@redhat.com>

 Praha 7.4.2017

Service managers

"Something that sets up your system during boot and takes care of it during its life-time."

UNIX systems - INIT

• process running as a pid 1

• starts other processes during boot

• parent of orphaned processes

  • needs to handle SIGCHLDS

• default handler of Ctrl+Alt+Delete

  • SIGINT

• can't be SIGKILLed

• Linux kernel will panic if init dies

SYSVINIT

• ~1980
• AT&T - System III
• http://minnie.tuhs.org/cgi-bin/utree.pl?file=SysIII/usr/src/cmd/init.c

• 1991
• Miquel van Smoorenburg
• https://savannah.nongnu.org/projects/sysvinit

• based around the concept of runlevels
• simple configuration
• respawn

SYSVINIT

RUNLEVELS

• key concept of sysvinit
• actions are performed during transition between states

sysvinit

/etc/inittab

• configuration for sysvinit
• four value separated by ":"
  • id
  • runlevel 
  • action
  • process

sysvinit

process tracking

• CHILD structure for every line in inittab
• monitors SIGCHLDS 

INITscripts

• handling of service directly with sysvinit would  not be practical

  • minimal set of options

  • can't manually start single service

  • ...

• standardized scripts wrapping deamon and service

  • described in LSB

  • but in reality they differs between distributions

• written in shell

• /lib/lsb/init-functions - "library" of shared functions

• strictly synchronous execution

INITscriptS

• shell scripts in /etc/rc.d/init.d
• start, stop, restart, force-reload, status
• enable initscript in runlevel X
  • symlink /etc/rcX.d/SNNservice -> /etc/init.d/service
• disable initscript in runlevel X
  • symlink /etc/rcX.d/KNNservice -> /etc/init.d/service
• order determined by NN number - pre-calculated
  • either directly specified in comment in initscript (chkconfig header)
  • or calculated based on dependencies in the LSB header
• /etc/rc X

for i in /etc/rcX.d/K* ; do $i stop  ; done
for i in /etc/rcX.d/S* ; do $i start ; done

INITscriptS

#!/bin/sh -e
### BEGIN INIT INFO
# Provides:          apache2
# Required-Start:    $local_fs $remote_fs $network $syslog
# Required-Stop:     $local_fs $remote_fs $network $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# X-Interactive:     true
# Short-Description: Start/stop apache2 web server
### END INIT INFO
#
# apache2		This init.d script is used to start apache2.
#			It basically just calls apache2ctl.

ENV="env -i LANG=C PATH=/usr/local/bin:/usr/bin:/bin"

#[ $(ls -1 /etc/apache2/sites-enabled/ | wc -l | sed -e 's/ *//;') -eq 0 ] && \
#echo "You haven't enabled any sites yet, so I'm not starting apache2." && \
#echo "To add and enable a host, use addhost and enhost." && exit 0

#edit /etc/default/apache2 to change this.
HTCACHECLEAN_RUN=auto
HTCACHECLEAN_MODE=daemon
HTCACHECLEAN_SIZE=300M
HTCACHECLEAN_DAEMON_INTERVAL=120
HTCACHECLEAN_PATH=/var/cache/apache2/mod_disk_cache
HTCACHECLEAN_OPTIONS=""

set -e
if [ -x /usr/sbin/apache2 ] ; then
	HAVE_APACHE2=1
else
	echo "No apache MPM package installed"
	exit 0
fi

. /lib/lsb/init-functions

test -f /etc/default/rcS && . /etc/default/rcS
test -f /etc/default/apache2 && . /etc/default/apache2

APACHE2CTL="$ENV /usr/sbin/apache2ctl"
HTCACHECLEAN="$ENV /usr/sbin/htcacheclean"

PIDFILE=$(. /etc/apache2/envvars && echo $APACHE_PID_FILE)
if [ -z "$PIDFILE" ] ; then
	echo ERROR: APACHE_PID_FILE needs to be defined in /etc/apache2/envvars >&2
	exit 2
fi


check_htcacheclean() {
	[ "$HTCACHECLEAN_MODE" = "daemon" ] || return 1

	[ "$HTCACHECLEAN_RUN"  = "yes"    ] && return 0

	[ "$HTCACHECLEAN_RUN"  = "auto" \
	  -a -e /etc/apache2/mods-enabled/disk_cache.load ] && return 0
	
	return 1
}

start_htcacheclean() {
	$HTCACHECLEAN $HTCACHECLEAN_OPTIONS -d$HTCACHECLEAN_DAEMON_INTERVAL \
			-i -p$HTCACHECLEAN_PATH -l$HTCACHECLEAN_SIZE
				
}

stop_htcacheclean() {
	pkill htcacheclean 2> /dev/null || echo ...not running
}

pidof_apache() {
	# if there is actually an apache2 process whose pid is in PIDFILE,
	# print it and return 0.
	if [ -e "$PIDFILE" ]; then
		if pidof apache2 | tr ' ' '\n' | grep $(cat $PIDFILE); then
			return 0
		fi
	fi
	return 1
}

apache_stop() {
	if $APACHE2CTL configtest > /dev/null 2>&1; then
		# if the config is ok than we just stop normaly
                $APACHE2CTL stop 2>&1 | grep -v 'not running' >&2 || true
	else
		# if we are here something is broken and we need to try
		# to exit as nice and clean as possible
		PID=$(pidof_apache) || true

		if [ "${PID}" ]; then
			# in this case it is everything nice and dandy
			# and we kill apache2
			log_warning_msg "We failed to correctly shutdown apache, so we're now killing all running apache processes. This is almost certainly suboptimal, so please make sure your system is working as you'd expect now!"
                        kill $PID
		elif [ "$(pidof apache2)" ]; then
			if [ "$VERBOSE" != no ]; then
                                echo " ... failed!"
			        echo "You may still have some apache2 processes running.  There are"
 			        echo "processes named 'apache2' which do not match your pid file,"
			        echo "and in the name of safety, we've left them alone.  Please review"
			        echo "the situation by hand."
                        fi
                        return 1
		fi
	fi
}

apache_wait_stop() {
	# running ?
	PIDTMP=$(pidof_apache) || true
	if kill -0 "${PIDTMP:-}" 2> /dev/null; then
	    PID=$PIDTMP
	fi

	apache_stop

	# wait until really stopped
	if [ -n "${PID:-}" ]; then
		i=0
		while kill -0 "${PID:-}" 2> /dev/null;  do
        		if [ $i = '60' ]; then
        			break;
        	 	else
        			if [ $i = '0' ]; then
                			echo -n " ... waiting "
        			else
                	      		echo -n "."
        		 	fi
        			i=$(($i+1))
        			sleep 1
        	      fi
		 done
	fi
}

case $1 in
	start)
		log_daemon_msg "Starting web server" "apache2"
		if $APACHE2CTL start; then
			if check_htcacheclean ; then
				log_progress_msg htcacheclean
				start_htcacheclean || log_end_msg 1
			fi
                        log_end_msg 0
                else
                        log_end_msg 1
                fi
	;;
	stop)
		if check_htcacheclean ; then
			log_daemon_msg "Stopping web server" "htcacheclean"
			stop_htcacheclean
			log_progress_msg "apache2"
		else
			log_daemon_msg "Stopping web server" "apache2"
		fi
		if apache_wait_stop; then
                        log_end_msg 0
                else
                        log_end_msg 1
                fi
	;;
	graceful | reload | force-reload)
		if ! $APACHE2CTL configtest > /dev/null 2>&1; then
                    $APACHE2CTL configtest || true
                    log_end_msg 1
                    exit 1
                fi
                log_daemon_msg "Reloading web server config" "apache2"
		if pidof_apache > /dev/null ; then
                    if $APACHE2CTL graceful $2 ; then
                        log_end_msg 0
                    else
                        log_end_msg 1
                    fi
                fi
	;;
	restart)
		if check_htcacheclean ; then
			log_daemon_msg "Restarting web server" "htcacheclean"
			stop_htcacheclean
			log_progress_msg apache2
		else
			log_daemon_msg "Restarting web server" "apache2"
		fi
		PID=$(pidof_apache) || true
		if ! apache_wait_stop; then
                        log_end_msg 1 || true
                fi
		if $APACHE2CTL start; then
			if check_htcacheclean ; then
				start_htcacheclean || log_end_msg 1
			fi
                        log_end_msg 0
                else
                        log_end_msg 1
                fi
	;;
	start-htcacheclean)
		log_daemon_msg "Starting htcacheclean"
		start_htcacheclean || log_end_msg 1
		log_end_msg 0
	;;
	stop-htcacheclean)
		log_daemon_msg "Stopping htcacheclean"
			stop_htcacheclean
			log_end_msg 0
	;;
	status)
		PID=$(pidof_apache) || true
		if [ -n "$PID" ]; then
			echo "Apache is running (pid $PID)."
			exit 0
		else
			echo "Apache is NOT running."
			exit 1
		fi
	;;
	*)
		log_success_msg "Usage: /etc/init.d/apache2 {start|stop|restart|reload|force-reload|start-htcacheclean|stop-htcacheclean|status}"
		exit 1
	;;
esac

INITscriptS

status function

• daemon during start writes its PID to pidfile
• initscripts status reads the pidfile and check if there is process with such pid
• PID recyclation problem
  • daemon dies, but due to limit amount of PIDs there might be a new process with same PID
  • ​​additional check (name of the process, path to binary,..)
  • not completely reliable

SYSVinit + INITscriptS

advantages & disadvantages

• simple design
• old 
  • limited amount of new bugs
  • users are used to the old bugs
• synchronous
• shell (engineer's duck-tape)
• piles of shell code across the packages
• old design
  • does not use directly new kernel features
• modular but without defined API

System initialization on FreeBSD

• operating systems from BSD family don’t use System V init
• instead, PID 1 is “BSD style init”
• doesn’t read /etc/inittab
• there is no concept of run levels
• in general there are two runtime modes
  • single-user mode
  • multi-user mode
• after basic system initialization PID1 runs /etc/rc script
• two configuration files
  • /etc/rc.conf
  • /etc/defaults/rc.conf
• initscripts are stored in two separate directories
  • /etc/rc.d (for base system services)
  • /usr/local/etc/rc.d (for user installed software)
• it is not possible to attach debugger to PID 1

Deficiencies of

System V and BSD

• service management is fundamental task for all system administrators
• problems
  • “What services are running?”
  • “what services are failing?”
  • “What services are available on the system?”
• traditional UNIX “service management model” is simple but insufficient
  • processes + init-scripts
• having service management as an integral part operating system also brings significant benefits
  • better discoverability
  • standardized error reporting
  • better overall understanding of the system
  • creating the common interface for all user software
  • standardized tools

OPENRC

• Roy Marples
• 5 April 2007
• not an init (PID 1)
• replacement for old initscripts
• mainly used in Gentoo
• portable to non-Linux systems
• cleaner syntax
• parallel boot (off by default)
• partly implemented in C

OPENRC - initscript

#!/sbin/openrc-run
# Copyright 1999-2016 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

extra_commands="configtest modules virtualhosts"
extra_started_commands="configdump fullstatus graceful gracefulstop reload"

description_configdump="Dumps the configuration of the runing apache server. Requires server-info to be enabled and www-client/lynx."
description_configtest="Run syntax tests for configuration files."
description_fullstatus="Gives the full status of the server. Requires lynx and server-status to be enabled."
description_graceful="A graceful restart advises the children to exit after the current request and reloads the configuration."
description_gracefulstop="A graceful stop advises the children to exit after the current request and stops the server."
description_modules="Dump a list of loaded Static and Shared Modules."
description_reload="Kills all children and reloads the configuration."
description_virtualhosts="Show the settings as parsed from the config file (currently only shows the virtualhost settings)."
description_stop="Kills all children and stops the server."

# Apply default values for some conf.d variables.
PIDFILE="${PIDFILE:-/var/run/apache2.pid}"
TIMEOUT=${TIMEOUT:-15}
SERVERROOT="${SERVERROOT:-/usr/lib64/apache2}"
CONFIGFILE="${CONFIGFILE:-/etc/apache2/httpd.conf}"
LYNX="${LYNX:-lynx -dump}"
STATUSURL="${STATUSURL:-http://localhost/server-status}"
RELOAD_TYPE="${RELOAD_TYPE:-graceful}"

# Append the server root and configuration file parameters to the
# user's APACHE2_OPTS.
APACHE2_OPTS="${APACHE2_OPTS} -d ${SERVERROOT}"
APACHE2_OPTS="${APACHE2_OPTS} -f ${CONFIGFILE}"

# The path to the apache2 binary.
APACHE2="/usr/sbin/apache2"

depend() {
        need net
        use mysql dns logger netmount postgresql
        after sshd
}

configtest() {
        ebegin "Checking ${SVCNAME} configuration"
        checkconfig
        eend $?
}

checkconfd() {
        if [ ! -d ${SERVERROOT} ]; then
                eerror "SERVERROOT does not exist: ${SERVERROOT}"
                return 1
        fi
}

checkconfig() {
        checkpath --directory /run/apache_ssl_mutex
        checkconfd || return 1

        OUTPUT=$( ${APACHE2} ${APACHE2_OPTS} -t 2>&1 )
        ret=$?
        if [ $ret -ne 0 ]; then
                eerror "${SVCNAME} has detected an error in your setup:"
                printf "%s\n" "${OUTPUT}"
        fi

        return $ret
}

start() {
        checkconfig || return 1

        if [ -n "${STARTUPERRORLOG}" ] ; then
                # We must make sure that we only append to APACHE2_OPTS
                # in start() and not in stop() or anywhere else that may
                # be executed along with start(), see bug #566726.
                APACHE2_OPTS="${APACHE2_OPTS} -E ${STARTUPERRORLOG}"
        fi

        ebegin "Starting ${SVCNAME}"
        # Use start stop daemon to apply system limits #347301 
        start-stop-daemon --start -- ${APACHE2} ${APACHE2_OPTS} -k start

        local i=0 retval=1
        while [ $i -lt ${TIMEOUT} ] ; do
                if [ -e "${PIDFILE}" ] ; then
                        retval=0
                        break
                fi
                sleep 1 && i=$(expr $i + 1)
        done

        eend ${retval}
}

stop() {
        if [ "${RC_CMD}" = "restart" ]; then
                checkconfig || return 1
        fi

        PID=$(cat "${PIDFILE}" 2>/dev/null)
        if [ -z "${PID}" ]; then
                einfo "${SVCNAME} not running (no pid file)"
                return 0
        fi

        ebegin "Stopping ${SVCNAME}"
        ${APACHE2} ${APACHE2_OPTS} -k stop

        local i=0 retval=0
        while ( test -f "${PIDFILE}" || pgrep -P ${PID} apache2 >/dev/null ) \
                && [ $i -lt ${TIMEOUT} ]; do
                sleep 1 && i=$(expr $i + 1)
        done
        [ -e "${PIDFILE}" ] && retval=1

        eend ${retval}
}

reload() {
        checkconfig || return 1

        if [ "${RELOAD_TYPE}" = "restart" ]; then
                ebegin "Restarting ${SVCNAME}"
                ${APACHE2} ${APACHE2_OPTS} -k restart
                eend $?
        elif [ "${RELOAD_TYPE}" = "graceful" ]; then
                ebegin "Gracefully restarting ${SVCNAME}"
                ${APACHE2} ${APACHE2_OPTS} -k graceful
                eend $?
        else
                eerror "${RELOAD_TYPE} is not a valid RELOAD_TYPE. Please edit /etc/conf.d/${SVCNAME}"
        fi
}

graceful() {
        checkconfig || return 1
        ebegin "Gracefully restarting ${SVCNAME}"
        ${APACHE2} ${APACHE2_OPTS} -k graceful
        eend $?
}

gracefulstop() {
        checkconfig || return 1
        ebegin "Gracefully stopping ${SVCNAME}"
        ${APACHE2} ${APACHE2_OPTS} -k graceful-stop
        eend $?
}

modules() {
        checkconfig || return 1
        ${APACHE2} ${APACHE2_OPTS} -M 2>&1
}

fullstatus() {
        if ! type -p $(set -- ${LYNX}; echo $1) 2>&1 >/dev/null; then
                eerror "lynx not found! you need to emerge www-client/lynx"
        else
                ${LYNX} ${STATUSURL}
        fi
}

virtualhosts() {
        checkconfig || return 1
        ${APACHE2} ${APACHE2_OPTS} -S
}

configdump() {
        INFOURL="${INFOURL:-http://localhost/server-info}"

        checkconfd || return 1

        if ! type -p $(set -- ${LYNX}; echo $1) 2>&1 >/dev/null; then
                eerror "lynx not found! you need to emerge www-client/lynx"
        else
                echo "${APACHE2} started with '${APACHE2_OPTS}'"
                for i in config server list; do
                        ${LYNX} "${INFOURL}/?${i}" | sed '/Apache Server Information/d;/^[[:space:]]\+[_]\+$/Q'
                done
        fi
}

# vim: ts=4 filetype=gentoo-init-d

OPENRC

process tracking

• pidfiles - same as in old initscripts
  • plus name of the binary
• WIP cgroup tracking
• tries to keep state of a service
• separate "respawn" daemon
  • directly starts the service
  • service can't fork

OPENRC

dependencies

• declared by depend function
• dependencies
  • need
  • use
• ordering dependencies
  • after
  • before
• dependency tree is build after changing the configuration

OPENRC

Advantages & disadvantages

• "modern initscripts"
• richer dependencies
• no additional daemon
• still piles of shell code

UPSTART

• Scott James Remnant
• August 24 2006
• Linux-only
• "event-based execution"
• parallel start of services
• declarative configuration
  • "job files"
  • /etc/init/*.conf
• using ptrace to watch processes
• libnih library

UPSTART

configuratioN 

description "apache2 http server"
start on runlevel [2345]
stop on runlevel [!2345]
pre-start script
mkdir -p /var/run/apache2 || true
install -d -o www-data /var/lock/apache2 || true
# ssl_scache shouldn't be here if we're just starting up.
# (this is bad if there are several apache2 instances running)
rm -f /var/run/apache2/*ssl_scache* || true
end script
limit cpu 300 300
env APACHE_RUN_USER=www-data
env APACHE_RUN_GROUP=www-data
env APACHE_PID_FILE=/var/run/apache2.pid
# Give up if restart occurs 10 times in 30 seconds.
respawn limit 10 30
exec /usr/sbin/apache2 -D NO_DETACH
respawn

UPSTART

Dependencies

• does not have traditional dependencies
• actions are started based on events -> no need for building dependency graphs
• "start apache when network is on" or "start network when apache is starting"
• socket,file...-activation through bridges

UPSTART

process tracking

• actively tracks forks of the started process using ptrace
• user needs to specify if the service performs double-fork or not
• waits for SICHLD from the main process 

UPSTART

advantages & DISADVANTAGES

• simple declarative configuration
• simple way to react to events in system
• ptrace tracking is not that reliable
• complicated way to write dependencies
• problematic CLA 

systemd

• 2010
• Lennart Poettering
• https://www.freedesktop.org/wiki/Software/systemd/
• https://github.com/systemd/systemd

• not just init or simple service manager
• set of daemons and tool
• goal: create a unified base for Linux user-space 

SYSTEMD - COMPONENTS

• SYSTEMD

  • init

  • systemctl

• SYSTEMD-UDEVD

  • device manager

• SYSTEMD-LOGIND

  • track user session
  • limits for users
  • shutdown/sleep inhibition logic

  • loginctl

  • systemd-inhibit

• SYSTEMD-JOURNALD

  • log agregator

  • journalctl

• SYSTEMD-MACHINED

  • tracks VMs and containers

  • work with images

  • machinectl

• SYSTEMD-NSPAWN

  • tool for creating namespace containers

SYSTEMD - COMPONENTS

• SYSTEMD-HOSTNAMED

  • maintains system's hostname

  • hostnamectl

• SYSTEMD-LOCALED

  • locale settings

  • keymaps

  • localectl

• SYSTEMD-TIMEDATED

  • timezone
  • set-time
  • ntp on/off

  • timedatectl

• SYSTEMD-TIMESYNCD

  • SNTP

• SYSTEMD-IMPORTD

  • downloads containers images

  • machinectl pull-*

• SYSTEMD-TMPFILES

  • creates volatiles files and directories

• SYSTEMD-SYSCTL

  • configure kernel parameters at boot

SYSTEMD - COMPONENTS

• SYSTEMD-NETWORKD

  • sets up network

  • networkctl

• SYSTEMD-RESOLVED

  • provides network name resolution

  • DNSSEC

  • systemd-resolve

• SYSTEMD-FIRSTBOOT

  • initialize system on first boot

SYSTEMD-SYSUSERS

• allocate system users and groups

SYSTEMD - PID1

• systemd is a service manager
• dependency based execution engine
• configurable via simple .ini like files
• abstract object representing system resource is unit
  • service, socket, mount,  timer, path, device, ...
  • socket, path, time - based activation
• process tracking capabilities (cgroups)
• rich dependencies
• parallel system start-up
• DBus APIs
• compatible with initscripts
  • generator that automatically encapsulate initscript in a unit

SYSTEMD - configuration

$ cat /usr/lib/systemd/system/cups.service

[Unit]
Description=CUPS Printing Service

[Service]
ExecStart=/usr/sbin/cupsd -f
PrivateTmp=true

[Install]
Also=cups.socket cups.path

SYSTEMD - dependecies

• Relational dependencies
  • Wants
  • Requires
  • BindsTo
  • PartOf
  • Requisite
  • Conflicts
• Ordering dependencies
  • After
  • Before

SYSTEMD - Transaction

1. create job for the specified unit (anchor)
2. add recursively jobs for all dependencies
3. minimalize the transaction in a loop
   1. get rid of NOP jobs
   2. get rid of jobs not referenced by anchor
4. check for ordering loops in the graph in a loop
   1. break the loop by deleting a job
   2. get rid of jobs not referenced by anchor
5. merge mergable jobs
6. get rid of jobs not referenced by anchor
7. merge jobs with simmilar one already in job queue
8. add the jobs to job queue

Intermezzo - cgroups

• kernel subsystem used for process aggregation and resource limiting
• in kernel since 2008
• tool to group processes in a tree-like structures
• if process do a fork(), child will be placed in the same cgroup
• accessible via cgroup filesystems (cgroup, cgroup2)
• cgroups v1
  • process can be placed to any node in the tree
  • you can have multiple orthogonal trees
  • resource limiting by attaching controllers to the cgroup tree
    • cpuset, cpu, cpuacct, blkio, memory, devices, freezer, net cls, net prio, pids
  • release_agent
    • path to binary that is executed when a cgroup is empty
• cgoups v2
  • only one hierarchy
  • controllers can be attached to subtrees
    • controllers are mostly WIP
  • cgroup.events
    • file that says if there are any processes in that subtree
• fun fact: you can use both simultaneously (although you can't use one controller in both)

SYSTEMD - process tracking

• every service runs in its own cgroup
  • so we can track every child process of a deamon
  • only leaf cgroups are used (compatibility v1, v2)
  • non-leaf nodes of the tree are represented by slice units
    • used for resource management
• PID are still used for tracking
  • determining the main PID of a deamon
• systemd can be notified that service is down
  • normally through SIGCHLD from main process
  • through cgroup notifications
• systemd can kill all children of a service

systemd

advantages & DISADVANTAGES

• simple declarative configuration
• exhaustive API
• rich dependencies
• complex
• non-synchronouse

Service Management Facility (SMF)

• system service modeled as high-level object
• SMF is a collection of services
• PID 1 on Solaris is actually still System V compatible daemon
  • understand concepts of runlevels
  • uses /etc/inittab as a main configuration file
• two main components
  • svc.startd
  • svc.configd
• each service has well-defined state and set of relations to other system services
• service states
  • uninitialized
  • online
  • offline
  • degraded
  • maintenance
  • disabled
  • legacy_run

SMF capabilities

• observe and manage system services.
• help with identification of failing or misbehaving system services (svcs -x)
• delegation of tasks to non-root users
• persistent service configuration across reboots and system upgrades
• compatibility with legacy init scripts

SMF - Basic Concepts

• different types of services wrt. number of processes and process lifetime
  • long running
    • single process (e.g. cron)
    • complex multi-process (e.g. in memory database of ERP system)
  • short lived
    • configure and exit type of services (e.g. sysctl)
    • dhcp client
    • random shell scripts
• in order to monitor services SMF introduces new kernel API called “process contracts”

SMF - Process contract

• technology used for process tracking purposes
• implemented in Solaris kernel
• userspace interface via API filesystem mounted by default under /system/contract
• process contract allows process to register an interest in a process and all of its children when important changes occur, such as
  • all processes in contract exit
  • hardware error occurs in any of the processes
  • processes in the contract coredump
  • any process in the contract receives a fatal signal

SMF - Process Contract Example

SMF - Command Line Tools

SMF - CONFIGURATION

• SMF permanent and runtime configuration are both stored in a central repository
• repository content is initially populated from XML based manifests
• repository is internally the SQL lite database and it is managed by svc.config daemon and svccfg command line tool
• services names are in form of Fault Management Resource Identifier (FMRI)
  • example: svc:/network/ssh:default
• service can have more than one instance
• when service has only single instance then we can use abbreviations (just service name) instead of full FMRI in various commands
• XML manifests describe services

SMF - Configuration

<?xml version='1.0'?>
<!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
<service_bundle type='manifest' name='export'>
  <service name='system/sysding' type='service' version='0'>
    <dependency name='fs' grouping='require_all' restart_on='none' type='service'>
      <service_fmri value='svc:/system/filesystem/minimal:default'/>
    </dependency>
    <dependency name='identity' grouping='require_all' restart_on='none' type='service'>
      <service_fmri value='svc:/system/identity:node'/>
      <service_fmri value='svc:/system/identity:domain'/>
    </dependency>
    <property_group name='startd' type='framework'>
      <propval name='duration' type='astring' value='transient'/>
    </property_group>
    <instance name='system' enabled='true'>
      <dependency name='single-user' grouping='require_all' restart_on='none' type='service'>
        <service_fmri value='svc:/milestone/single-user:default'/>
      </dependency>
      <dependency name='filesystem_local' grouping='require_all' restart_on='none' type='service'>
        <service_fmri value='svc:/system/filesystem/local:default'/>
      </dependency>
      <dependency name='rpcbind' grouping='optional_all' restart_on='none' type='service'>
        <service_fmri value='svc:/network/rpc/bind:default'/>
      </dependency>
      <dependent name='sysidtoolsystem_sysconfig' restart_on='none' grouping='optional_all'>
        <service_fmri value='svc:/milestone/sysconfig'/>
      </dependent>
      <exec_method name='start' type='method' exec='/lib/svc/method/sysding' timeout_seconds='0'/>
      <exec_method name='stop' type='method' exec=':true' timeout_seconds='0'/>
      <property_group name='config' type='application'>
        <propval name='finished' type='boolean' value='false'/>
      </property_group>
    </instance>
    <stability value='Unstable'/>
    <template>
      <common_name>
        <loctext xml:lang='C'>sysding</loctext>
      </common_name>
      <documentation>
        <manpage title='sysding' section='1M' manpath='/usr/man'/>
      </documentation>
    </template>
  </service>
</service_bundle>

SMF - Examples

• list running services
  • svcs
• list all services
  • svcs -a
• display service details
  • svcs -l cron
• show services failed services
  • svcs -x
• start a service on boot and make sure it is running
  • svcadm enable cron

• restart service
  • svcadm restart cron
• apply configuration changes
  • svcadm refresh cron
• stop a service permanently
  • svcadm disable dbus
• put a service into maintenance/degraded state
  • svcadm mark maintenance dbus
  • svcadm mark degraded dbus
• clear maintenance state flag
  • svcadm clear dbus

Service management on Windows

Services

• service is a console application that runs on the background possibly w/o user being logged on the system
• services are managed by Service Control Manager (SCM)
• two types of “services”
  • service applications - conforms to rules imposed by SCM
  • driver services - they don’t interact w/ SCM
• in general we recognize 3 types of programs wrt. service management
  • service programs (e.g. svchost.exe)
  • service configuration programs (e.g. application installer)
  • service control programs (e.g. sc utility)

Service Control Manager

• started by wininit.exe
• provides RPC based interface for service management (RPC layer abstracted away by WINAPI)
• maintains database of currently installed services (both normal and driver services)
• it can start services
  • during system boot
  • on demand
  • on triggers (power outage)
• makes sure that services are started in correct order
• each service must be registered with Service Control Manager (SCM)
• services are created using CreateService API

Service Control Manager

CreateService API

SC_HANDLE WINAPI CreateService(
  _In_      SC_HANDLE hSCManager,
  _In_      LPCTSTR   lpServiceName,
  _In_opt_  LPCTSTR   lpDisplayName,
  _In_      DWORD     dwDesiredAccess,
  _In_      DWORD     dwServiceType,
  _In_      DWORD     dwStartType,
  _In_      DWORD     dwErrorControl,
  _In_opt_  LPCTSTR   lpBinaryPathName,
  _In_opt_  LPCTSTR   lpLoadOrderGroup,
  _Out_opt_ LPDWORD   lpdwTagId,
  _In_opt_  LPCTSTR   lpDependencies,
  _In_opt_  LPCTSTR   lpServiceStartName,
  _In_opt_  LPCTSTR   lpPassword
);

SERVICE CONTROL MANAGER

CREATESERVICE API - important arguments

• lpServiceName, name of a service
• dwServiceType, type of a service
  • SERVICE_FILE_SYSTEM_DRIVER
  • SERVICE_KERNEL_DRIVER (in user-space driver)
  • SERVICE_WIN32_OWN_PROCESS
  • SERVICE_WIN32_SHARE_PROCESS
• dwStartType
  • SERVICE_AUTO_START
  • SERVICE_DEMAND_START
• lpBinaryPathName
• lpDependencies
• lpServiceStartName (name the account under which a service should run)
  • NULL -> LocalSystem
  • NT AUTHORITY\LocalService -> LocalService
  • NT AUTHORITY\NetworkService -> NetworkService

Service Programs

• implements one or more services
• services of type SERVICE_WIN32_OWN_PROCESS contain code for only single service
• on the other hand services type SERVICE_WIN32_SHARE_PROCESS contain code for more than one service
• shared service model is very common on Windows (prime example is svchost.exe)
  • single process implements numerous system services
  • usually there are multiple instances and each one is running different set of system services
  • service specific functionality is implemented via shared library (DLL)
  • code for particular service runs in a separate thread
• main thread of a service process communicates with SCM via named pipe

useful links

• https://savannah.nongnu.org/projects/sysvinit
• http://refspecs.linuxbase.org/LSB_3.0.0/LSB-PDA/LSB-PDA/iniscrptact.html
• https://github.com/OpenRC/openrc
• http://upstart.ubuntu.com/cookbook/
• https://www.freedesktop.org/wiki/Software/systemd/
  • The systemd for Administrators Blog Series
  • Documentation for Developers
• https://www.kernel.org/doc/Documentation/cgroup-v1/
• https://www.kernel.org/doc/Documentation/cgroup-v2.txt
• https://msdn.microsoft.com/en-us/library/windows/desktop/ms681921(v=vs.85).aspx
• https://www.usenix.org/legacy/event/lisa05/tech/full_papers/adams/adams.pdf
• http://www.freebsd.cz/doc/handbook/system-administration.html