Introduction

Services are sh(1) scripts, installed in the /etc/rc.d/ directory. Enabled services are executable (chmod +x /etc/rc.d/$service); disabled services aren’t.

rc.subr(8) clarifies what such a service is supposed to be doing:

Every script under /etc/rc.d follows this pattern:

1. Define the daemon variable.
2. Define service-specific defaults for one or more daemon_* variables (optional).
3. Source rc.subr, which defines default shell functions and variable values.
4. Override the pexp variable or any of the rc_* functions and set the rc_bg or rc_reload variables, if needed.
5. Define an rc_pre and/or rc_post function, if needed.
6. Call the rc_cmd function as “rc_cmd $1”.

As an example, here is the current /etc/rc.d/httpd:

#!/bin/ksh
#
# $OpenBSD: httpd,v 1.9 2022/08/29 19:14:25 ajacoutot Exp $

daemon="/usr/sbin/httpd"

. /etc/rc.d/rc.subr

rc_configtest() {
	# use rc_exec here since daemon_flags may contain arguments with spaces
	rc_exec "${daemon} -n ${daemon_flags}"
}

rc_cmd $1

Now, here’s the fun part:

The code is relatively straightforward, it’ll help fix some wrong assumptions you may have¹, and you’ll be better equipped for debugging. The documentation is usually thorough, but still requires to be read cautiously.

Debugging

rcctl(8) comes with a -d option, which is actually only clearly documented in rc.d(8):

-d Setting this option will print the function names as they are called and prevent the rc.subr(8) framework from redirecting stdout and stderr to /dev/null. This is used to allow debugging of failed actions.

If this isn’t sufficient, well, you know that rcctl(8) and rc.subr(8) are sh(1) scripts, so hack around. Beware of altering the exit status of some functions/commands…

Process identification

I’ve mentioned in a previous footnote that the process identification isn’t as “convenient” as a pid file associated to the process. Instead, it relies on a regexp, build from $daemon and $daemon_flags, called $pexp.

Now, in most cases, this should work out of the box, but there are cases – we’ll see one in the next section – where we have to either manually tweak $pexp, or some of the code relying on it (e.g. rc_check(), rc_stop()).

If you have to tweak $pexp, you might encounter the following issue:

Once you’ve started a service, a file is created in /var/run/rc.d/$service, e.g.

daemon_class=daemon
daemon_execdir=
daemon_flags=-c /zm/config.json -d /zm/ -u /run/zm/zm-backend.sock -f -o
daemon_logger=daemon.info
daemon_rtable=0
daemon_timeout=30
daemon_user=root
pexp=/zm-backend -c /zm/config.json -d /zm/ -u /run/zm/zm-backend.sock -f -o
rc_reload=
rc_reload_signal=HUP
rc_stop_signal=TERM
rc_usercheck=

Now the trick is that this file will be loaded (by _rc_parse_conf()), in particular for its $pexp, once the service has been loaded. So for example if you:

1. Start a service;
2. Update its service file, in particular its pexp;
3. Try to stop the service.

Then the $pexp used will be the one registered when starting the service, that is, the one from /var/run/rc.d/$service and not the one from /etc/rc.d/$service. I don’t believe this is documented in the man pages, so this might save you a bit of debugging.

Example: a chroot(1)-ed Go webserver

Assume you have an elementarily structured Go HTTP server, something like:

package main

import (
	"log"
	"net/http"
)

func main() {
	port := ":8090"
	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		w.Write([]byte("hello, world!"))
	})
	log.Println("Listening on "+port)
	log.Fatal(http.ListenAndServe(port, nil))
}

Now, you might be tempted to daemonize it. But Go programs can be difficult to daemonize, generally speaking: the language doesn’t come equipped with enough tooling to handle daemonization, because it’s tricky to guarantee than no goroutines will be executed before we try to daemonize; if I understand correctly, they could be started in the init() of some of the modules you might import, and wouldn’t be inherited by the child (standard daemonization procedure requires to fork(2) so as to re-attach to init(8), but this is beyond the scope of this article, and well-documented elsewhere since forever).

Of course you might still attempt to make your Go daemon well-behaved, but let’s assume you do not, and instead, decide to rely on rc.d(8), as is recommended.

Then, because that daemon will run in the foreground by default, you’ll want to specify

rc_bg="YES"

in your /etc/rc.d/$service. Furthermore if you wish to log its standard output, you can rely on the default logging mechanism, enabled via:

daemon_logger="daemon.info"

The logs would then appear, flagged by your $service name, in /var/log/daemon:

[...]

2024-07-29T17:58:26.006Z main zm[50631]: 2024/07/29 17:58:26 Caught signal terminated: shutting down.
2024-07-29T17:58:32.459Z main zm[66771]: 2024/07/29 17:58:32 Listening on unix:/run/zm/zm-backend.sock (fastcgi: true)
2024-07-29T17:58:46.899Z main zm[66771]: 2024/07/29 17:58:46 Caught signal terminated: shutting down.
2024-07-29T17:58:46.900Z main zm[66771]: 2024/07/29 17:58:46 accept unix /run/zm/zm-backend.sock: use of closed network connection
2024-07-29T17:58:55.906Z main zm[3994]: 2024/07/29 17:58:55 Listening on unix:/run/zm/zm-backend.sock (fastcgi: true)

Note: Maybe it’d be wiser for an HTTP daemon to log things in the typical www chroot; i.e. in /var/www/logs/, next to httpd(8)’s logs. I haven’t bothered fine tuning.

Alright, now we also want to chroot(8) our binary “from the outside”; all in all, based on what we said earlier in this article, we should be able to cook a service file close to:

#!/bin/ksh
#
# /etc/rc.d/zm

# Just using variables for clarity here: those aren't
# special variables like $daemon or $daemon_flags.
chroot_dir="/var/www/"
chroot_user="www"
chroot_group="daemon"
chroot_cmd="chroot -u $chroot_user -g $chroot_group $chroot_dir"

zm_binary="/zm-backend"
daemon="$chroot_cmd $zm_binary"
daemon_flags="-c /zm/config.json -d /zm/ -u /run/zm/zm-backend.sock -f -o"

daemon_logger="daemon.info"

. /etc/rc.d/rc.subr

rc_bg="YES"

rc_cmd $1

But it’ll fail. Here’s the catch: chroot(8) will run its supplied command via execve(2) (or associated): hence the process name will change to whatever command is being run via chroot(8) (see usr.sbin/chroot/chroot.c). But by default, rc.d(8) & cie will compute $pexp, the regular expression to identify our process, in particular from our ${daemon} variable. Note furthermore that the match must be exact (pgrep -x, see pgrep(1)).

What we would want instead is to identify the process without the chroot part of the command. We could tweak both rc_check() and rc_stop(), or simply provide the correct $pexp:

#!/bin/ksh
#
# /etc/rc.d/zm

chroot_dir="/var/www/"
chroot_user="www"
chroot_group="daemon"
chroot_cmd="chroot -u $chroot_user -g $chroot_group $chroot_dir"

zm_binary="/zm-backend"
daemon="$chroot_cmd $zm_binary"

daemon_flags="-c /zm/config.json -d /zm/ -u /run/zm/zm-backend.sock -f -o"
daemon_logger="daemon.info"

. /etc/rc.d/rc.subr

pexp="${zm_binary} ${daemon_flags}"

rc_bg="YES"

rc_cmd $1

And there it is: a Go HTTP daemon, chrooted, lauched with special user/group, managed by OpenBSD’s rc.d(8), with proper logging. And without instrumenting the Go code.

main# rcctl check zm
zm(ok)
main# rcctl stop zm
zm(ok)
main# rcctl start zm
zm(ok)
main# rcctl check zm
zm(ok)

Comments

By email, at mathieu.bivert chez: