Update: While this article focuses on bringing up an OpenBSD box with a configured pf(8) and sshd(8), the underlying code has been updated to support SSL/TLS via Let’s Encrypt, and thus also manage httpd(8) and relayd(8).

This is the main article of the series, explaining how to automatize a typical OpenBSD box setup via sugar(1), focusing on basic services (httpd(8), pf(8) and sshd(8)). By the end of this article, a complete, idempotent setup of the box can be performed by a single command (think, salt’s highstate):

(local)% sugar -s $name setup
hook-pre-setup-pf                       : Preparing pf.conf for sync...
hook-pre-setup-pf                       : Ensure we have a backup ssh link...
hook-pre-setup-sshd                     : Preparing sshd_config for sync...
hook-pre-setup-sshd                     : Ensure we have a backup ssh link...
setup-pf                                : Asserting pf is enabled...
setup-pf                                : Installing pf.conf...
setup-pf                                : Checking pf.conf syntax...
setup-pf                                : Loading pf with new pf.conf...
setup-sshd                              : Asserting sshd is enabled...
setup-sshd                              : Installing /etc/ssh/sshd_config...
setup-sshd                              : Checking sshd_config...
setup-sshd                              : Restarting sshd
hook-post-setup-pf                      : Ensuring port 22 is closed from outside...
hook-post-setup-pf                      : Ensuring ssh port is opened from outside...
hook-post-setup-pf                      : Close backup ssh link...
OK

Setting up

sugar(1) requires an existing machine with a password-less root ssh(1) connection. We’ll use here an OpenBSD running on an OVH VPS, installed by following this article.

Note: A local machine with a running sshd(8) could be managed as well as a remote one.

For clarity, we’ll use the following notations:

$ip remote machine IP;
$name remote machine “sugar” name, identifier;
$port remote machine non-standard SSH port;
$HOME/gits/sugar is where the sugar git repository is cloned;
$HOME/gits/sugard-openbsd is where the sugard-openbsd git repository is cloned;
raw/ or ready/ prefix are sometimes added to indicate if a file/directory is located in the server’s raw or ready directory. Recall that the raw/ directory is the one you versionned, while the ready/ directory is the one that get rsync(1)’d, and that is created by the pre-hooks.

A first thing we want to do is to change the default sshd(8) port to allow a non-standard port to reduce exposure to brute-forcing bots. We can leave 22 open for now; default pf(4) configuration should allow traffic on both ports, we’ll soon be managing sshd_config(5) via sugar anyway.

#!/bin/sh

set -e

port=35410
conf=/etc/ssh/sshd_config

if [ -n "$1" ]; then port="$1"; fi
if [ -n "$2" ]; then conf="$2"; fi

sed -i /^Port/d $conf

cat <<EOF| ed $conf > /dev/null 2>/dev/null
/^#Port 22
a
Port 22
Port $port
.
wq
EOF

rcctl restart sshd

Deploy and install manually; also install the default rsync(1) package:

(local)% scp switchport.sh root@$ip:/tmp/
(local)% ssh root@$ip "sh /tmp/switchport.sh $port"
sshd(ok)
sshd(ok)
(local)% ssh -p $port root@$ip echo OK
OK
(local)% ssh -p $port root@$ip pkg_add rsync--
quirks-4.54 signed on 2022-02-12T18:54:43Z
The following new rcscripts were installed: /etc/rc.d/rsyncd
See rcctl(8) for details.

Note: This is typically the kind of operations that we want and will automatize in a moment.

Wagashi, 和菓子, making tools by Lzen9216 through wikimedia.org – CC-BY-SA-4.0

We can now configure sugar(1) to use this machine with a $HOME/.sugarf, and ensure we can can ping it through sugar:

(local)% cat >> $HOME/.sugarf <<EOF
Host $name
    Hostname $ip
    User     root
    Port     $port
EOF
(local)% sugar $name uname -a
OpenBSD $hostname.openstacklocal 7.0 GENERIC.MP#232 amd64
OK

We can furthermore setup a versioned raw directory with a basic xping script:

# Create the raw directory under a place likely to host
# gits repositories
(local)% mkdir -p $HOME/gits/sugard-$name/bin
(local)% cat >    $HOME/gits/sugard-$name/bin/xping <<EOF
#!/bin/sh
echo pong
EOF
# For clarity: this is automatically performed anyway.
(local)% chmod +x $HOME/gits/sugard-$name/bin/xping

# Link it from $SUGARD/raw/
(local)% mkdir -p $HOME/.sugard/raw $HOME/.sugard/ready
(local)% ln -sv $HOME/gits/sugard-$name/ $HOME/.sugard/raw/$name
'$HOME/.sugard/raw/$name' -> '$HOME/gits/sugard-$name/'

# Synchronize the local sugard directory with the remote one (-s)
# so that xping will be "available" via sugar; execute it.
(local)% sugar -s $name xping
pong
OK

# That's our script.
(local)% sugar $name cat /var/sugard/bin/xping
#!/bin/sh
echo pong
OK

# It has been rsync(8)-ed from the ready/bin/ directory,
# itself created by being rsync(8)-ed from the raw/bin/
# directory.
(local)% cat $HOME/.sugard/ready/$name/bin/xping
#!/bin/sh
echo pong

sugar.lib

Out of habits, I’ll be using sh(1) scripts to configure the OS, but note that any language would do. Shell scripts are often considered brittle, but they are also very close to how systems are manually administered, thus providing little overhead. Obviously, for complex systems, one may want to use more sophisticated/solid languages.

sugar(1) comes with a sugar.lib, which provides basic functions to be used in sugar scripts, including:

a “configuration system” (loadconfs(), working from config.* and config sh(1) files);
a “logging system” (log(), logerr(), fail());
a “templating system” (samurai(), simple pattern substitution from values provided in configuration files);
OS-agnostic code wrapping system (loadoslib() will source sugar.lib.$OS; assuming all those files share the same interface, we have a basic mechanism to write OS-agnostic code);
silent command execution (quietrun(), which buffers output, displaying it only on error);
some tmux(1) based functions to setup/tear-down backup ssh(1) links (mksshlink(), rmsshlink(); useful for pf(4) setup for instance)
etc.

In order to use sugar.lib, we can for now copy it manually to our raw/bin: it will end up in the remote’s $SUGARD/bin which is automatically added to the $PATH when executing remote commands via sugar, so we can then load it with a . $(which sugar.lib) within shell scripts remotely executed with sugar.

Instead of manually copying it, we could also do so automatically in the main pre-hook raw/pre-hook, either directly or through sugar.imports. Such options will be presented later: we’ll slowly introduce the features.

Note: At first, we won’t be using the templating and the configuration system for clarity. We’ll introduce them with pf(4), and rework our sshd(8) setup accordingly.

Note: Don’t be fooled by the shiny list: those “features” are actually implemented rather sparingly, in around 100 LoC. Bear in mind that our goal is to reduce as much as possible the abstraction layer, just enough to help automate manual administration.

sshd(8)

As a rule, we’ll try to make sure that remotely executed code is as simple and bland as possible, like moving files around, and restarting services. Smarter things, such as preparing configuration files could be managed in pre-hooks: this is locally executed code, whose output can be checked before pushing it to production.

For now, we’ll satisfy ourselves with a static sshd_config(5) file, raw/$name/sshd_config, such as:

# Unconventional port
Port 35410

# Allow passwordless root login
PermitRootLogin without-password

# Public keys authentication only
PubkeyAuthentication   yes
PasswordAuthentication no
ChallengeResponseAuthentication no

# Public keys storage
AuthorizedKeysFile	.ssh/authorized_keys

# sFTP enabled
Subsystem	sftp	/usr/libexec/sftp-server

The files needs to be installed from the raw/ directory to the ready/ one by a pre-hook. We also want to keep an opened SSH link in case of issues; the link could be removed once we know that the server is still reacheable after the setup, in a post-hook.

Remember that hooks are executed with the following parameters:

$1: server “sugar” name (i.e. $name)
$2: path to raw/ directory
$3: path to ready/ directory

Pre-hook for raw/$name/bin/setup-sshd, $raw/$name/bin/hook-pre-setup-sshd:

#!/bin/sh

set -e

. `which sugar.lib`

log Preparing sshd_config for sync...
cp $2/sshd_config $3/sshd_config

log Ensure we have a backup ssh link...
mksshlink backup-ssh-link-sshd $1

Post-hook for bin/setup-sshd, bin/hook-post-setup-sshd:

#!/bin/sh

set -e

. `which sugar.lib`

log Ensure we can ssh to remote server...
ssh -p 35410 root@127.0.0.1 echo ping | grep -F ping > /dev/null

log Close backup ssh link...
rmsshlink backup-ssh-link-sshd

Note: See how redundant we are regarding sshd(8) port and the IP address. The configuration mechanism provided by sugar.lib through loadconfs() precisely solves this issue, and will be used shortly.

And finally, an install script raw/$name/bin/setup-sshd:

#!/bin/sh

set -e

. `which sugar.lib`

log Asserting sshd is enabled...
rcctl get sshd | grep -F sshd_flags | grep -vqF 'sshd_flags=NO'

log Installing /etc/ssh/sshd_config...
install -o root -g wheel -m 644 $SUGARD/sshd_config /etc/ssh/sshd_config

log Checking sshd_config...
quietrun sshd -t

log Restarting sshd
quietrun rcctl restart sshd

This can all be run with:

(local)% sugar -s $name setup-sshd
hook-pre-setup-sshd                     : Preparing sshd_config for sync...
hook-pre-setup-sshd                     : Ensure we have a backup ssh link...
setup-sshd                              : Asserting sshd is enabled...
setup-sshd                              : Installing /etc/ssh/sshd_config...
setup-sshd                              : Checking sshd_config...
setup-sshd                              : Restarting sshd
hook-post-setup-sshd                    : Ensure we can ssh to remote server...
hook-post-setup-sshd                    : Close backup ssh link...
OK

Daifuku, 大福餅, 2007 by yugoQ through flickr.com wikimedia.org – CC-BY-SA-2.0

pf(8)

We can setup pf(4) in a similar way:

a pre-hook to copy a pf.conf(5) from raw/ to ready/ and to open a backup ssh(1) link;
a post-hook to ensure the setup is correct enough from the outside, and to close the backup ssh(1) link;
a setup script that install the pf.conf(5), ensure it’s syntactically correct, and restart the service.

We’ll now move a step forward, and use the primitive “templating” mechanism and the “configuration” system provided by sugar.lib to implement it. We’ll revisit our sshd(8) setup in the next subsection, so as to use it there too.

Note: On OpenBSD, most configuration files allow to declare variables with a similar syntax (e.g. pf.conf(5), httpd.conf(5), relayd.conf(5)). If it wasn’t for sshd_config(5), we could simply concatenate one block of such variable definitions on top of all configurations instead of using a template system. More generally, there could be other, better ways to manage shared configuration items depending on what the OS provides.

Note: The template system is extremely primitive, e.g. compared to Jinja: it’s basically just a substitution mechanism. A sh(1) script in a pre-hook could provide more advanced features.

First, we need to declare a few common variables in raw/$name/config:

#!/bin/sh

# Main network interface
main_int="vio0"

# Main loopback IPv4 address
lo_ipv4="127.0.0.1"

# Main IPv4 address
main_ipv4="127.0.0.1"

# sshd(8) non-standard listening port
sshd_port="34510"

# httpd(8), relayd(8): HTTP/HTTPs listening ports.
http_port=80
https_port=443

# "Local" port, used for relayd(8) to communicate decrypted TLS
# traffic to httpd(8).
http_lport=8080

# httpd(8) chroot directory
httpd_chroot="/var/www"

# acme-client(1) directory, relative to httpd_chroot
acme_dir="acme"

# local directory containing git(1) repositories
gits_dir="$HOME/gits"

# gmail user
gmail_email="foo@gmail.com"

# gmail secret
gmail_secret=""

# ip2location token
ip2location_token=''

samurai(), the brave template system, will be applied to a “template” raw/$name/pf.conf.base to generate the final ready/$name/pf.conf:

#!/sbin/pfctl -f

# Main network interface
main_int  = "%%main_int%%"

# (My) IPv4 address
main_ipv4   = "%%main_ipv4%%"

# SSH/HTTP/HTTPs ports
sshd_port  = %%sshd_port%%
http_port  = %%http_port%%
https_port = %%https_port%%

# No filtering for loopback
set skip on lo

# Block all incoming traffic
block in

# (Explicitely) allow all outgoing traffic
pass out all

# Allow incoming ping
pass in on $main_int inet proto icmp all icmp-type 8 code 0

# Allow incoming SSH/HTTP/HTTPs
pass in on $main_int inet proto tcp from any to $main_ipv4 port $sshd_port
pass in on $main_int inet proto tcp from any to $main_ipv4 port $http_port
pass in on $main_int inet proto tcp from any to $main_ipv4 port $https_port

# Disgraceful commies
block in quick from 222.73.54.32
block in quick from 220.249.106.152
block in quick from 47.75.162.226
block in quick from 111.70.4.216

A raw/$name/bin/hook-pre-setup-pf, very similar to what we had for sshd(8), but this time, we’re using samurai() to generate the configuration file from the previous template instead of cp(1)-ying it:

#!/bin/sh

set -e

. `which sugar.lib`

log Preparing pf.conf for sync...
samurai $2 $2/pf.conf.base > $3/pf.conf

log Ensure we have a backup ssh link...
mksshlink backup-ssh-link-pf $1

Then, a raw/$name/bin/hook-post-setup-pf, again very similar to what we had for sshd(8):

#!/bin/sh

set -e

. `which sugar.lib`
loadconfs $2

# Timeout after which we consider remote port
# to be unreacheable.
w=2

log Ensuring port 22 is closed from outside...
if nc -vz -w $w $main_ipv4 22 >/dev/null 2>&1; then exit 1; fi

log Ensuring ssh port is opened from outside...
nc -vz $main_ipv4 $sshd_port >/dev/null 2>&1

log Close backup ssh link...
rmsshlink backup-ssh-link-pf

Note: The $main_ipv4 we use in the code here is from the config: those are the same that we’ve been using in the template.

And finally, the raw/$name/bin/setup-pf itself, which share the same structure as the sshd(8) one:

Assert that the service is enabled;
Install the configuration file;
Ensure the new configuration file is correctly formatted;
Relaunch the service to use the new configuration file.

#!/bin/sh

set -e

. `which sugar.lib`

log Asserting pf is enabled...
assertenabled pf

log Installing pf.conf...
install -o root -g wheel -m 600 $SUGARD/pf.conf /etc/pf.conf

log Checking pf.conf syntax...
pfctl -nf /etc/pf.conf

log Loading pf with new pf.conf...
pfctl -f  /etc/pf.conf

sshd(8), bis

We can now tweak our sshd(8) setup to share the $sshd_port variable defined in config with pf.conf(5): our sshd_config will now contain:

Port %%sshd_port%%

# Allow passwordless root login
PermitRootLogin without-password

# Public keys authentication only
PubkeyAuthentication   yes
PasswordAuthentication no
ChallengeResponseAuthentication no

# Public keys storage
AuthorizedKeysFile	.ssh/authorized_keys

# Let sFTP enabled
Subsystem	sftp	/usr/libexec/sftp-server

Which will require this raw/$name/bin/hook-pre-setup-sshd:

#!/bin/sh

set -e

. `which sugar.lib

log Preparing sshd_config for sync...
samurai $2 $2/sshd_config > $3/sshd_config

log Ensure we have a backup ssh link...
mksshlink backup-ssh-link-sshd $1

raw/$name/bin/setup-sshd is unchanged, but we need to slightly tweak raw/$name/bin/hook-post-setup-sshd so as to avoid hardcoding the sshd(8) port and the server’s main IP address:

#!/bin/sh

set -e

. `which sugar.lib`
loadconfs $2

log Ensure we can ssh to remote server...
ssh -p $sshd_port root@$main_ipv4 echo ping | grep -F ping > /dev/null

log Close backup ssh link...
rmsshlink backup-ssh-link-sshd

Wagashi 和菓子 and Matcha 抹茶 by Chris Gladis through flickr.com wikimedia.org – CC-BY-SA-3.0

setup.auto

The *.auto files located at the root of the raw/ directories allow to create “virtual” commands, linearly executing other more primitive command, taking hooks into account. This is the most basic form of dependencies management.

For instance, a setup.auto containing:

setup-pf
xping
setup-sshd

Will generate a ready/$name/bin/hook-pre-setup:

#!/bin/sh

set -e

if echo "$DEBUG" | grep -q x; then set -x; fi

hook-pre-setup-pf $@
hook-pre-setup-sshd $@

A ready/$name/bin/setup:

#!/bin/sh

set -e

if echo "$DEBUG" | grep -q x; then set -x; fi

setup-pf $@
xping $@
setup-sshd $@

And a ready/$name/bin/hook-post-setup:

#!/bin/sh

set -e

if echo "$DEBUG" | grep -q x; then set -x; fi

hook-post-setup-pf $@
hook-post-setup-sshd $@

Note: xping was included to explicitly show that non-existing per-command hooks aren’t called in generated pre/post hooks.

The .auto files management needs to be enable in the main pre-hook, raw/$name/pre-hook; we can also now automatically copy sugar.lib from there too, that we’ll soon be importing with sugar.imports:

#!/bin/sh

set -e

cp `which sugar.lib` $3/bin
sugar.mkautos $@

That all can be run with a:

(local)% sugar -s $name setup
hook-pre-setup-pf                       : Preparing pf.conf for sync...
hook-pre-setup-pf                       : Ensure we have a backup ssh link...
hook-pre-setup-sshd                     : Preparing sshd_config for sync...
hook-pre-setup-sshd                     : Ensure we have a backup ssh link...
setup-pf                                : Asserting pf is enabled...
setup-pf                                : Installing pf.conf...
setup-pf                                : Checking pf.conf syntax...
setup-pf                                : Loading pf with new pf.conf...
setup-sshd                              : Asserting sshd is enabled...
setup-sshd                              : Installing /etc/ssh/sshd_config...
setup-sshd                              : Checking sshd_config...
setup-sshd                              : Restarting sshd
hook-post-setup-pf                      : Ensuring port 22 is closed from outside...
hook-post-setup-pf                      : Ensuring ssh port is opened from outside...
hook-post-setup-pf                      : Close backup ssh link...
OK

In the series:

On sugar, a sweeter Salt: one OpenBSD box (3/5)

Comments

By email, at mathieu.bivert chez:

On Sugar, a Sweeter Salt: One OpenBSD Box

Table Of Contents

In the series:

Setting up

sugar.lib

sshd(8)

pf(8)

sshd(8), bis

setup.auto

In the series:

Comments