Update: While this article focuses on bringing up an OpenBSD box with a configured pf(8) and sshd(8), the underlying code has been updated to support SSL/TLS via Let’s Encrypt, and thus also manage httpd(8) and relayd(8).
This is the main article of the series, explaining how to automatize a typical OpenBSD box setup via sugar(1), focusing on basic services (httpd(8), pf(8) and sshd(8)). By the end of this article, a complete, idempotent setup of the box can be performed by a single command (think, salt’s highstate):
(local)% sugar -s $name setup
hook-pre-setup-pf : Preparing pf.conf for sync...
hook-pre-setup-pf : Ensure we have a backup ssh link...
hook-pre-setup-sshd : Preparing sshd_config for sync...
hook-pre-setup-sshd : Ensure we have a backup ssh link...
setup-pf : Asserting pf is enabled...
setup-pf : Installing pf.conf...
setup-pf : Checking pf.conf syntax...
setup-pf : Loading pf with new pf.conf...
setup-sshd : Asserting sshd is enabled...
setup-sshd : Installing /etc/ssh/sshd_config...
setup-sshd : Checking sshd_config...
setup-sshd : Restarting sshd
hook-post-setup-pf : Ensuring port 22 is closed from outside...
hook-post-setup-pf : Ensuring ssh port is opened from outside...
hook-post-setup-pf : Close backup ssh link...
OK
Setting up
sugar(1) requires an existing machine with a
password-less root
ssh(1) connection. We’ll use
here an OpenBSD running on an OVH VPS, installed
by following this article.
Note: A local machine with a running sshd(8) could be managed as well as a remote one.
For clarity, we’ll use the following notations:
$ip
remote machine IP;$name
remote machine “sugar” name, identifier;$port
remote machine non-standard SSH port;$HOME/gits/sugar
is where the sugar git repository is cloned;$HOME/gits/sugard-openbsd
is where the sugard-openbsd git repository is cloned;raw/
orready/
prefix are sometimes added to indicate if a file/directory is located in the server’s raw or ready directory. Recall that theraw/
directory is the one you versionned, while theready/
directory is the one that get rsync(1)’d, and that is created by the pre-hooks.
A first thing we want to do is to change the default sshd(8)
port to allow a non-standard port to reduce exposure to brute-forcing
bots. We can leave 22
open for now; default pf(4) configuration
should allow traffic on both ports, we’ll soon be managing
sshd_config(5) via sugar anyway.
#!/bin/sh
set -e
port=35410
conf=/etc/ssh/sshd_config
if [ -n "$1" ]; then port="$1"; fi
if [ -n "$2" ]; then conf="$2"; fi
sed -i /^Port/d $conf
cat <<EOF| ed $conf > /dev/null 2>/dev/null
/^#Port 22
a
Port 22
Port $port
.
wq
EOF
rcctl restart sshd
Deploy and install manually;Β also install the default rsync(1) package:
(local)% scp switchport.sh root@$ip:/tmp/
(local)% ssh root@$ip "sh /tmp/switchport.sh $port"
sshd(ok)
sshd(ok)
(local)% ssh -p $port root@$ip echo OK
OK
(local)% ssh -p $port root@$ip pkg_add rsync--
quirks-4.54 signed on 2022-02-12T18:54:43Z
The following new rcscripts were installed: /etc/rc.d/rsyncd
See rcctl(8) for details.
Note: This is typically the kind of operations that we want and will automatize in a moment.
We can now configure sugar(1) to use
this machine with a $HOME/.sugarf
, and ensure we can
can ping it through sugar
:
(local)% cat >> $HOME/.sugarf <<EOF
Host $name
Hostname $ip
User root
Port $port
EOF
(local)% sugar $name uname -a
OpenBSD $hostname.openstacklocal 7.0 GENERIC.MP#232 amd64
OK
We can furthermore setup a versioned raw
directory with a
basic xping
script:
# Create the raw directory under a place likely to host
# gits repositories
(local)% mkdir -p $HOME/gits/sugard-$name/bin
(local)% cat > $HOME/gits/sugard-$name/bin/xping <<EOF
#!/bin/sh
echo pong
EOF
# For clarity: this is automatically performed anyway.
(local)% chmod +x $HOME/gits/sugard-$name/bin/xping
# Link it from $SUGARD/raw/
(local)% mkdir -p $HOME/.sugard/raw $HOME/.sugard/ready
(local)% ln -sv $HOME/gits/sugard-$name/ $HOME/.sugard/raw/$name
'$HOME/.sugard/raw/$name' -> '$HOME/gits/sugard-$name/'
# Synchronize the local sugard directory with the remote one (-s)
# so that xping will be "available" via sugar; execute it.
(local)% sugar -s $name xping
pong
OK
# That's our script.
(local)% sugar $name cat /var/sugard/bin/xping
#!/bin/sh
echo pong
OK
# It has been rsync(8)-ed from the ready/bin/ directory,
# itself created by being rsync(8)-ed from the raw/bin/
# directory.
(local)% cat $HOME/.sugard/ready/$name/bin/xping
#!/bin/sh
echo pong
sugar.lib
Out of habits, I’ll be using sh(1) scripts to configure the OS, but note that any language would do. Shell scripts are often considered brittle, but they are also very close to how systems are manually administered, thus providing little overhead. Obviously, for complex systems, one may want to use more sophisticated/solid languages.
sugar(1) comes with a sugar.lib
, which provides basic
functions to be used in sugar scripts, including:
- a “configuration system” (
loadconfs()
, working fromconfig.*
andconfig
sh(1) files); - a “logging system” (
log()
,logerr()
,fail()
); - a “templating system” (
samurai()
, simple pattern substitution from values provided in configuration files); - OS-agnostic code wrapping system (
loadoslib()
will sourcesugar.lib.$OS
; assuming all those files share the same interface, we have a basic mechanism to write OS-agnostic code); - silent command execution (
quietrun()
, which buffers output, displaying it only on error); - some tmux(1) based functions to setup/tear-down backup
ssh(1) links (
mksshlink()
,rmsshlink()
; useful for pf(4) setup for instance) - etc.
In order to use sugar.lib
, we can for now copy it manually to our raw/bin
:
it will end up in the remote’s $SUGARD/bin
which is automatically added to
the $PATH
when executing remote commands via sugar
, so we can then load it
with a . $(which sugar.lib)
within shell scripts remotely executed with sugar
.
Instead of manually copying it, we could also do so automatically in the main pre-hook
raw/pre-hook
, either directly or through sugar.imports
. Such options will
be presented later: we’ll slowly introduce the features.
Note: At first, we won’t be using the templating and the configuration system for clarity. We’ll introduce them with pf(4), and rework our sshd(8) setup accordingly.
Note: Don’t be fooled by the shiny list: those “features” are actually implemented rather sparingly, in around 100 LoC. Bear in mind that our goal is to reduce as much as possible the abstraction layer, just enough to help automate manual administration.
sshd(8)
As a rule, we’ll try to make sure that remotely executed code is as simple and bland as possible, like moving files around, and restarting services. Smarter things, such as preparing configuration files could be managed in pre-hooks: this is locally executed code, whose output can be checked before pushing it to production.
For now, we’ll satisfy ourselves with a static sshd_config(5)
file, raw/$name/sshd_config
, such as:
# Unconventional port
Port 35410
# Allow passwordless root login
PermitRootLogin without-password
#Β Public keys authentication only
PubkeyAuthentication yes
PasswordAuthentication no
ChallengeResponseAuthentication no
# Public keys storage
AuthorizedKeysFile .ssh/authorized_keys
# sFTP enabled
Subsystem sftp /usr/libexec/sftp-server
The files needs to be installed from the raw/
directory to
the ready/
one by a pre-hook. We also want to keep an opened
SSH link in case of issues; the link could be removed once
we know that the server is still reacheable after the setup,
in a post-hook.
Remember that hooks are executed with the following parameters:
$1
: server “sugar” name (i.e.$name
)$2
: path toraw/
directory$3
: path toready/
directory
Pre-hook for raw/$name/bin/setup-sshd
, $raw/$name/bin/hook-pre-setup-sshd
:
#!/bin/sh
set -e
. `which sugar.lib`
log Preparing sshd_config for sync...
cp $2/sshd_config $3/sshd_config
log Ensure we have a backup ssh link...
mksshlink backup-ssh-link-sshd $1
Post-hook for bin/setup-sshd
, bin/hook-post-setup-sshd
:
#!/bin/sh
set -e
. `which sugar.lib`
log Ensure we can ssh to remote server...
ssh -p 35410 root@127.0.0.1 echo ping | grep -F ping > /dev/null
log Close backup ssh link...
rmsshlink backup-ssh-link-sshd
Note: See how redundant we are regarding sshd(8)
port and the IP address. The configuration mechanism provided by
sugar.lib
through loadconfs()
precisely solves this issue, and
will be used shortly.
And finally, an install script raw/$name/bin/setup-sshd
:
#!/bin/sh
set -e
. `which sugar.lib`
log Asserting sshd is enabled...
rcctl get sshd | grep -F sshd_flags | grep -vqF 'sshd_flags=NO'
log Installing /etc/ssh/sshd_config...
install -o root -g wheel -m 644 $SUGARD/sshd_config /etc/ssh/sshd_config
log Checking sshd_config...
quietrun sshd -t
log Restarting sshd
quietrun rcctl restart sshd
This can all be run with:
(local)% sugar -s $name setup-sshd
hook-pre-setup-sshd : Preparing sshd_config for sync...
hook-pre-setup-sshd : Ensure we have a backup ssh link...
setup-sshd : Asserting sshd is enabled...
setup-sshd : Installing /etc/ssh/sshd_config...
setup-sshd : Checking sshd_config...
setup-sshd : Restarting sshd
hook-post-setup-sshd : Ensure we can ssh to remote server...
hook-post-setup-sshd : Close backup ssh link...
OK
pf(8)
We can setup pf(4) in a similar way:
- a pre-hook to copy a pf.conf(5) from
raw/
toready/
and to open a backup ssh(1) link; - a post-hook to ensure the setup is correct enough from the outside, and to close the backup ssh(1) link;
- a setup script that install the pf.conf(5), ensure it’s syntactically correct, and restart the service.
We’ll now move a step forward, and use the primitive “templating” mechanism
and the “configuration” system provided by sugar.lib
to implement it.
We’ll revisit our sshd(8) setup in the next subsection, so as
to use it there too.
Note: On OpenBSD, most configuration files allow to declare variables with a similar syntax (e.g. pf.conf(5), httpd.conf(5), relayd.conf(5)). If it wasn’t for sshd_config(5), we could simply concatenate one block of such variable definitions on top of all configurations instead of using a template system. More generally, there could be other, better ways to manage shared configuration items depending on what the OS provides.
Note: The template system is extremely primitive, e.g. compared to Jinja: it’s basically just a substitution mechanism. A sh(1) script in a pre-hook could provide more advanced features.
First, we need to declare a few common variables in raw/$name/config
:
#!/bin/sh
# Main network interface
main_int="vio0"
# Main loopback IPv4 address
lo_ipv4="127.0.0.1"
# Main IPv4 address
main_ipv4="127.0.0.1"
# sshd(8) non-standard listening port
sshd_port="34510"
# httpd(8), relayd(8): HTTP/HTTPs listening ports.
http_port=80
https_port=443
# "Local" port, used for relayd(8) to communicate decrypted TLS
# traffic to httpd(8).
http_lport=8080
# httpd(8) chroot directory
httpd_chroot="/var/www"
# acme-client(1) directory, relative to httpd_chroot
acme_dir="acme"
# local directory containing git(1) repositories
gits_dir="$HOME/gits"
# gmail user
gmail_email="foo@gmail.com"
# gmail secret
gmail_secret=""
# ip2location token
ip2location_token=''
samurai()
, the brave template system, will be applied to a “template”
raw/$name/pf.conf.base
to generate the final ready/$name/pf.conf
:
#!/sbin/pfctl -f
# Main network interface
main_int = "%%main_int%%"
# (My) IPv4 address
main_ipv4 = "%%main_ipv4%%"
# SSH/HTTP/HTTPs ports
sshd_port = %%sshd_port%%
http_port = %%http_port%%
https_port = %%https_port%%
# No filtering for loopback
set skip on lo
# Block all incoming traffic
block in
# (Explicitely) allow all outgoing traffic
pass out all
# Allow incoming ping
pass in on $main_int inet proto icmp all icmp-type 8 code 0
# Allow incoming SSH/HTTP/HTTPs
pass in on $main_int inet proto tcp from any to $main_ipv4 port $sshd_port
pass in on $main_int inet proto tcp from any to $main_ipv4 port $http_port
pass in on $main_int inet proto tcp from any to $main_ipv4 port $https_port
# Disgraceful commies
block in quick from 222.73.54.32
block in quick from 220.249.106.152
block in quick from 47.75.162.226
block in quick from 111.70.4.216
A raw/$name/bin/hook-pre-setup-pf
, very similar to what we had for
sshd(8), but this time, we’re using samurai()
to generate the configuration file from the previous template instead
of cp(1)-ying it:
#!/bin/sh
set -e
. `which sugar.lib`
log Preparing pf.conf for sync...
samurai $2 $2/pf.conf.base > $3/pf.conf
log Ensure we have a backup ssh link...
mksshlink backup-ssh-link-pf $1
Then, a raw/$name/bin/hook-post-setup-pf
, again very similar to what we
had for sshd(8):
#!/bin/sh
set -e
. `which sugar.lib`
loadconfs $2
# Timeout after which we consider remote port
# to be unreacheable.
w=2
log Ensuring port 22 is closed from outside...
if nc -vz -w $w $main_ipv4 22 >/dev/null 2>&1; then exit 1; fi
log Ensuring ssh port is opened from outside...
nc -vz $main_ipv4 $sshd_port >/dev/null 2>&1
log Close backup ssh link...
rmsshlink backup-ssh-link-pf
Note: The $main_ipv4
we use in the code here
is from the config
: those are the same that we’ve been using in
the template.
And finally, the raw/$name/bin/setup-pf
itself, which share the same structure
as the sshd(8) one:
- Assert that the service is enabled;
- Install the configuration file;
- Ensure the new configuration file is correctly formatted;
- Relaunch the service to use the new configuration file.
#!/bin/sh
set -e
. `which sugar.lib`
log Asserting pf is enabled...
assertenabled pf
log Installing pf.conf...
install -o root -g wheel -m 600 $SUGARD/pf.conf /etc/pf.conf
log Checking pf.conf syntax...
pfctl -nf /etc/pf.conf
log Loading pf with new pf.conf...
pfctl -f /etc/pf.conf
sshd(8), bis
We can now tweak our sshd(8) setup to share the
$sshd_port
variable defined in config
with
pf.conf(5): our sshd_config
will now contain:
Port %%sshd_port%%
# Allow passwordless root login
PermitRootLogin without-password
#Β Public keys authentication only
PubkeyAuthentication yes
PasswordAuthentication no
ChallengeResponseAuthentication no
# Public keys storage
AuthorizedKeysFile .ssh/authorized_keys
# Let sFTP enabled
Subsystem sftp /usr/libexec/sftp-server
Which will require this raw/$name/bin/hook-pre-setup-sshd
:
#!/bin/sh
set -e
. `which sugar.lib
log Preparing sshd_config for sync...
samurai $2 $2/sshd_config > $3/sshd_config
log Ensure we have a backup ssh link...
mksshlink backup-ssh-link-sshd $1
raw/$name/bin/setup-sshd
is unchanged, but we need to slightly tweak
raw/$name/bin/hook-post-setup-sshd
so as to avoid hardcoding the
sshd(8) port and the server’s main IP address:
#!/bin/sh
set -e
. `which sugar.lib`
loadconfs $2
log Ensure we can ssh to remote server...
ssh -p $sshd_port root@$main_ipv4 echo ping | grep -F ping > /dev/null
log Close backup ssh link...
rmsshlink backup-ssh-link-sshd
setup.auto
The *.auto
files located at the root of the raw/
directories
allow to create “virtual” commands, linearly executing other more primitive
command, taking hooks into account. This is the most basic form of
dependencies management.
For instance, a setup.auto
containing:
setup-pf
xping
setup-sshd
Will generate a ready/$name/bin/hook-pre-setup
:
#!/bin/sh
set -e
if echo "$DEBUG" | grep -q x; then set -x; fi
hook-pre-setup-pf $@
hook-pre-setup-sshd $@
A ready/$name/bin/setup
:
#!/bin/sh
set -e
if echo "$DEBUG" | grep -q x; then set -x; fi
setup-pf $@
xping $@
setup-sshd $@
And a ready/$name/bin/hook-post-setup
:
#!/bin/sh
set -e
if echo "$DEBUG" | grep -q x; then set -x; fi
hook-post-setup-pf $@
hook-post-setup-sshd $@
Note: xping
was included to explicitly show that non-existing
per-command hooks aren’t called in generated pre/post hooks.
The .auto
files management needs to be enable in the main pre-hook,
raw/$name/pre-hook
; we can also now automatically copy sugar.lib
from
there too, that we’ll soon be importing with sugar.imports
:
#!/bin/sh
set -e
cp `which sugar.lib` $3/bin
sugar.mkautos $@
That all can be run with a:
(local)% sugar -s $name setup
hook-pre-setup-pf : Preparing pf.conf for sync...
hook-pre-setup-pf : Ensure we have a backup ssh link...
hook-pre-setup-sshd : Preparing sshd_config for sync...
hook-pre-setup-sshd : Ensure we have a backup ssh link...
setup-pf : Asserting pf is enabled...
setup-pf : Installing pf.conf...
setup-pf : Checking pf.conf syntax...
setup-pf : Loading pf with new pf.conf...
setup-sshd : Asserting sshd is enabled...
setup-sshd : Installing /etc/ssh/sshd_config...
setup-sshd : Checking sshd_config...
setup-sshd : Restarting sshd
hook-post-setup-pf : Ensuring port 22 is closed from outside...
hook-post-setup-pf : Ensuring ssh port is opened from outside...
hook-post-setup-pf : Close backup ssh link...
OK
Comments
By email, at mathieu.bivert chez: