Migrating my servers to OpenBSD – Update No. 2

This post was published on 17 Mar 2025

The OpenBSD adventures continue! And because I believe I will be playing around with OpenBSD quite a bit in the future, I have made a special category on my blog’s overview page under a new heading titled “Series”. But that’s not what I wanted to talk about today; instead, I have further improved upon my relayd and httpd configs and have made it so that there is now a special ACME server that listens on port 9999 that handles all the ACME stuff. In my very first blog post about migrating my servers to OpenBSD, I mentioned how I had wanted to do this from the start, but the routing just didn’t seem to want to work how I wanted it to (well, to be exact, it didn’t work at all). Namely the match request path "/.well-known/acme-challenge/**" forward to <acme> statement in my <http_acme> protocol did not seem to actually get triggered.

Well, I have finally found a solution to this problem. Firstly, I found out that relayd has a special keyword you can use, namely quick in combination with the match request or pass request statements in your protocol blocks. A quick look at the man page reveals its function:

quick

If a connection is matched by a rule with the quick option set, the rule is considered to be the last matching rule and any further evaluation is skipped.

My understanding of this is that if I were to rewrite my match statement to match request path quick ..., as soon as a request matches it, all other statements are ignored so that if something requests the /.well-known/acme-challenge/ path, the traffic is forced into being forwarded to whatever table I define at the end of the match statement; and that does indeed appear to be what happens! Despite that, however, using the match statement did not work still and I had to change it to pass request quick path "/.well-known*" forward to <acme>. Now, finally, all traffic that goes to the /.well-known path gets sent to a special ACME server!

For this, I had to change the relayd.conf accordingly and go into the httpd.conf and configure a new server there. I have configured a completely new server titled default that listens on port 9999. In the relayd.conf, I had to add a new forward to <acme> port 9999 check tcp into each one of my relay blocks; and last but not least, I made it so that regular HTTP traffic is no longer routed only to localhost, but to the <webservers> table I defined. Additionally, I moved birds.bateleur.org over to OpenBSD as well and as that site is dependend on Ruby, I made a new table simply titled <ruby> with a list of servers that has Ruby installed and has my Ruby site running on it. Oh and I saved the IPs of my backends in their own variables, namely www1 and www2. The configs now look as follows:

relayd.conf

# Last edit: 2025-03-16

# ===== PORTS ===== #
# Webservers -> 10000
# Ruby Server -> 4567
# ================= #
#
# ===== SERVERS ===== #
# localhost -> current machine (obvs?)
# 10.5.0.11 -> OpenBSD VM via WG
# ================= #


# ===== LOGGING ===== #
#log state changes
#log connection

# ===== EXTERNAL IPS ===== #
ipv4="203.0.113.52"
ipv6="2001:db8:0:e291::1000:1"

# ===== WEBSERVERS ===== #
www1="localhost"
www2="10.5.0.11"

# ===== SERVER LISTS ===== #
# List of httpd servers that serve the websites. They have to be configured the same way and should all host the same websites with up-to-date files. They should all listen to :10000.
table <webservers> { $www1 $www2 }

# Because the encryption is handled only by relayd (so far, at least), we must make sure that ACME requests are handled by *only* one server
# I have designated this server to `localhost` because it just makes most sense, I would think.
table <acme> { $www1 }

# Other webservers or services
table <ruby> { $www2 }

http protocol "http_acme" {
	# Pass well-known challenges to ACME server
	# Using the "quick" keyword, once this is evaluated, everything else gets ignored
	# Thus, if the path matches ".well-known", all other "pass request" statements are ignored
	pass request quick path "/.well-known*" forward to <acme>

	# Block all requests that aren't excplicitly allowed below
	# This is mostly so that going to the server's IP won't show anything
	block request header "Host" value "*"

	# Specifically allow the following domains to pass and route them to their specified servers
	# == DOMAIN1.TLD == #
	pass request header "Host" value "domain1.tld" forward to <webservers>
	pass request header "Host" value "www.domain1.tld" forward to <webservers>

	# == DOMAIN2.TLD == #
	pass request header "Host" value "domain2.tld" forward to <webservers>
	pass request header "Host" value "www.domain2.tld" forward to <webservers>

	# == BIRDS.DOMAIN1.TLD == #
	pass request header "Host" value "birds.domain1.tld" forward to <ruby>
}

http protocol "https" {
	# Some recommended TCP options for SSL that I found
	tcp { nodelay, sack, socket buffer 65536, backlog 100 }

	# Pass well-known challenges to ACME server
	# Using the "quick" keyword, once this is evaluated, everything else gets ignored
	# Thus, if the path matches ".well-known", all other "pass request" statements are ignored
	pass request quick path "/.well-known*" forward to <acme>

	# Block all requests that aren't excplicitly allowed below
	# This is mostly so that going to the server's IP won't show anything
	block request header "Host" value "*"

	# Load TLS keypairs
	# Keypairs have to be in the following format: /etc/ssl/hostname:port.crt & /etc/ssl/private/hostname:port.key
	tls keypair "domain1.tld"
	tls keypair "domain2.tld"

        # Specifically allow the following domains to pass and route them to their specified servers
        # == DOMAIN1.TLD == #
	pass request header "Host" value "domain1.tld" forward to <webservers>
	pass request header "Host" value "www.domain1.tld" forward to <webservers>

	# == DOMAIN2.TLD == #
	pass request header "Host" value "domain2.tld" forward to <webservers>
	pass request header "Host" value "www.domain2.tld" forward to <webservers>

	# == BIRDS.DOMAIN1.TLD == #
	pass request header "Host" value "birds.domain1.tld" forward to <ruby>
}

# ===== RELAYS ===== #
#
# === HTTP RELAY ===#
# This relay is for sending regular HTTP requests to the backend servers
# It also handles ACME requests (as those obviously have to be handled by HTTP
relay "http_relay" {
	listen on $ipv4 port 80
	listen on $ipv6 port 80
	protocol "http_acme"

	forward to <webservers> port 10000
	forward to <ruby> port 4567
	forward to <acme> port 9999
}

# === HTTPS RELAY ===#
# There need to be *two* HTTPS relays because, for some reason, putting them both in the same block just doesn't seem to work (works with HTTP though as can be seen above).
# This relay is for HTTPS traffic on IPv4
relay "https_relay" {
	listen on $ipv4 port 443 tls
	protocol "https"

	forward to <webservers> port 10000 mode loadbalance check tcp
	forward to <ruby> port 4567 check tcp
	forward to <acme> port 9999 check tcp
}

# This relay is for HTTPS traffic on IPv6
relay "https_relay_v6" {
	listen on $ipv6 port 443 tls
	protocol "https"

	forward to <ruby> port 4567 check tcp
	forward to <webservers> port 10000 mode loadbalance check tcp
	forward to <acme> port 9999 check tcp
}

httpd.conf

server "default" {
    listen on localhost port 9999
    root "/htdocs/acme"

    location "/.well-known/acme-challenge/*" {
        root "/htdocs/acme"
        request strip 2
    }
}

server "domain1.tld" {
    alias "www.domain1.tld"

    listen on localhost port 10000
    root "/htdocs/domain1.tld"
}

server "domain2.tld" {
    alias "www.domain2.tld"

    listen on localhost port 10000
    root "/htdocs/domain2.tld"
}

OpenBSD as a VM host (and a new PC)

In order to move my stuff over to OpenBSD, I ended up getting a new VPS from a VPS provider that only provides OpenBSD VMs (obsda.ms) and I didn’t even think much about how they virtualised everything. Now, I don’t know much of anything about virtualisation (even if I do use it quite a bit on Proxmox), but I was surprised to find out that not only does their entire hardware stack run OpenBSD, but also that virtualisation on OpenBSD is actually still somewhat new.

OpenBSD seems to use vmm and vmd for its virtualisation and those only started appearing in OpenBSD 5.9 which came out in 2016; and even obsda.ms, on their websites, says that:

Keep in mind that vmm(4)/ vmd(8) is in active development, things break!

Alongside it, they provide a link to a page of known issues. Amongst other things, the most notable restriction is probably the fact that you can only pass one CPU core to a VM, so no multithreaded VMs are possible! I also read that virtualising anything but OpenBSD on OpenBSD may not actually work at all (though someone did mention that they got Alpine to run). And when looking at random Reddit posts of people asking whether or not it’s a good idea to run OpenBSD as a VM host, a lot of them say that’s it’s not really recommended and that OpenBSD as a VM host is (at least for now) a bad idea.

Because I found that quite interesting (and I found it even more interesting that obsda.ms seems to make it work!), I decided to hop on eBay and buy a cheap old Lenovo ThinkCenter with a 4th gen i5 and 32 GB of DDR3 memory (also has an SSD but I’ll swap that out for another one anyway) because I don’t actually have any computer I can run OpenBSD on at the moment (at least not on physical hardware, only virtualised). I am going to be using that as my OpenBSD (and maybe some other stuff!) box for testing things like virtualisation! Because I am now very interested in finding out how well it actually works. It’s not a good machine in any way, shape or form, but it should be plenty for some OpenBSD shenanigans ^v^