Thaiger Sprint 2025

Starring H2O, the HTTP server (& dropping Nginx)

Date put out
Time burden
9–14 minutes
Word score
2871
Keywords
thailand
hackathon
nix
h2o
Writer
Crafted in
จันทบุรี ไทย
License
CC-BY-NC-SA-4.0 (https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode)

Abstract

Thaiger Sprint 2025 was a coding sprint for Nix in เชียงใหม่ ไทย (Chiang Mai, Thailand) during the middle of February. It’s usually a good idea to come with a plan on what to hack on. Many folks, myself included, decided to chew on an idea in their backlog while having the space & time to do so. In 2024 I took on the Movim project, both package & module. In another web-focused approach, 2025’s goal was chewing on an H2O module. While I had the fortune of H2O technically being already packaged, the package compiled, but was incomplete & H2O provided some new challenges as it’s breadth as a web server extends beyond a web app.

H2O, an overlooked HTTP server

At some point in 2024 I was curious if there was an decent HTTP servers as a C library; that search led me to H2O which, after looking the documentation, seemed like it was something more. Looking at the ‘Configure’ section’s heading it became clear this wasn’t just some http.server or simple-http-server or a basic library, but a full-featured web server & reverse proxy that could replace the likes of Nginx, Apache, & Caddy – & even, according to their benchmarks, outperform as well. This was a bit surprising to me since I had never heard of it… & I have done my fair share of poking around for Nginx alternatives after getting frustrated with its configuration language on many occassions. If the curiousy choice of Mruby wasn’t raising an eyebrow to hint at it, clicking the blog, the reader is greeted with a wall of Japanese script. Maybe it just isn’t a popular in the West? This is my only guess.

In the past I always sticking with Nginx since it was well-supported in Nix & other communities while being quicker than Apache. I mentioned my own frustration with the config language, but I am definitely not the only one (H2O uses YAML). I have never heard of anyone using Nginx as a library either… Nginx is something you put in front of another application for load balancing and/or proxying — maybe you would script it with OpenResty (Lua) if you were bold (I know I did in 2015). H2O however, you could learn for this use case, but also use it as a library if you needed a web server with good performance & features. This started to look like an appealing tool to me …if only it were better supported with my tooling.

H2O was already packaged! Sorta…

Unlike needing to learn how to package a PHP to cover Movim, Nixpkgs already had pkgs.h2o for users. I was ready to start writing my module but quickly learned that without testing, it was a very incomplete package. What needed tweaking?

  • H2O moved to a rolling release strategy where now all pushes to the default branch are considered release-worthy. I would assume with its maturity, it became pretty arbitrary when to cut new release when HTTP hasn’t been moving much post-HTTP/3 & a lot of the features are already there. This meant the version in Nixpkgs was quite behind without any new ‘releases’ happening.
  • meta.mainProgram was missing.
  • The binary requires Perl be on the $PATH or as $H2O_PERL for Perl things.
  • OpenSSL needs to be on the $PATH for TLS things.
  • Mruby support was silently not being added since ruby & bison were missing. I made it configurable with a withMruby option to slim down the build if you really need it.
  • We would want some basic tests for this too since it would be such an integral part of a larger system.

These are all covered now for everyone!

TLS needs & ‘web server’ shortcomings

Despite the other features, for the intents of a NixOS module H2O makes the most sense as reverse proxy. As such, I had to have a look at Nginx & Caddy in particular to see what sort of features a user would want/need.

One thing that stuck out to me as bad design (assumed for a lack of refactoring) is the TLS situation. For starters: the name. SSL is the deprecated name in favor of TLS in 2015, so why not start here & name the options tls making downstream usage of either name an implementation detail… in fact, everything related to TLS should be scoped here (similarly, HTTP settings should be scoped to http). NixOS’s Nginx module has an assertions for whether addSSL, onlySSL, forceSSL, rejectSSL are compatible or mutually exclusive… reject seems to be a workaround so why not use enum over a stack of booleans? I went with tls.policy = types.enum [ "add" "only" "force" ] as quite clear & requires if you use TLS from the H2O NixOS module, you must set a valid policy by contstruction; instead of having heaps of boolean logic, you can just check tls.policy == "force" & be on with some code.

Nginx has services.nginx.recommendedTlsSettings = true, but this has some flaws. First again comes the name as being mismatched between the options saying SSL, but also this one switches to a casing that loses information which doesn’t match most of the config using Tls instead of the expected initialism TLS (acronyms/intialism are encoded with capital letters to precisely show the are a stand-in for a longer phrase). There’s a comment to in the code: 

# Keep in sync with https://ssl-config.mozilla.org/#server=nginx&config=intermediate

but the configuration Mozilla offering up is just JSON & with 3 settings levels depending on needs. Why not make this an enum too‽ The H2O module supports tls.recommendations = [ "modern" "intermediate" "old" ] to cover all the levels (with a top-level defaultTLSRecommendations option too), but the recommendation is also directly to the actual JSON (& a mirror) which is then folded into our config instead of some manual process to review in syncing… which you stroll a bit further down in the config to see how Nginx is deviating from the config which isn’t “in sync” at all. The other nice part about choice this level of flexibity is a users can choose to go recommendations = "old" & override their H2O OpenSSL library with something like insecure openssl_1_1 to cover a legacy device if that user needs to while still getting that same support level as the "intermediate" & "modern" values from the NixOS module. The documentation block is thorough too after a lengthy review.

Biased, but I think these are improvements… however, one downside now is lacking compatibility with the ‘web server’ where a couple of things like the ACME tests allow Apache (httpd) & Nginx to drop-in replace one another. I am okay with breaking this compatibility for less asserts, clearer hierarchy for virtual host configuration, & not manually maintaining security stuff.

ACME testing

Speaking of ACME… I doubt we could consider H2O a replacement for Nginx in your setup if ACME wasn’t supported & supported well enough. The only way to have some certainy of this claim for users to trust it is to validate a test run of the ACME challenges. I was pointed by a Thaiger Sprint antendee to nixos/tests/step-ca.nix which did the job similating ACME which was great, except now I needed to test all my first-attempt code. This code didn’t work naturally, but a little coercing, we finally proved to some degree that H2O is capable of serving our challenges too!

& basic testing

Basic functionality is poked out via nixos/tests/web-services/h2o which covers basic tests of a variety of features using VMs. The basic tests cover responses, compression, adding mime types, Unicode, TLS, & some settings overrides. mruby verifies inline & file handlers work. tls-recommendations over that those Mozilla recommendations are working as expected.

Real world spin via an XMPP social media

As the maintainer for the Movim module too, I decided to add H2O support here as well. Many PHP + FastCGI applications allow inserting a virtual host & helping out with good defaults & routing for the end user. This was also going to be a good test the proxies & websockets worked too (with a bonus of now testing the Ejabberd XMPP server as well). This pointed out some issues H2O module… not in terms of things being broken, but in terms of usability needed some minor tweaks — especially documentation. Needless to say, it did all work as expected, even in production, but there was learning curve to the syntax that made it take a bit longer than expected.

What should I do with this H2O module?

Performance & stability have been good my last couple of days with H2O in production. I decided to go all-in on it to make sure everything worked & it was kind of exciting to see something I hadn’t seen on a server in years: 

$ systemctl status nginx.service
Unit nginx.service could not be found.

At this point, I think this NixOS H2O module is more than worth playing around with. You can actually start throwing some real projects at it & I am pleased/proud of the usability aspect of it relative to my experience with Nginx. This could be a great fit for the crowd that is intrigued by Caddy for UX fixes, but do not wish to double their hardware requirements & take a cut in performance. An example config today would look like: 

services.h2o = {
   enable = true;
   settings = {
      access-log = "/var/log/h2o/access.log";
      compress = "ON";
      http2-reprioritize-blocking-assets = "ON";
      ssl-offload = "kernel";
   };
   hosts."example.org" = {
      tls = {
         policy = "force";
         recommendations = "modern";
      };
      acme.enable = true;
      settings = {
          paths."/static" = {
             "file.dir" = import ./static/file/linkfarm.nix;
          };
          paths."/" = {
             "proxy.reverse.url" = "http://127.0.0.1:42069";
          };
      };
   };
}

Aside from the module, it is very easy to make a little H2O development server for a project as well such as the one I am using with Flakes 

pkgs.writeShellApplication {
   name = "dev-server.sh";
   runtimeInputs = with pkgs; [ h2o ];
   text = ''
      DEV_SERVER_HOST=''${DEV_SERVER_HOST:-"127.0.0.1"}
      DEV_SERVER_PORT=''${DEV_SERVER_PORT:-"8866"}

      H2O_conf="$(mktemp)"

      cat << EOF > "$H2O_conf"
      ssl-offload: 'kernel'
      http2-reprioritize-blocking-assets: 'ON'
      hosts:
         devserver:
            listen:
               host: '$DEV_SERVER_HOST'
               port: $DEV_SERVER_PORT
            paths:
               /:
                  # Or sit afront any other build directory
                  file.dir: '$(nix --extra-experimental-features flakes build --offline --fallback --builders "" --print-out-paths --no-link .#${name})'
      EOF

      echo "Starting dev server …"
      echo "Listen: $DEV_SERVER_HOST:$DEV_SERVER_PORT"
      echo "Config: $H2O_conf"
      h2o --mode=worker --conf="$H2O_conf"

      trap 'rm -f "$H2O_conf"' EXIT
   '';
}

What are some things that could come later?

Some basic Mruby scripts could be nice like having basic auth not being something everyone would need to manually write. Mruby also should be able to support some Gems, but H2O wants to look in a particular location relative to itself which would involve symlinking Gems into the store (granted this is an h2o not nixos/h2o issue); this wouldn’t be too hard, but I don’t yet have a use case (nor do I want even more scope creep as is). A log directory should be able to be set so it can be used in other places, but I would need to think about that log rotation directive; however, some use cases could be clear like using fail2ban to scan this log to temporarily ban that scanner scourge trying to find long-hanging, outdated WordPress installs.

Since the output is YAML, I am not sure it needs a whole lot more would be need since outputing is simple & less confusing than Nginx or even Apache for Nix so no need to replicate 80% of the config features in the module. The YAML does have some downsides tho in that Nix can represent duplicate key names in a block & I don’t think this colud ever be solved.

On the Thaiger Sprint event itself

Having more expectations this year made the event a little clearer to me what sort of plan / scope I wanted to set before the event. Making my 2nd visit to the Nortern City made things slightly easier as well — tho I can’t say I had any problems the previous year. Since I don’t get a lot of in-person opportunities to be around fellow tech nerds in my smaller town, a large crowd offered more folks to spend time with — I tried to mingle a little with everyone which was refreshing to see both old & new faces. I got the networking needs, bashed the technologies I loudly deplore, showed many folks many new things, & had food + drink to relax with many friends.

This year I passed on many of the events. There is a FOMO regardless, which is good since no one should feel pressured or not to go do anything they don’t want. There was always a good crowd focused on general socializing & hacking out their plan …but I definitely had a hard time passing up a free meal. As a long-time resident & night owl, I didn’t feel the desire to rise for stuff I have seen similar of around the country already which kept me in a better mood thru the event.

While the experience & folks were good, I didn’t enjoy the particular location in Nimman. Chiang Mai is a tourist center, so it is no surprise to see many tourists, but staying in a subdistrict that is primarily for tourism was a let down. The rooms were more expensive than the previous year. It was difficult to find cheap food or coffee without a significant walk. I know to the Europeans it all seems relatively inexpensive, but my budget for everything was like double in my town — & not everyone there was even from a wealthy Western country. Being an extension off one of the airport runways also had a clear sonic downside both for conversation but also sleep. Chiang Mai is also one of the cities that just doesn’t have regular vested motorcycle taxis or cheap transport (even the red Mazdas are expensive) where everyone is demanding you install some data-sucking app (which is only available on the Android & iOS duopoly; this made travel inconvenient as I had to ask others to book rides for me (what’s next? will they go cashless too‽)). Perhaps another season would be better too — like not the burning season with all the ash & haze. I would have probably been happier in a more remote area so long as the company & caffeine were good.

Also, it sucked the whole thing was organized on Matrix which is a slow, expensive, wasteful protocol (by design) with slow, expensive, & wasteful clients (due to protocol design & usage of Element-only features) that is more bloat than needed for organizing (I don’t think we need multiple, eventually-consistent copies of the chat for the event). This had a knock-on effect of folks wanting to contact me (the contacting part I appreciate) on that protocol after the event since it was the only thing used & gateways/bridges weren’t working. The ends up creating the sorts of locked-in, network-effected situations of other platforms — except, aside from be FOSS, this chat platform is not a good design to encourage self-hosting or efficiency. The irony was at the tail end of the event, we got the message that even Matrix.org can no longer afford their servers — wanting to cut the gateways/bridges first to try to save costs, which was ironic since so many folks advertized Matrix as the future for this feature (while not realizing “gateways” as a concept existed for long time, before ‘rebranding’ it “bridges” in the way Discord lies about “servers”). I had to keep a second phone on me for the event since

  1. I didn’t want to install a Matrix client on my new phone knowing how resource intensive it is (ironic since 2 phones is less efficient, but the new phone has less performance & runs Linux so the culling of background apps isn’t amazing)
  2. Today’s kids can’t handle you not replying in minutes let alone wait til the next day for me to open Matrix & wait 5 minutes for the sync on my laptop browser the next day.

While I am a fan of the features of XMPP & would have preferred that, IRC would have been good enough, or at least using IRC as the linga franca anyone could use as a gateway like the previous year. …At least there was an effort to use more geo: URI schemes and/or OpenStreetMap instead of exclusively Google Maps™ shortlinks that both hide the GPS coordinates but also require JavaScript on a proprietary service.

That aside, the friends & the knowledge shared were more than worth any specific location / chat protocol grumbles & I am grateful that this event didn’t involve me needing to get a visa or learning a new language to combat swindlers. On some occassions even I got to help others make sure they were getting good deals or translate something off the cuff. It’s also nice to know after all of these years, I still have a great time traveling thru my country regardless of where — where can trade jokes in exchange for learning something new about the language. I would definitely go again if it were offered again — hopefully with a lot of similar faces. & if it were more financially accessible, I would like to go to more Nix hackathons elsewhere as well.

๛