The dragonhive/dergnz web stack

Setting up this website reminded me how happy I am with my current automation and web stack.

It just works, it’s not flawless but I barely ever have to touch it, and if I want to add something new, I just copy a folder with a template docker-compose file, customize it to what I need it to do, and voila, a new service is up and running, automatically updated on a daily basis, and backed up to an external server with some extra scripts (not in this tutorial).

Setting up something like that takes a little effort, especially the figuring out part, but once you have it going, it is absolutely the most blissful and managable way to deal with more than a few services, even though this is obviously a little bit much to set up if you only care about a single web app.

But how does it work?

The core idea

I like Containers. Some people may prefer Virtual Machines, and that’s fine too, you can even combine the two concepts if you like. But this current concept is very easy to deploy, and I’ve been pondering turning it into a quick deploy stack with some basic web interface to add more containers and services to run, as it is very robust and very easy to use so far.

The core concept, is to have a few simple containers that together form the base for a very easy to use webservices host, while also being very easy to update, back up and manage. It contains the following base containers:

  • An nginx frontend webserver that sits on both 80 and 443, completely vanilla nginx, with its configuration folders exported out of the container.
  • JWilders’ nginx-proxy/docker-gen container will look at a read-only copy of the docker socket to query what containers are running, and what their desired outside hostnames would be, to generate configuration files for our frontend nginx webserver to use
  • acme-companion, which will request letsencrypt certificates and cooperate with the docker-gen container to set up HTTPS
  • Any number of other services, with at very least an environment variable for the VIRTUAL_HOST name (domain/hostname) and one for the LETSENCRYPT_HOST name (the https domain/hostname). In this case I will demonstrate it with Jekyll, which is what this website runs on :)

A basic example

For our core we need to set up a container host program. There are some options you could pick, but for simplicity sake and most widespread usage, I’ll focus on Docker for now. One could also pick for example Podman, which should in theory also have an API socket that docker-gen could use, but I have not gotten this to work myself so YMMV.

One of the advantages (and disadvantages) of Docker is that it’s centrally managed by a daemon, which you can query and command over a socket to do your bidding automatically, and we’re going to use this socket. This has some downsides in terms of security of course, if a hacker ever manages to compromise a container that has access to the docker socket, they can read everything that’s running on the docker host– as long as the docker socket is mounted read only, otherwise it’s much more dangerous and will allow the attacker to spawn new containers that could do anything. But it works exceptionally well for automation purposes, if you secure the machines that have access to it well, and realize that these are one of your weak points.

JWilders’ nginx-proxy/docker-gen

A tiny but very useful little tool I found, which uses the docker socket to see what containers there are running, checks the environment variables, and uses these to automatically generate a nginx configuration. Optionally, one can add letsencrypt-companion to also generate matching LetsEncrypt certificates, to make everything nice and HTTPS-y.

  • More info about nginx-proxy can be found here
  • More info about LetsEncrypt-Companion can be found here

Setting up a full server and Jekyll

The best way to see how this works, is by just diving right in and trying to set up the stack, with some test application like Jekyll.

  1. Install your favorite Linux distro. It doesn’t matter much if you’re using Fedora, Ubuntu, Debian, OpenSuse or something else, as long as you have Docker available. Note that some distros (Like Fedora) may have SELinux enabled, which can be painful to configure correctly during setup. If you have SELinux, I recommend using setenforce 0 to temporarily set it to permissive mode. You can always generate an SELinux policy based on the logs it’ll generate at a later stage
  2. Install both docker and docker-compose
  3. Set up your files the way you like; Be it some externally mounted disk on its own /serverfiles mounting point, or somewhere in the existing filesystem, it doesn’t matter too much what you pick. For the sake of this tutorial I’m going to use /serverfiles as the top level directory for our stored files. You can search/replace this string with whatever you end up using in practice.
  4. Create a folder for your docker-compose files, and your container’s working files. You can put those in the same directory, but I recommend splitting them out for easier backups and management, although in the end you’ll have to backup both if you want to keep your data, so it’s up to you. I’m going to assume using /serverfiles/compose-files for the compose files, and /serverfiles/data for the data that’s mounted inside the container during runtime. Docker also supports a virtual file system stored elsewhere, which is more compatible with Swarm, but for this tutorial I’ll focus on the old folder mounting system.
  5. Let’s get the actual webserver running first, which won’t do anything yet, but we’ll need in order to make Jekyll show up later on.
    • Make sure docker is running (systemctl enable --now docker or something alike for your distro)
    • Create the folder named something like /serverfiles/compose-files/nginx-frontweb and open it, then create a new docker-compose.yml file with the following contents:
 version: '2'
services:
# First we set up an nginx container as our frontend proxy. We grab and export the config folders so we can control its behavior from jwilder's docker-gen container.
  nginx-proxy:
    image: nginx:alpine
    container_name: nginx-proxy
    environment:
      - DHPARAM_GENERATION=false
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /serverfiles/srv/nginx-frontweb/conf:/etc/nginx/conf.d
      - /serverfiles/srv/nginx-frontweb/vhost:/etc/nginx/vhost.d
      - /serverfiles/srv/nginx-frontweb/html:/usr/share/nginx/html
      - /serverfiles/srv/nginx-frontweb/certs:/etc/nginx/certs:ro
    privileged: true


# This container will automatically generate nginx configs based on the environment variable `VIRTUAL_HOST` of running containers. This allows you to add any number of services very easily without every having to touch nginx configs.
  docker-gen:
    image: nginxproxy/docker-gen
    container_name: nginx-proxy-gen
    command: -notify-sighup nginx-proxy -watch /etc/docker-gen/templates/nginx.tmpl /etc/nginx/conf.d/default.conf
    environment:
      - DHPARAM_GENERATION=false
    volumes_from:
      - nginx-proxy
    volumes:
      - /serverfiles/srv/nginx-frontweb/nginx.tmpl:/etc/docker-gen/templates/nginx.tmpl:ro
      - /run/docker.sock:/tmp/docker.sock:ro # We map it read-only so that if our container gets hacked, we cannot write to the docker socket.
      - /serverfiles/srv/nginx-frontweb/conf:/etc/nginx/conf.d
      - /serverfiles/srv/nginx-frontweb/vhost:/etc/nginx/vhost.d
      - /serverfiles/srv/nginx-frontweb/html:/usr/share/nginx/html
      - /serverfiles/srv/nginx-frontweb/certs:/etc/nginx/certs:ro
    labels:
      - "com.github.jrcs.letsencrypt_nginx_proxy_companion.docker_gen"
    privileged: true


# And finally the Acme-companion requests certificates from LetsEncrypt for every machine that has the `LETSENCRYPT_HOST` variable defined
      acme-companion:
    image: nginxproxy/acme-companion
    container_name: nginx-proxy-acme
    environment:
      - DHPARAM_GENERATION=false
    volumes_from:
      - nginx-proxy
    volumes:
      - /serverfiles/srv/nginx-frontweb/certs:/etc/nginx/certs:rw
      - /serverfiles/srv/nginx-frontweb/acme:/etc/acme.sh
      - /run/docker.sock:/var/run/docker.sock:ro
      - /serverfiles/srv/nginx-frontweb/conf:/etc/nginx/conf.d
      - /serverfiles/srv/nginx-frontweb/vhost:/etc/nginx/vhost.d
      - /serverfiles/srv/nginx-frontweb/html:/usr/share/nginx/html
    privileged: true

# Here we're defining an external network for other containers to join, which is required for nginx to reach those containers internally.
networks:
  default:
    external:
      name: frontendweb

This will serve as the base for all your webservices, as it will take care of all the subdomain routing, certificate requesting, et cetera, for you. All you have to do after this, is set up extra containers with whatever services you might like, and they’ll be usable and have HTTPS certificates as soon as you start them!

  1. Now that we have that, we can set up Jekyll and have it be forwarded to the internet.
    • Go back up one folder, and create a new folder named something like mywebsite-jekyll, open it, and create another docker-compose.yml file.
    • Here’s the docker-compose.yml file that I’m using for this very website (paths changed): ``` version: ‘3’ services: jekyll: image: jekyll/jekyll environment:
      • VIRTUAL_HOST=derg.nz,dragonhive.net
      • VIRTUAL_PROTO=http
      • VIRTUAL_PORT=4000
      • LETSENCRYPT_HOST=derg.nz,dragonhive.net

      command: jekyll serve –watch –trace –incremental
      expose: - 4000 volumes: - /serverfiles/srv/mywebsite-jekyll:/srv/jekyll - /serverfiles/srv/mywebsite-jekyll/vendor/bundle:/usr/local/bundle:cached networks: default: external: name: frontendweb ``` Notice the VIRTUAL_HOST, VIRTUAL_PROTO, and VIRTUAL_PORT environment variables? That’s the magic. That’s what’s going to instruct docker-gen to create a nginx config appropriate for that (sub)domain name, and also request a HTTPS certificate while at it.

  2. Jekyll requires some manual set up before it can be used. To prepare Jekyll to be used the first time, execute the following:
    • docker-compose run jekyll jekyll new . # docker-compose run (container name) (the jekyll command) (new subcommand) (current folder)
    • you should now have the Jekyll files in your /serverfiles/srv/mywebsite-jekyll folder
    • docker-compose run jekyll jekyll build # will build the website and make it ready for serving (seems optional as it does it automatically on container startup though)
  3. Start your engines!
    • Please note, I did not explain any DNS aspects in this tutorial, I will assume you have a wildcard subdomain pointing to your server before you configure any subdomains or domains in the config files. If a DNS tutorial is desired, let me know! I’ll be happy to write one or add it here if needed.
    • Neither did I explain port forwarding here. I’ll assume you have both 80 and 443 forwarded to your webserver, as this is required for HTTP and HTTPS traffic. You’ll still need port 80 even if you only intend to use 443, as it’s a hard requirement for the LetsEncrypt certificates to be renewed.
    • cd into your frontweb-nginx folder and run: docker-compose up -d. This will bring up the frontend webserver stack, and if all goes well it should stay running (you can check with docker-compose ps)
    • cd into your ../mywebsite-jekyll folder and execute docker-compose up -d again.
    • If all went well, you should be see the Jekyll default page now, and you can start customizing it by editing the markdown files!

F.A.Q, common issues, and fixes

  • If the container doesn’t start, try running it without -d (daemon), it’ll spit out all the logs and stay attached to the console output of the containers while they run. You can also use docker-compose logs to get the last logs.
  • as of 22-8-2022 it’s still an early version of this large writeup, I hope I got everything right, if not, please feel free to poke me.

Updated: