Simple DNS for your basic needs

Intro

Sometimes you don't need (or want) some complex solution to simple problem like DNS. Therefore this one will be the simplified version of previous guide with bind as only element.

For this one you will again need podman. If you are (like me in this case) doing this on centOS or similar machine, getting podman is as simple as:

# dnf install podman

If you are on some other distro, it shouldn't be that complicated.

Building container image

You could probably find one on dockerhub or some other image repo, but building our own is simple and makes sure we know what we are running.

The goal of this will be simple, create a dns server that will provide a custom local domain, and forward everything else to upstream of choice.

All my guides follow same folder structure to keep all the files:

/containers
├── build
│   └── bind
└── run
    └── bind
        └── etc

Run following commands to create it:

# mkdir -p /containers/build/bind
# mkdir -p /containers/run/bind/etc

We'll create a simple Dockerfile in /containers/build/bind with following content:

FROM alpine:latest

LABEL maintainer="Marvin Sinister"

RUN addgroup -S -g 2021 bind && adduser -S -u 2001 -G bind bind; \
    apk add --no-cache ca-certificates bind-tools bind; \
    rm -rf /var/cache/apk/*; \
    mkdir /var/cache/bind;

RUN chown -R bind: /etc/bind; \
    chown -R bind: /var/cache/bind;

HEALTHCHECK --interval=5s --timeout=3s --start-period=5s CMD nslookup ns.domain.tld 127.0.0.1 || exit 1

USER bind

CMD ["/bin/sh", "-c", "/usr/sbin/named -g -4"]

There is really nothing special about it, we are using latest alpine base image, installing bind, fixing some permissions and adding healthcheck. Make sure you use your domain instead of ns.domain.tld in HEALTHCHECK.

We can build the container now:

# podman build . -t bind
STEP 1: FROM alpine:latest
STEP 2: LABEL maintainer="Marvin Sinister"
--> Using cache 573e94441dfdbd7dcfa8e232ce7baedb9860192d749efd679b2bb8ccf64a797d
--> 573e94441df
STEP 3: RUN addgroup -S -g 2021 bind && adduser -S -u 2001 -G bind bind;     apk add --no-cache ca-certificates bind-tools bind;     rm -rf /var/cache/apk/*;     mkdir /var/cache/bind;
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/community/x86_64/APKINDEX.tar.gz
(1/53) Installing ca-certificates (20191127-r5)

...

(53/53) Installing bind (9.16.11-r0)
Executing bind-9.16.11-r0.pre-install
Executing bind-9.16.11-r0.post-install
wrote key file "/etc/bind/rndc.key"
Executing busybox-1.32.1-r2.trigger
Executing ca-certificates-20191127-r5.trigger
OK: 41 MiB in 67 packages
--> e90abd46aba
STEP 4: RUN chown -R bind: /etc/bind;     chown -R bind: /var/cache/bind;
--> c5b324d0cff
STEP 5: HEALTHCHECK --interval=5s --timeout=3s --start-period=5s CMD nslookup ns.domain.tld 127.0.0.1 || exit 1
--> 3c7e5ae2cf3
STEP 6: USER bind
--> bbb48348aa0
STEP 7: CMD ["/bin/sh", "-c", "/usr/sbin/named -g -4"]
STEP 8: COMMIT bind
--> 1bd34f66b06
1bd34f66b066f714124253631db94d50a21643bf3b8c2fbb6539862d447bee32

If you are running recent enought version of podman you might get warnings about HEALTCHECK not being supported. In that case, just add --format docker to end of build command.

You should now see the image:

# podman image ls
REPOSITORY                  TAG         IMAGE ID      CREATED        SIZE
localhost/bind              latest      1bd34f66b066  3 minutes ago  41.9 MB
docker.io/library/alpine    latest      e50c909a8df2  5 days ago     5.88 MB

Creating configurations

The main config goes into etc/named.conf:

// This is the primary configuration file for the BIND DNS server named.

// If you are just adding zones, please do that in /etc/bind/named.conf.local

include "/etc/bind/named.conf.options";
include "/etc/bind/named.conf.local";

Next, lets do options, etc/named.conf.options:

options {
        directory "/var/cache/bind";

        forwarders {
                1.1.1.1;
                1.0.0.1;
        };

        recursion yes;
        allow-query { lan; };

        dnssec-validation auto;

        auth-nxdomain no;    # conform to RFC1035
        listen-on { any; };
};

Here we define our forwarders, in this case cloudflares 1.1.1.1, allow the recursion and limit the queries to our local network (defined later).

Now, let's define local domains:

acl lan {
        127.0.0.1;
        };

zone "domain.tld" {
        type master;
        file "/etc/bind/db.domain.tld";
};

zone "122.168.192.in-addr.arpa" {
        type master;
        notify no;
        file "/etc/bind/db.122.168.192.in-addr.arpa";
};

In this configuration we define the acl, we are only allowing localhost because bind will run in container and all calls will be from localhost from containers point of view. Next we define our domain domain.tld and our reverse domain 122.168.192.in-addr.arpa. The reverse zone should correspond to your ip range, in this case 192.168.122.0/24.

Let's define the zones themself, first the forward zone domain.tld:

; BIND data file for domain.tld zone
;
$TTL    86400
@       IN      SOA     ns.domain.tld. root.domain.tld. (
                              5         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                          86400 )       ; Negative Cache TTL
;
@       IN      NS      ns.domain.tld.
ns      IN      A       192.168.122.254

; hosts
containers              IN      A       192.168.122.254

This is where we add our dns entries, in this case we have two, ns.domain.tld and containers.domain.tld both pointing to 192.168.122.254, the address of our podman host containers.domain.tld.

And lats, the reverse zone:

;
; BIND reverse data file for lan zone
;
$TTL    604800
@       IN      SOA     ns.domain.tld. root.domain.tld. (
                              5         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
;
@       IN      NS      ns.domain.tld.
254     IN      PTR     ns.domain.tld.

; hosts
254     IN      PTR     containers.domain.tld.

Similar to forward zone, the reverse zone defines the reverse pointers so you can find what hosts are behind some address.

Since bind uses DNSSEC to check the upstream repositories, we need to provide the upstream keys, you can download them from ICS website from bind.keys (at the time of writing this document). Download the file and save it as etc/bind.keys.

Since we specified user with id 2021 in Dockerfile we will make that user owner of those files:

# chown -R 2021 /containers/run/bind

Running the pod

While we could run this as a container, the decision was made to standardize on pods. There is no major reason to go one way or other, but standardization generally provides benefits longtime. In that spirit, let's create the pod:

# podman pod create --name dns -p '192.168.122.254:53:53/udp'

And run the bind container within:

# podman run -d --name bind --pod dns -v '/containers/run/bind/etc:/etc/bind:Z' --user 2021 localhost/bind:latest

To allow external machines access to our new dns, we need to open 53/udp on firewall:

# firewall-cmd --add-service=dns --permanent
# firewall-cmd --reload

Once everything starts, we can try running some dns queries to check if everyting is okay, install the bind-utils to get dig command:

# dnf install bind-utils

And then check if you can resolv some addresses:

# dig +short @192.168.122.254 google.com
172.217.16.110
# dig +short @192.168.122.254 ns.domain.tld
192.168.122.254

And that's it, now you can point your network to 192.168.122.254 to resolv you local domain.

Infrastructure monitoring with grafana and friends

Table of content

Intro
Creating configuration files
1. grafana
2. node_exporter
3. prometheus
4. loki
5. promtail
podman
Web access
prometheus
loki

Intro

In this guide we will look into how to configure infrastructure monitoring using the Grafana. Besides grafana itself, we'll use prometheus for metrics aggregation, node_exporter for log collection, loki for log agregation and promtail for log collection.

For this one you will obviously need podman. If you are (like me in this case) doing this on centOS or similar machine, getting podman is as simple as:

# dnf install podman

If you are on some other distro, it shouldn't be that complicated.

Now that we have podman let's talk about what exactly we are doing. We want to achieve following:

collect metrics from local and remote machines
collect logs from local and remote machines
display everything in pretty dashboards

For those who are not familiar, let's go through each component.

grafana is a web dashboard for visualizing data. It's most commonly used to visualize different metrics.

prometheus is a monitoring system with time series database and alerting capabilities.

node_exporter is one of many metrics exporters for prometheus, in this case exporting the metrics of node it's running on.

loki is a log aggregation system inspired by prometheus.

promtail is an agent to collect logs and send them to loki.

With all this out of our way, let's get started.

Creating configuration files

Not unlike last time, we build a simple folder structure to keep all the files:

/containers
└── run
    ├── grafana
    ├── loki
    └── prometheus

So there is containers folder in root of the filesystem, that holds runtime files for our containers. We don't need to build any containers, we'll use upstream for all our needs.

grafana

The grafana config resides in following folder structure:

grafana
├── etc
│   └── grafana
│       └── provisioning
│           ├── dashboards
│           ├── datasources
│           ├── notifiers
│           └── plugins
└── var
    └── lib
        └── grafana

To create it, run following commands:

# mkdir -p /containers/run/grafana/etc/grafana/provisioning/{datasources,plugins,notifiers,dashboards}
# mkdir -p /containers/run/grafana/var/lib/grafana

Here, we will keep both configuration and persistent data. To begin, let's grab the default config from git:

# cd /containers/run/grafana/etc/grafana
# wget https://raw.githubusercontent.com/grafana/grafana/master/conf/sample.ini -O grafana.ini

We will be running the container with user 2012 so let's change the ownership of that folder structure:

# chown -R 2012 /containers/run/grafana

We can now modify the config to suit our needs, since this is not a production, we won't change much or worry too much about security. We'll do the following:

[server]

# The public facing domain name used to access grafana from a browser
domain = grafana.domain.tld

# Redirect to correct domain if host header does not match domain
# Prevents DNS rebinding attacks
enforce_domain = true

[security]

# default admin password, can be changed before first start of grafana,  or in profile settings
admin_password = YOUR_STRONG_PASSWORD

[log]

# Either "console", "file", "syslog". Default is console and  file
# Use space to separate multiple modes, e.g. "console file"
mode = console

You should by now have a dns server, so add an entry for grafana and fill in the domain in config. We'll force the redirect for good measure. Under security configure the admin password. Since this runs within container we don't want it to write logs to disk, so we'll change the log mode to console only.

You can look through the config and change whatever else you feel is important.

node_exporter

Before starting to configure prometheus, let's figure what data we want to collect. Since we want to monitor some kind of IT infrastructure, let's assume that we are interested in node metrics. To get the data into prometheus, we need some kind of exporter. To have some kind of data wihtin pormetheus, let's get the metrics from our container host.

Let's add the ibotty/prometheus-exporter repo and install the node_exporter package:

# curl -Lo /etc/yum.repos.d/_copr_ibotty-prometheus-exporters.repo https://copr.fedorainfracloud.org/coprs/ibotty/prometheus-exporters/repo/epel-8/ibotty-prometheus-exporters-epel-8.repo
# dnf install node_exporter

Once installed, we can run and enable the service:

# systemctl start node_exporter
# systemcrl enable node_exporter

We can check if everything is okay by running:

# curl localhost:9100/metrics
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
go_gc_duration_seconds{quantile="1"} 0
go_gc_duration_seconds_sum 0
go_gc_duration_seconds_count 0
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
...

And a lot more metrics from the node.

prometheus

Now that we have some data to work with, let's get it into prometheus. As usual we'll start with desired folder structure:

prometheus
├── etc
│   └── prometheus
├── prometheus
│   └── data
└── var
    └── lib
        └── prometheus

To create it, run:

# mkdir -p /containers/run/prometheus/etc/prometheus
# mkdir -p /containers/run/prometheus/prometheus/data
# mkdir -p /containers/run/prometheus/var/lib/prometheus

Here, we will provide very simple config for prometheus to scrape the local machine in etc/prometheus/prometheus.yml:

# A scrape configuration scraping a Node Exporter and the Prometheus server
# itself.
global:
  scrape_interval: 5s
scrape_configs:
  # Scrape Prometheus itself every 5 seconds.
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']  # Scrape the Node Exporter every 5 seconds.
  - job_name: 'containers.domain.tld'
    static_configs:
      - targets: ['192.168.122.254:9100']

With this config, prometheus will collect metrics for itself, and for the container node it's running on. Notice that since prometheus is runnnig within container, we can't scrape the node config on localhost, but have to use the IP address of the host.

Just fix the permissions:

chown -R 2011 /containers/run/prometheus

And we are done with prometheus config.

loki

As usual, first we need a folder structure to hold all the data and config, for loki it looks like this:

loki
├── etc
│   └── loki
└── loki

And we can create it by running:

# mkdir -p /containers/run/loki/etc/loki
# mkdir -p /containers/run/loki/loki

We can get the default config from git:

# cd /containers/run/loki/etc/loki
# wget https://raw.githubusercontent.com/grafana/loki/master/cmd/loki/loki-local-config.yaml

The only thing we'll change inside is the location where loki will store chunks:

storage_config:
  # ...
  filesystem:
    directory: /loki/chunks

Changing it from /tmp/loki/chunks to /loki/chunks.

The only thing left to do is to fix the ownership of the folder:

chown -R 2013 /containers/run/loki

promtail

Unlike prometheus which scrapes the metrics endpoints, loki needs logs pushed into it. For that role we'll use promtail. Since we'll push logs from host, we'll install it on our container host and configure to push logs to loki.

Promtail ships as a single binary, so we can download it from directly from github. Let's download the package and put it in /usr/local/bin:

# cd /usr/local/bin
# wget https://github.com/grafana/loki/releases/download/v2.1.0/promtail-linux-amd64.zip
# unzip promtail-linux-amd64.zip
# mv promtail-linux-amd64 promtail
# rm promtail-linux-amd64.zip

Next, let's create an user that will run promtail:

# useradd --system promtail
# usermod -aG systemd-journal promtail

We have added the user to systemd-journal group, so it will be able to read journal and send it to loki.

Next, we will create the config folder:

# mkdir /etc/promtail

And add congig in /etc/promtail/promtail-local-config.yaml

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://127.0.0.1:3100/loki/api/v1/push

scrape_configs:
- job_name: journal
  journal:
    max_age: 12h
    labels:
      job: systemd-journal
  relabel_configs:
    - source_labels: ['__journal__systemd_unit']
      target_label: 'unit'

The config is fine for our needs, it reads systemd journal and pushes it to localhost on port 3100.

Let's create a systemd service file for promtail in /etc/systemd/system/promtail.service:

[Unit]
Description=Promtail service
After=network.target

[Service]
Type=simple
User=promtail
ExecStart=/usr/local/bin/promtail -config.file /etc/promtail/promtail-local-config.yaml

[Install]
WantedBy=multi-user.target

And finally update the systemd daemon files:

# systemctl daemon-reload

Podman

At last, it's time to create and start the pod. To create pod run:

# podman pod create --name mon -p '127.0.0.1:9090:9090/tcp' -p '127.0.0.1:3000:3000/tcp' -p '127.0.0.1:3100:3100/tcp'

For a tyniest bit of theory you can check last post.

We are runing three containers within mon pod, and we map the ports for grafana (3000/tcp), prometheus (9090/tcp) and loki (3100/tcp). All of them are mapped to localhost, so we will need some kind of proxy in front which we'll do later with NGINX.

First, create and start grafana:

# podman run -d --name grafana --pod mon -v '/containers/run/grafana/var/lib/grafana:/var/lib/grafana:Z' -v '/containers/run/grafana/etc/grafana:/etc/grafana:Z' --user 2012 grafana/grafana

If we try to connect to it:

# curl localhost:3000
<a href="http://grafana.domain.tld:3000/">Moved Permanently</a>.

We see that we are redirected to grafana.domain.tld, so grafana is running and our config is used.

Next, lets do prometheus:

podman run -d --name prometheus --pod mon -v '/containers/run/prometheus/var/lib/prometheus:/var/lib/prometheus:Z' -v '/containers/run/prometheus/prometheus:/prometheus:Z' -v '/containers/run/prometheus/etc/prometheus:/etc/prometheus:Z' --user 2011 prom/prometheus --config.file=/etc/prometheus/prometheus.yml --web.route-prefix=/ --storage.tsdb.retention.time=200h --web.enable-lifecycle

And again, running:

# curl localhost:9090
<a href="/graph">Found</a>.

Shows us that something is running on that port (spoiler: it's prometheus).

And last one, loki:

# podman run -d --name loki --pod mon -v '/containers/run/loki/loki:/loki:Z' -v '/containers/run/loki/etc/loki:/etc/loki:Z' --cpus=1 --user 2013 grafana/loki -config.file=/etc/loki/loki-local-config.yaml

And aftera few moments we can check:

# curl localhost:3100/ready
ready

Telling us that loki is ready to receive data.

Web access

At this point you should have a dns server with custom domain. Add the entries for our services to your dns:

grafana.domain.tld
prometheus.domain.tld
loki.domain.tld

Also, you should have NGINX installed. If not, simply run:

# dnf install nginx

And don't forget SELINUX if you have it enabled:

# setsebool -P httpd_can_network_connect 1

Once everything is prepared, create configs new virtual hosts in /etc/nginx/conf.d/.

grafana.conf for grafana:

server {
  listen 192.168.122.254:80;
  server_name grafana.domain.tld;
  root /usr/share/nginx/html;
  index index.html index.htm;

  location / {
   proxy_pass http://127.0.0.1:3000/;
   proxy_set_header Host $host;
  }

  access_log /var/log/nginx/grafana.access.log;
  error_log /var/log/nginx/grafana.error.log;
}

prometheus.conf for prometheus:

server {
  listen 192.168.122.254:80;
  server_name prometheus.domain.tld;
  root /usr/share/nginx/html;
  index index.html index.htm;

  location / {
   proxy_pass http://127.0.0.1:9090/;
   proxy_set_header Host $host;
  }

  access_log /var/log/nginx/prometheus.access.log;
  error_log /var/log/nginx/prometheus.error.log;
}

And, you guessed it, loki.conf for loki:

server {
  listen 192.168.122.254:3100;
  server_name loki.domain.tld;
  root /usr/share/nginx/html;
  index index.html index.htm;

  location / {
   proxy_pass http://127.0.0.1:3100/;
   proxy_set_header Host $host;
  }

  access_log /var/log/nginx/loki.access.log;
  error_log /var/log/nginx/loki.error.log;
}

If you are following carefully, you will notice that loki is running on port 3100, and if you have any experience with SELINUX you'll know that it loves to complain, so let's fix it by adding port 3100 to list of http ports in http_port_t variable:

# semanage port -a -t http_port_t -p tcp 3100

And now we can (re)start and enable NGINX:

# systemctl restart nginx
# systemctl enable nginx

There is one last obstacle before we can connect to our services, the firewall:

# firewall-cmd --add-service=http --permanent
# firewall-cmd --add-service=https --permanent
# firewall-cmd --add-port=3100/tcp --permanent
# firewall-cmd --reload

At this point we should be able to access all our services. So let's do some exploration and configuration.

prometheus

Go to http://prometheus.domain.tld in your browser, then in menu choose Status and Targets. You should see something similar to this:

prometheus targets

Make sure everything is UP and there are no errors.

gafana metrics

Now that we know that prometheus is collecting metrics, let's see some visualizations in grafana. Go to http://grafana.domain.tld/datasources/new, login and choose prometheus.

add datasource

Enter http://localhost:9090 as URL and click Save & Test at the bottom of the page.

prometheus datasource

Next, hop to http://grafana.domain.tld/dashboard/import.

grafana import

Enter 1860 as dashboard id, and press load. On the next screen select our (only) prometheus as data source, and press Import. You should be revarded with a dashboard similar to one on the picture below.

node_exporter

You can now start exploring what kind of exporters and dashboards are availalbe for prometheus and grafana.

loki

Now that we have some metrics, let's tackle the logs. If you remember we have added the promtail to our node, but we never started it, so let's do that now:

# systemctl start promtail
# systemctl enable promtail

With that sorted, let's turn to loki. To see what data we have in loki, we'll again use grafana. Add another source in grafana by visiting http://grafana.domain.tld/datasources/new again. But this time choose Loki and set http://localhost:3100 as URL, save with Save & Test at the bottom.

loki datasource

Unfortunately i haven't found any nice dashboards to show data from loki, however, if you visit http://grafana.domain.tld/explore and choose Loki as source, you can query all logs shiped to loki. If you search for {job="systemd-journal"}, you'll see everything currently in loki.

explore loki

You can of course play around with creating different queries or creating a custom dashboard to suit your needs. You can also add other logs to promtail and add them to loki.

This concludes this part of infrastructure monitoring, but there is much more to be done in future.

Overcomplicated homelab DNS configuration

Table of content

Intro
Buildning container images
1. BIND
2. cloudflared
3. pihole
Creating configuration files
1. cloudlfared
2. pihole
3. BIND
Podman
Web access for pihole

Intro

In this guide we will look into how to configure an overcomplicated DNS setup using pihole, bind and cloudflared, running inside a podman pod. For this one you will obviously need podman. If you are (like me in this case) doing this on centOS or Red Hat machine, getting podman is as simple as:

# dnf install podman

If you are on some other distro, it shouldn't be that complicated.

Now that we have podman let's talk about what exactly we are doing. We want to achieve following:

custom domain(s) for home lab
DNS over HTTPS to cloudflare
DNS blackhole with pihole

For those who are not familiar, let's go through each component.

pihole is a dns blackhole, it has lists of malicious and/or unwanted addresses and discards them. You can find it at pihole.net and consider it a network wide AD blocker. It also has a web interface that you can use for configuration and tracking of dns queries.

bind is a nameserver. It's probably most common nameserver in the world, it has many features and it's able to run ISP sized DNS servers. In this case we will just use it to provide a local domain. Speaking of domain, you need to decide what you will use, in this example i'll just use domain.tld.

Cloudflare is a company that provides internet services related to security and performance. Similar to googles 8.8.8.8 dns, cloudflare provides their own dns server at 1.1.1.1. Since cloudflare is not an AD revenue driven corporation, I prefer them over google. cloudflared is a daemon that forwards UDP dns requests over HTTPS to cloudflare.

So the path of request will be as follows:

origin -> bind -> pihole -> cloudflared -> cloudflare

Building container images

Now that we have all the basics covered, let's start building the images. First, we build a simple folder structure to keep all the files:

/containers
├── build
│   ├── bind
│   └── cloudflared
└── run
    ├── bind
    └── pihole

So there is containers folder in root of the filesystem, that holds build and runtime files for our containers. We need to build two containers, bind and cloudflared. We'll start with bind.

BIND

For bind, we create a simple Dockerfile in /containers/build/bind with following content:

FROM alpine:latest

LABEL maintainer="Marvin Sinister"

RUN addgroup -S -g 2001 bind && adduser -S -u 2001 -G bind bind; \
    apk add --no-cache ca-certificates bind-tools bind; \
    rm -rf /var/cache/apk/*; \
    mkdir /var/cache/bind;

RUN chown -R bind: /etc/bind; \
    chown -R bind: /var/cache/bind;

HEALTHCHECK --interval=5s --timeout=3s --start-period=5s CMD nslookup -port 5053 ns.domain.tld 127.0.0.1 || exit 1

USER bind

CMD ["/bin/sh", "-c", "/usr/sbin/named -g -4 -p 5053"]

What we are doing here is building a simple container from alpine, creating user, installing the service and fixing some permissions. Two things to notice:

healthcheck running nslookup on localhost port 5053 for address of our nameserver for chosen domain domain.tld
the bind command itself running service on port 5053

And build the container image with:

# podman build . -t bind
STEP 1: FROM alpine:latest
STEP 2: LABEL maintainer="Marvin Sinister"
--> aebbb98fb2e
STEP 3: RUN addgroup -S -g 2001 bind && adduser -S -u 2001 -G bind bind;     apk add --no-cache ca-certificates bind-tools bind;     rm -rf /var/cache/apk/*;     mkdir /var/cache/bind;
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/community/x86_64/APKINDEX.tar.gz
(1/53) Installing ca-certificates (20191127-r5)
(2/53) Installing brotli-libs (1.0.9-r3)

...

(52/53) Installing bind-dnssec-root (9.16.11-r0)
(53/53) Installing bind (9.16.11-r0)
Executing bind-9.16.11-r0.pre-install
Executing bind-9.16.11-r0.post-install
wrote key file "/etc/bind/rndc.key"
Executing busybox-1.32.1-r2.trigger
Executing ca-certificates-20191127-r5.trigger
OK: 41 MiB in 67 packages
--> c72cd8464c8
STEP 4: RUN chown -R bind: /etc/bind;     chown -R bind: /var/cache/bind;
--> dcdfc85fd12
STEP 5: HEALTHCHECK --interval=5s --timeout=3s --start-period=5s CMD nslookup -port 5053 ns.domain.tld 127.0.0.1 || exit 1
--> 0d742c04892
STEP 6: USER bind
--> 54b96184563
STEP 7: CMD ["/bin/sh", "-c", "/usr/sbin/named -g -4 -p 5053"]
STEP 8: COMMIT bind:latest
--> 3a18e54af59
3a18e54af590947fd0230193a02675f26010ab2a177e859305f0f3f98d9c22e6

If you are running recent enought version of podman you might get warnings about HEALTCHECK not being supported. In that case, just add --format docker to end of build command.

Once finished we can list the images with:

# podman image ls
REPOSITORY                       TAG          IMAGE ID      CREATED             SIZE
localhost/bind                   latest       3a18e54af590  About a minute ago  89.2 MB
docker.io/library/alpine         latest       e50c909a8df2  2 days ago          5.88 MB

cloudflared

Same procedure as bind, create a Dockerfile in /containers/build/cloudflared:

ARG ARCH=amd64
FROM golang:alpine as gobuild

ARG GOARCH
ARG GOARM

RUN apk update; \
    apk add git gcc build-base; \
    go get -v github.com/cloudflare/cloudflared/cmd/cloudflared

WORKDIR /go/src/github.com/cloudflare/cloudflared/cmd/cloudflared

RUN GOARCH=${GOARCH} GOARM=${GOARM} go build ./

FROM multiarch/alpine:${ARCH}-edge

LABEL maintainer="Marvin Sinister"

ENV DNS1 1.1.1.1
ENV DNS2 1.0.0.1

RUN addgroup -S -g 2002 cloudflared && adduser -S -u 2002 -G cloudflared cloudflared; \
        apk add --no-cache ca-certificates bind-tools; \
        rm -rf /var/cache/apk/*;

COPY --from=gobuild /go/src/github.com/cloudflare/cloudflared/cmd/cloudflared/cloudflared /usr/local/bin/cloudflared
HEALTHCHECK --interval=5s --timeout=3s --start-period=5s CMD nslookup -po=5054 cloudflare.com 127.0.0.1 || exit 1

USER cloudflared

CMD ["/bin/sh", "-c", "/usr/local/bin/cloudflared proxy-dns --address 127.0.0.1 --port 5054 --upstream https://${DNS1}/dns-query --upstream https://${DNS2}/dns-query"]

Similar to bind, but this time we use golang:alpine to build and multiarch\alpine:amd64-edge to run the cloudflared daemon. And this time the health check is for cloudflared.com address, and the port where service is running is 5054.

And run:

# podman build . -t cloudflared:latest
STEP 1: FROM golang:alpine AS gobuild
STEP 2: ARG GOARCH
--> 793a7c5f9f9
STEP 3: ARG GOARM
--> 3bb89556acd
STEP 4: RUN apk update;     apk add git gcc build-base;     go get -v github.com/cloudflare/cloudflared/cmd/cloudflared
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/community/x86_64/APKINDEX.tar.gz
v3.13.1-12-g1c7edde73a [https://dl-cdn.alpinelinux.org/alpine/v3.13/main]
v3.13.1-13-g9824e6cf00 [https://dl-cdn.alpinelinux.org/alpine/v3.13/community]
OK: 13878 distinct packages available
(1/26) Installing libgcc (10.2.1_pre1-r3)

...

(26/26) Installing git (2.30.0-r0)
Executing busybox-1.32.1-r2.trigger
OK: 208 MiB in 41 packages
github.com/cloudflare/cloudflared (download)

...

github.com/cloudflare/cloudflared/cmd/cloudflared
--> 07c20f86a9d
STEP 5: WORKDIR /go/src/github.com/cloudflare/cloudflared/cmd/cloudflared
--> 92fd2da44f5
STEP 6: RUN GOARCH=${GOARCH} GOARM=${GOARM} go build ./
# github.com/mattn/go-sqlite3
sqlite3-binding.c: In function 'sqlite3SelectNew':
sqlite3-binding.c:125322:10: warning: function may return address of local variable [-Wreturn-local-addr]
125322 |   return pNew;
       |          ^~~~
sqlite3-binding.c:125282:10: note: declared here
125282 |   Select standin;
       |          ^~~~~~~
--> 36f3ba92883
STEP 7: FROM multiarch/alpine:amd64-edge
STEP 8: LABEL maintainer="Marvin Sinister"
--> a133bb4e290
STEP 9: ENV DNS1 1.1.1.1
--> c68bcb25fe6
STEP 10: ENV DNS2 1.0.0.1
--> ce1efac0a83
STEP 11: RUN addgroup -S -g 2002 cloudflared && adduser -S -u 2002 -G cloudflared cloudflared;         apk add --no-cache ca-certificates bind-tools;         rm -rf /var/cache/apk/*;
fetch http://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
(1/18) Installing fstrm (0.6.0-r1)

...
(18/18) Installing ca-certificates (20191127-r5)
Executing busybox-1.33.0-r1.trigger
Executing ca-certificates-20191127-r5.trigger
OK: 22 MiB in 38 packages
--> 3fc0d971570
STEP 12: COPY --from=gobuild /go/src/github.com/cloudflare/cloudflared/cmd/cloudflared/cloudflared /usr/local/bin/cloudflared
--> f99b87627f7
STEP 13: HEALTHCHECK --interval=5s --timeout=3s --start-period=5s CMD nslookup -po=5054 cloudflare.com 127.0.0.1 || exit 1
--> c0b3a24146e
STEP 14: USER cloudflared
--> 66804934da3
STEP 15: CMD ["/bin/sh", "-c", "/usr/local/bin/cloudflared proxy-dns --address 127.0.0.1 --port 5054 --upstream https://${DNS1}/dns-query --upstream https://${DNS2}/dns-query"]
STEP 16: COMMIT cloudflared:latest
--> 7d8b02575fe
7d8b02575fea97bc09ca94fdf85569c47d15112f3930405e8c25d477fa491aaf

After a while we have the new image:

# podman image ls
REPOSITORY                       TAG          IMAGE ID      CREATED         SIZE
localhost/cloudflared            latest       7d8b02575fea  3 minutes ago   95.7 MB
localhost/bind                   latest       3a18e54af590  18 minutes ago  89.2 MB
docker.io/multiarch/alpine       amd64-edge   901ee590dcdb  22 hours ago    25.8 MB
docker.io/library/golang         alpine       54d042506068  2 days ago      308 MB
docker.io/library/alpine         latest       e50c909a8df2  2 days ago      5.88 MB

pihole

We will use the upstream docker images for pihole.

Creating configuration files

Now that we have images, we will create the configuration files to run services.

cloudflared

Let's begin with the simplest one. cloudflared doesn't need any kind of configuration, we'll run it as pure ephemeral container.

pihole

There is not much we need to configure for pihole. We will just prepare the folders for persistence. The configuration options will be provided as environment variables on runtime. We need to create the following folder structure under /containers/run/:

pihole/
└── etc
    ├── dnsmasq.d
    └── pihole

And that's it for pihole for the moment.

BIND

The most configuration will be required for bind. Here we will define our own domain. Create the following structure under /containers/run/:

bind/
└── etc

Here we have two options, either copy the config from existing bind; by either running a container without mounting volumes or having existing bind or creating them ourselves. Since it's simple config I'll just put all files here.

The main config goes into etc/named.conf:

// This is the primary configuration file for the BIND DNS server named.
//
// Please read /usr/share/doc/bind9/README.Debian.gz for information on the 
// structure of BIND configuration files in Debian, *BEFORE* you customize 
// this configuration file.
//
// If you are just adding zones, please do that in /etc/bind/named.conf.local

include "/etc/bind/named.conf.options";
include "/etc/bind/named.conf.local";

And then we need to create the rest of the bunch, etc/named.conf.options:

options {
        directory "/var/cache/bind";

        forwarders {
                127.0.0.1 port 5054;
        };

        recursion yes;
        allow-query { lan; };

        dnssec-validation auto;

        auth-nxdomain no;    # conform to RFC1035
        listen-on port 5053 { any; };
};

The things configured here:

forward the unknown queries to 127.0.0.1 on port 5054 which is the address of pihole
allow recursion
limit queries to lan acl (defined later)
listen on port 5053 on all interfaces

Next, etc/named.conf.local:

acl lan {
        10.88.0.0/16;
        127.0.0.1;
        };

zone "domain.tld" {
        type master;
        file "/etc/bind/db.domain.tld";
};

zone "122.168.192.in-addr.arpa" {
        type master;
        notify no;
        file "/etc/bind/db.122.168.192.in-addr.arpa";
};

The things configured here:

the lan access control list allowing 10.88.0.0/16, which is the default address space of podman and 127.0.0.1 which is localhost
the forward zone for domain.tld (defined later)
the reverse zone for 192.168.122.0/24 (defined later)
any other zones (not in scope of this document)

Now we can create our own domain.tld zone in etc/db.domain.tld file with following content:

; BIND data file for domain.tld zone
;
$TTL    86400
@       IN      SOA     ns.domain.tld. root.domain.tld. (
                              5         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                          86400 )       ; Negative Cache TTL
;
@       IN      NS      ns.domain.tld.
ns      IN      A       192.168.122.254

; hosts
containers              IN      A       192.168.122.254

; services
pihole                  IN      A       192.168.122.254

Reverse domain will in this case go into db.122.168.192.in-addr.arpa, where the numbers in file name corespod to reverse of the ip range, in this case we are doing reverse zone for 192.168.122.0/24 so the file is named 122.168.192 (droping the last 0 because it's a /24 domain).

;
; BIND reverse data file for lan zone
;
$TTL    604800
@       IN      SOA     ns.domain.tld. root.domain.tld. (
                              5         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
;
@       IN      NS      ns.domain.tld.
254     IN      PTR     ns.domain.tld.

; hosts
254     IN      PTR     containers.domain.tld.

A few notes:

If your domain spans across multiple subnets, you will wnat multiple reverse zones
Notice that we have two forward addresses (containers.domain.tld and pihole.domain.tld) but only one reverse entry (containers.domain.tld). While you can have multiple reverse (PTR) records, you don't have to.

Since we specified user with id 2001 in Dockerfile we will make that user owner of those files:

# chown -R 2001 /containers/run/bind

And the config is done. Almost there.

Podman

Finally it's time to create and start the pod. To create pod run:

# podman pod create --name dns -p '192.168.122.254:53:53/udp' -p '127.0.0.1:8080:80/tcp'  -p '127.0.0.1:8443:443/tcp'

A bit of theory time. Pods are similar to containers, but within a pod, you can run multiple containers. The networks within pod is shared, therefore, you define the port mappings on pod level, and not for each container. In this case we'll run three containers within pod, and we map the following ports:

The starndard dns port 53/udp to our host address, so that it's available from whole network
80/tcp and 443/tcp to our localhost. This is pihole web interface. We are making it available only from localhost, but we can stick a reverse proxy in front later.

We also give the pod a name, in this case dns.

And once whe have a pod running, we can start the containers:

# podman run -d --name cloudflared --pod dns --user 2002 localhost/cloudflared:latest
# podman run -d --name bind --pod dns -v '/containers/run/bind/etc:/etc/bind:Z' --user 2001 localhost/bind:latest
# podman run -d --name pihole --pod dns -v '/containers/run/pihole/etc/pihole:/etc/pihole:Z' -v '/containers/run/pihole/etc/dnsmasq.d:/etc/dnsmasq.d:Z' -e=ServerIP='192.168.122.254' -e=DNS1='127.0.0.1#5053' -e=DNS2='no' -e=IPv6='false' -e=TZ='Europe/Berlin' -e=WEBPASSWORD='MY_STRONG_PASSWORD' pihole/pihole:latest

As you can notice, we are binding the appropriate folders to each container, and providing a few extra options to pihole, of wich password is the one you should probably change.

Once everything starts, you can try running some dns queries to check if everyting is okay, if you don't have them install bind-utils to get dig command:

# dnf install bind-utils

And then check if you can resolv some addresses:

# dig +short @192.168.122.254 google.com
172.217.16.110
# dig +short @192.168.122.254 ns.domain.tld
192.168.122.254

To access the dns service from outside the host, we need to open 53/udp on firewall:

# firewall-cmd --add-service=dns
# firewall-cmd --add-service=dns --permanent

Web access for `pihole`

To access web interface of pihole we need a proxy in front, in this case we'll use NGINX. First thing we need to do is install it:

# dnf install nginx

Once done, create the config file for new virtual host in /etc/nginx/conf.d/pihole.conf:

server {
  listen 192.168.122.0:80;
  server_name pihole.domain.tld;
  root /usr/share/nginx/html;
  index index.html index.htm;

  location / {
   proxy_pass http://127.0.0.1:8080/;
  }

  access_log /var/log/nginx/pihole.access.log;
  error_log /var/log/nginx/pihole.error.log;
}

And start and enable the service:

# systemctl start nginx
# systemctl enable nginx

A few more details, open the ports in firewall:

# firewall-cmd --add-service=http --permanent
# firewall-cmd --add-service=https --permanent
# firewall-cmd --reload

And allow nginx to connect to pihole on localhost in selinux:

# setsebool -P httpd_can_network_connect 1

You should be able to access pihole at http://192.168.122.254 in your web browser and login using your password. You can also point other machines on your network to use it as dns server. And of course, you should add pihole to your local domain.

Adding geoip2 to NGINX

The following guide will help with seting up GeoIP2 database and logging locations of source IP adresses in NGINX access log. The guide is for debian, but should be simple to adopt for other distributions.

To start, we need some packages, the packages are available in bullseye repo:

# apt update
# apt install libnginx-mod-http-geoip2

We also need the GeoIP database. You can download one for free at maxmind website. You need to create an account and create API key. The files can be downloaded from their webpage, but if you have the API key you can use the following links:

https://dev.maxmind.com/geoip/geoip2/geolite2/GeoLite2-Country&license_key=GEOIP2_API_KEY&suffix=tar.gz
https://dev.maxmind.com/geoip/geoip2/geolite2/GeoLite2-City&license_key=GEOIP2_API_KEY&suffix=tar.gz

Once you have those extract them and place the .mmdb files into /etc/nginx/geoip folder:

# ls /etc/nginx/geoip/
GeoLite2-City.mmdb  GeoLite2-Country.mmdb

With the database and packages sorted, we can configure NGINX. Create a new config /etc/nginx/conf.d/geoip2.conf:

##
# Add GEO IP support 
##

geoip2 /etc/nginx/geoip/GeoLite2-Country.mmdb {
  auto_reload 5m;
  $geoip2_metadata_country_build metadata build_epoch;
  $geoip2_data_country_code source=$remote_addr country iso_code;
  $geoip2_data_country_name country names en;
}

geoip2 /etc/nginx/geoip/GeoLite2-City.mmdb {
  $geoip2_data_city_name city names en;
}

# Enabling request time and GEO codes
log_format custom '$remote_addr - $remote_user [$time_local]'
                  ' "$request" $status $body_bytes_sent'
                  ' "$http_referer" "$http_user_agent"'
                  ' "$request_time" "$upstream_connect_time"'
                  ' "$geoip2_data_country_code" "$geoip2_data_country_name"'
                  ' "$geoip2_data_city_name"';

access_log /var/log/nginx/access.log custom;

Test the config with nginx -t and restart NGINX:

# systemctl restart nginx

After which the access log will contain lines with country codes:

# cat /var/log/nginx/access.log
198.20.99.130 - - [31/Jan/2021:11:58:00 +0100] "GET /robots.txt HTTP/1.1" 200 26 "-" "-" "0.000" "-" "NL" "Netherlands" "-"

Depending on your existing config you might end up with double logs of each request, the standard NGINX log, and the custom log with GeoIP. In that case edit /ext/nginx/nginx.conf and remove (coment) the access_log line.

Newer

Older

Simple DNS for your basic needs

Intro

Building container image

Creating configurations

Running the pod

Infrastructure monitoring with grafana and friends

Table of content

Intro

Creating configuration files

grafana

node_exporter

prometheus

loki

promtail

Podman

Web access

prometheus

gafana metrics

loki

Overcomplicated homelab DNS configuration

Table of content

Intro

Building container images

BIND

cloudflared

pihole

Creating configuration files

cloudflared

pihole

BIND

Podman

Web access for pihole

Adding geoip2 to NGINX

Web access for `pihole`