My homelab has considerably changed over the years as I learn more and experiment with new technologies. Currently, most of my personal “services” run in a hybrid kubernetes cluster - old laptops, pis, and a couple cloud vps instances (networked via wireguard). I run a lot of the popular tools like adguard, homeassistant, plex, etc. I now have cert-manager automatically provisioning letsencrypt certificates for domains I own, external-dns configured to update cloudflare, and multiple nginx ingress controllers to host certain apps depending on what I want exposed either on my lan, shared with my family over tailscale, and/or on the open internet. One thing lacking in the kubernetes dns space currently is a simple way to configure private dns entries. For example, I don’t need “adguard.home.sclem.dev” pointing to a class C ip address on cloudflare, nor do I want to rely on the internet for lan hosts. I wanted a simple way to serve private dns records for metallb loadbalancer/nginx ingress services via coredns, without etcd (the only currently supported way in external-dns to use coredns).
I managed to hack together a pretty simple solution using only kubectl and bash. This relies on the “hosts” plugin for coredns and the fact that configmaps in kubernetes will update as long as they are not mounted at a subpath.
First, we need to generate a hosts file for our services/ingress in
the cluster. This can be done via kubectl with the -o go-template option.
Template
{{- /*
For each ingress with an ip address that does not have the
"sclem.dev/external-dns" annotation, create a host rule.
For each service that has the "sclem.dev/private-dns" hostname
annotation, create a host rule.
*/
-}}
{{- range $key, $val := .items -}}
{{- $ingStatus := (index .status.loadBalancer "ingress") -}}
{{- if not $ingStatus -}}
{{- continue -}}
{{- end }}
{{- $ip := (index $ingStatus 0).ip }}
{{- if not $ip }}
{{- continue -}}
{{- end -}}
{{- if eq $val.kind "Ingress" -}}
{{- $external := (index $val.metadata.annotations "sclem.dev/external-dns") -}}
{{- if $external -}}
{{- continue -}}
{{- end -}}
{{- range $_, $val := $val.spec.rules -}}
{{- if $val.host -}}
{{- printf "# Ingress\n" -}}
{{- printf "%s\t\t%s\n" $ip $val.host -}}
{{- end -}}
{{- end -}}
{{- end -}}
{{- if eq $val.kind "Service" -}}
{{- $url := index $val.metadata.annotations "sclem.dev/private-dns" -}}
{{- if $url -}}
{{- printf "# Service\n" -}}
{{- printf "%s\t\t%s\n" $ip $url -}}
{{- end -}}
{{- end -}}
{{- end -}}
I add an annotation sclem.dev/private-dns: "true" for services I want
exposed. My external-dns deployment is configured to only update records with a
matching annotation sclem.dev/external-dns: "true", so those are skipped.
This template can be tested with:
kubectl get svc,ing -A -o go-template-file=./dns.gotmpl
It will print out an /etc/hosts style file. We can use this output as a configmap in kubernetes, and mount it in the external dns deployment to serve a custom zone from the hosts file.
ConfigMap
Next we need to create the configmap for coredns to read, and preferably automate this action so it runs on changes to ingress/service resources in the cluster.
HOSTS="$(kubectl get ing,svc -A -o go-template-file=./dns.gotmpl)"
kubectl create cm \
-n kube-system \
--dry-run=client \
private-dns-hosts \
--from-literal=hosts="$HOSTS" -o yaml | \
kubectl apply -f -
You should now have a configmap with your generated hosts file in the kube-system namespace (example):
kubectl describe cm -n kube-system private-dns-hosts
Name: private-dns-hosts
Namespace: kube-system
Labels: <none>
Annotations: <none>
Data
====
hosts:
----
# Ingress
192.168.99.100 homeassistant.home.sclem.dev
# Ingress
192.168.99.100 registry.home.sclem.dev
# Ingress
192.168.99.100 plex.home.sclem.dev
# Service
192.168.99.101 mqtt.home.sclem.dev
CoreDNS
Lets hook our configmap up to coredns. With the helm release, we can add the “hosts” plugin right below kubernetes.
- name: hosts
parameters: /etc/private-dns/hosts
configBlock: |-
fallthrough
Mount the configmap in coredns configuration as an extra volume (via helm chart for example):
- extraVolumes:
- name: private-dns
configMap:
name: private-dns-hosts
- extraVolumeMounts:
- name: private-dns
mountPath: /etc/private-dns
The “hosts” file will be mounted at /etc/private-dns/hosts and automatically be updated when the configmap changes (not instantly, but within ~1m).
Here is what the corefile looks like:
.:53 {
log
errors
health {
lameduck 5s
}
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
hosts /etc/private-dns/hosts {
fallthrough
}
prometheus 0.0.0.0:9153
forward . 1.1.1.1
cache 30
loop
reload
loadbalance
}
We can now test this (assuming coredns is a loadbalancer service) with dig:
dig @192.168.99.53 registry.home.sclem.dev
Will give back the entry in the custom hosts file.
Automating it
Now that we can successfully create custom records in a hosts file and have coredns serve them, even if the configmap changes, we can automate the configmap so it updates whenever ingress/service objects are added/removed from the cluster. Bitnami has a kubectl image we can use to script this out.
Lets write a bash script that will use kubectl --watch to listen for
service/ingress changes, incorporating our code from above to create the
configmap. The fifo pipe here is used because kubectl doesn’t allow you to use
--watch with multiple resource types.
#!/bin/bash
set -e
# dirty script to keep coredns private dns in sync.
update() {
HOSTS="$(kubectl get ing,svc -A -o go-template-file=/data/dns.gotmpl)"
kubectl create cm \
-n kube-system \
--dry-run=client \
private-dns-hosts \
--from-literal=hosts="$HOSTS" -o yaml | \
kubectl apply -f -
}
echo "starting..."
mkfifo /tmp/pipe
kubectl get ing --watch-only -A --no-headers > /tmp/pipe &
kubectl get svc --watch-only -A --no-headers > /tmp/pipe &
while IFS= read -r data; do
echo $data
update
done < /tmp/pipe
Now lets combine these, create a configmap that holds the go template
dns.gotmpl we defined above and the script we wrote to listen for changes
indefinitely and execute the kubectl create command.
kubectl create cm -n kube-system private-dns-entrypoint --from-file=dns.gotmpl --from-file=entrypoint.sh
Finally, create a deployment using the kubectl image that runs our script:
apiVersion: apps/v1
kind: Deployment
metadata:
name: private-dns-updater
namespace: kube-system
spec:
selector:
matchLabels:
name: private-dns-updater
template:
metadata:
labels:
name: private-dns-updater
spec:
serviceAccountName: private-dns-updater
volumes:
- name: entrypoint
configMap:
name: private-dns-entrypoint
defaultMode: 0755
containers:
- name: kubectl
image: docker.io/bitnami/kubectl:1.24.3
command: ["/data/entrypoint.sh"]
volumeMounts:
- name: entrypoint
mountPath: /data
We’ll also need RBAC so the pod can read cluster ingress/services and update the configmap:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: private-dns-updater
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: private-dns-updater-role
rules:
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["patch", "create", "get"]
- apiGroups: ["networking.k8s.io"]
resources: ["ingresses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["services"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: private-dns-updater-role-binding
subjects:
- kind: ServiceAccount
name: private-dns-updater
namespace: kube-system
roleRef:
kind: ClusterRole
name: private-dns-updater-role
apiGroup: rbac.authorization.k8s.io
Kubectl apply these resources and create:
services with annotation: sclem.dev/private-dns: something.my-domain.net
and ingress objects without the annotation: sclem.dev/external-dns: "true"
Use dig to verify coredns responds with the private DNS records.
Since my home lan uses adguard as primary dns upstream and my router runs
openwrt (dnsmasq), I added a secondary dns forwarding for the sclem.dev
domain to the coredns loadbalancer service. For example:
/sclem.dev/192.168.99.53. All dns requests for sclem.dev will hit coredns
first and fall through to upstream internet servers if not found.
That’s it. Pretty simple for a private dns updater, without etcd or relying on
internet-hosted dns servers. There are many ways this could be accomplished,
and one could argue a bash script with kubectl --watch is “ugly”. But its
simple, and it works. Writing an entire k8s operator in go to listen to the
event loop is over-engineering. Then again, my homelab is over-engineered.
Code for this in action can be found in my k8s-homelab repo on github:
https://github.com/sclem/k8s-homelab
For futureproofing, here is a direct link to the sha:
The templates/ folder in the project tree has the configmap and deployment defined.