I’m a big fan of Kubernetes. This site is ran on it! It’s no secret that I run a significantly sized bare metal cluster in my house.
I have always had a pain with exposing various HTTP/S services back into my home network, especially with much more strict browser settings, and I am slowly tiring of having to install my own certificate authority, configure haproxy and more, just to get a basic application deployed and available.
My current architecture is:
Client Device > DNS Cluster > HAproxy Cluster > Kubernetes Service exposed via MetalLB > Application Pods
This always felt a bit wasteful, needing to run two haproxy servers outside of my kubernetes stack, to manage inbound connectivity. It wasn’t too bad to look after with the likes of Ansible Playbooks and other automation tools. I just knew, it could be far better.
Breakdown of the services in play:
Nginx-Ingress is a prometheus operator designed to be able to expose websites or applications via a nginx pod acting as a load balancer. It uses kubernetes annotations to look for being invoked/configured, and supports endless possibilities when it comes to configuration.
MetalLB is another very useful service that allows us to expose LoadBalancer types to my external LAN via BGP. I have a configured BGP peering setup between my K8 cluster and my home routers (dual pfsense via CARP/VRRP), when I expose a service of LoadBalancer type, MetalLB announces an IP address from a supplied range, as a /32 to my router pair, with each kubernetes worker node as a route, allowing me to announce a single IP address for a service, from 6 worker nodes.
External-DNS is a kubernetes operator that allows us to specify DNS records via kubernetes annotations. When presented with an annotation, this service can reach out to external DNS servers and register those hostnames within the DNS zones dynamically. It supports many different DNS providers, but here we will be doing some clever things with split-horizon DNS via BIND9 and Azure DNS Zones.
Cert-Manager allows us to use annotations again, to provision Lets Encrypt (or any ACME compatible provider) certificates to our cluster via kubernetes secrets. Nginx-ingress can then read these secrets to expose our applications over HTTP/S. Cert-Manager allows all of the standard DNS challenges, webhooks etc to conform to the ACME standards for SSL certificate provisioning.
Azure DNS Zones
Finally, we have Azure DNS Zones. I chose these because I already have a fair few services in an Azure tenant for DR, and Azure DNS hosting is extremely cheap, however this could be replaced with any provider that works with the External-DNS operator.
New Process Flow
Getting it all set up
We need to get all of our various operators installed and configured, so lets do that first.
Installation of MetalLB
The best thing to do is follow the docs here to install the operator. Once installed, you need to configure MetalLB to interact with your BGP Peers, the documentation on how to do this is pretty verbose, but as an example - you can provide a configuration like this:
Define an address pool:
Define the bgp peers:
And finally, the advertisment:
once these are all applied, and you have configured the routing engine on your BGP peer, you can now announce services from Kube on “LAN” IP Addresses
to do this, you can simply add:
type: LoadBalancer to your existing
Installation of nginx-ingress
This one I found to be a bit tricky to get perfect. In my instance I already had it deployed and needed to change some settings, if you are deploying fresh it should be a little easier!
I use the official NGINX repos for the installation, I didn’t use Helm for my deployment and used the manifests as I find them a little easier to version control.
There is some good installation instructions here
In short - Clone the repo (I am using version 3.1.1) and..
args section to have the following:
You can find out what each of these settings do by looking at the docs here
The main part that made a difference for me, was setting these:
I found that there is another similar nginx-ingress resource out there, that uses different documentation and parameters. Don’t fall into the same trap I did!
kubectl apply -f common/ns-and-sa.yaml
kubectl apply -f rbac/rbac.yaml
kubectl apply -f common/nginx-config.yaml
kubectl apply -f common/ingress-class.yaml
kubectl apply -f common/crds/k8s.nginx.org_virtualservers.yaml
kubectl apply -f common/crds/k8s.nginx.org_virtualserverroutes.yaml
kubectl apply -f common/crds/k8s.nginx.org_transportservers.yaml
kubectl apply -f common/crds/k8s.nginx.org_policies.yaml
kubectl apply -f common/crds/k8s.nginx.org_globalconfigurations.yaml
kubectl apply -f daemon-set/nginx-ingress.yaml
kubectl apply -f service/loadbalancer.yaml
The final line is key, this deploys the nginx-ingress central service, and announces it via the MetalLB service.
For me, this is what it looked like:
kubectl get services --namespace nginx-ingress
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-ingress LoadBalancer 10.102.237.73 XXX.XXX.XXX.XXX 80:32335/TCP,443:31026/TCP 122m
If you visit the IP under EXTERNAL-IP on your browser, you should see a 404 response from NGINX.
Installation of External-DNS
This one is a nice and simple one. There are no huge manifests for this one, just some role setups and containers. I have it split into three files:
Here are links to my examples:
You will see that there are various placeholders with
XXXXX These will need to be changed to suit your network.
is a useful document on how to configure the
external-dns.yaml file to work with Azure.
There are a lot of examples in that repo, you will need to find one best suited to your situation and follow that.
Installation of Cert-Manager
Again, install the manifests and the CRD’s - these can be found here
I am running version
There are a few ways of configuring Cert-Manager, you can have it watch for only specific namespaces, or you can install it cluster-wide.
I run many namespaces for segregation, so I opted for cluster-wide by using a
I configured two resources:
To make this work on internally hosted sites, LE cannot hit a .acme/well-known path on your servers which is the usual method as we don’t have them exposed to the internet. So we must use a DNS challenge. This is where the Azure DNS comes in - our External-DNS service publishes the records within the Azure DNS service, which Lets Encrypt can then access as it is a public DNS zone. All of the A records point to internal IP’s, but LE doens’t care.
An Example staging config:
# The following is the secret we created in Kubernetes. Issuer will use this to present challenge to Azure DNS.
The same applies for prod, but the only difference is
You can use the same Service Principal credentials you created for External-DNS in the configuration file.
To create the secret
client-secret for the authentication, run the following command:
kubectl create secret generic azuredns-config --from-literal=client-secret=YOUR_APP_PASSWORD_FROM_THE_SP_HERE -n cert-manager
Using it all together
Now we have all of our moving parts done, we can now create a service for an application, here is the syntax you would use:
- host: something.domain.name
- path: /
And now we can see, the LE certificate is now present, and we can access our service my git server for example:
openssl s_client -showcerts -servername git.XXXXXX.XXXXX -connect git.XXXXXXX.XXXXXX:443 </dev/null
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
depth=1 C = US, O = Let's Encrypt, CN = R3
depth=0 CN = git.XXXXXXX.XXXXXX
0 s:CN = git.XXXXXXX.XXXXXX
i:C = US, O = Let's Encrypt, CN = R3
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Dec 22 00:19:13 2023 GMT; NotAfter: Mar 21 00:19:12 2024 GMT