ResCommunes

RAGTAGS

:: THE MACHINES HAVE TAKEN OVER ::

K8s with Traefik

The other day I learned that the Ingress NGINX project is being retired, and as someone who develops first and works with sysops second, I was shocked because I truly did not know there were strong alternatives. Ingress NGINX is being retired mainly because the project has collected too much technical and security related maintenance debt to keep pace with modern Kubernetes patterns. And here is the surprise: it has not been the best option for quite some time. With the rise of the Gateway API, Traefik steps in as a far more natural fit because it is easier to operate, more secure by design, and aligned with the direction the Kubernetes networking ecosystem is moving toward.

Let's look into what that means and how I found it useful for my use case.

My Use Case

For context, my work with Traefik and the Gateway API is part of a project I am building called Compass Ops. The purpose of Compass Ops is to offer a simple and dependable environment for local Kubernetes development on Minikube and also serve as a clean starting point when moving into a managed Kubernetes provider. Because the project needs to work smoothly in both environments, choosing a modern and reliable approach to traffic management is an important design choice.

These notes were created under traefik v3.5, helmfile v1.1, kubernetes v1.34 and minikube v1.37

localhost DNS and different operating systems

I created this guide on macOS, where any name ending in .localhost is resolved to 127.0.0.1 automatically. Other operating systems do not always handle .localhost the same way. If you have trouble reaching the HTTP endpoints with the .localhost names, continue through the examples as written. Once you complete section "3 Create the DNS records automatically", your DNS setup will take over and the example will work the same on all platforms.

Changes to Expect with the Gateway API

The Gateway API introduces a clearer and more structured method for handling traffic inside Kubernetes. Instead of placing every routing detail inside a single resource, the Gateway API separates the responsibilities into Gateways which control how traffic enters the cluster and Routes which control where traffic goes. This makes traffic rules easier to understand, easier to secure, and easier to expand as a project grows in complexity.

Giddy Up with Traefik

  1. Helmfile configuration and installation
    Traefik service needs to be configured for how you want to use it.

  2. Auto discovery of services and routes
    How services are discovered and routes are created.

  3. Create the DNS records

  4. TLS termination and management
    Making it work with Cert-Manager.

1. Helmfile configuration and installation

Update your repo configurations to include Traefik's Helm chart repository. 

# helmfile.yaml

repositories:
  - name: traefik
    url: https://traefik.github.io/charts

Include the Traefik release in your Helmfile releases section

# helmfile.yaml

releases:

  - name: traefik
    namespace: networking
    chart: traefik/traefik
    version: "~37.0.0"
    values:
      - charts/traefik/values.yaml.gotmpl

This configuration file charts/traefik/values.yaml.gotmpl is the shared values file for Traefik.

Here, we enable the Kubernetes Gateway provider to allow Traefik to work with the Gateway API.

kubernetes Gateway API

The kubernetesGateway provider is the part of Traefik that reads Gateway and Route resources from the cluster and turns them into real traffic configuration. When this provider is enabled, Traefik listens for Gateway API objects and uses them to decide how requests enter the cluster and which services they reach.

# charts/traefik/values.yaml.gotmpl

providers:
  kubernetesGateway:
    enabled: true

Define the Gateway listeners to accept HTTP traffic from all namespaces.

Note: We are going to start with HTTP only then add HTTPS later. 

# charts/traefik/values.yaml.gotmpl

gateway:
  listeners:
    web:
      port: 8000
      protocol: HTTP
      hostname: "*.localhost"
      namespacePolicy:
        from: All

Set the service type to LoadBalancer to allow external access to Traefik.

# charts/traefik/values.yaml.gotmpl
service:
  type: LoadBalancer

Configuration for the dashboard to monitor Traefik's status. You should not enable this in production without proper security measures. Later if we'll use the env specific values files we can enable it for local development only.

## charts/traefik/values.yaml.gotmpl
ingressRoute:
  dashboard:
    enabled: true
    matchRule: Host(`traefik.localhost`)
    entryPoints:
      - web

be sure to update your dependencies and sync the Helmfile to install Traefik

helmfile deps
helmfile -e dev sync

When a service in Kubernetes is marked as a LoadBalancer, the cluster expects a cloud provider to assign a real external IP address. Minikube does not have a cloud provider, so nothing assigns that address until you run the minikube tunnel command. The tunnel acts as a small network bridge on your machine. It listens for LoadBalancer services and gives them a working external IP so you can reach them from your browser. It will also keep the IP active for as long as the tunnel is running.

sudo minikube tunnel

🥳You should be able to access the Traefik dashboard at http://traefik.localhost/dashboard/

2. Auto discovery of services and routes

Choose your targe service and create Gateway and HTTPRoute resources to expose it via Traefik.

Demo Service Step

Here is a simple service to demonstrate how Traefik discovers a Service and routes traffic to it. Instead of creating a full Deployment in Kubernetes, we can use any HTTP service that runs on your local machine. This keeps the example simple and avoids introducing details that are not part of the routing demo.

Example Application

We only use this pattern in the guide to keep the demo simple.

If you already have a service you want to expose via Traefik, you can skip this demo service step and create the HTTPRoute resource for your existing service instead (assuming you have the service setup). See HTTPRoute below for reference.

You can start a simple http server like this:

python3 -m http.server 8080 --bind 0.0.0.0

This gives us a predictable HTTP endpoint on port 8080. The Service and EndpointSlice below let Kubernetes treat that demo process as if it were a normal in cluster Service, which is useful for showing the Gateway and HTTPRoute flow without creating any extra workload objects.

Update your Helmfile to include the new service chart

 # helmfile.yaml
 - name: app-example
   namespace: default
   chart: charts/app-example

The chart.yaml file 

# charts/app-example/Chart.yaml
apiVersion: v2
name: app-example
description: Example App
type: application
version: 0.1.0
appVersion: "1.0.0"

You must replace 192.168.2.18 with your host network facing IP and the port 8080 with the port you used to start the demo server.

# charts/app-example/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: app-example
  namespace: default
  labels:
    app: app-example
spec:
  type: ClusterIP
  clusterIP: None
  ports:
    - name: http
      port: 80
      targetPort: 8080
      protocol: TCP

---

apiVersion: v1
kind: Endpoints
metadata:
  name: app-example
  namespace: default
subsets:
  - addresses:
      - ip: 192.168.2.18 # <----- Replace with your network ip (NOT MINIKUBE IP) where you access the demo server
    ports:
      - name: http
        port: 8080 #  <-----  the port of the demo server
        protocol: TCP

HTTPRoute

let's create the HTTPRoute (charts/app-example/templates/httproute.yaml) resources to expose the example service via Traefik.

# charts/app-example/templates/httproute.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: app-example
  namespace: default
spec:
  parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: traefik-gateway
      namespace: networking
  hostnames:
    - app.localhost
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: app-example
          port: 80

be sure to updaet your dependencies and sync the helmfile 

helmfile deps
helmfile -e dev sync

🥳 now you can access the example app at http://app.localhost

3. Create the DNS records

For this demo we will use a real DNS name: dev.example.com. This domain is used only as an example — you should replace it with your own domain when applying this setup in practice.

Aside on DNS Providers for Cert-Manager

In the next step we will configure Cert-Manager to generate TLS certificates using DNS-based validation.
For this to work, your DNS provider must support an API that Cert-Manager can interact with.
Cert-Manager supports many providers — including Route53, Cloudflare, Google Cloud DNS, Azure DNS, DigitalOcean, and others.

You can find the full list here:
https://cert-manager.io/docs/configuration/acme/dns01/#supported-dns01-providers

Before continuing, confirm that your domain is hosted on a provider from that list, or you will not be able to use DNS01 validation.

Create the following DNS records with your DNS provider:

dev.example.com.  IN  A   127.0.0.1
*.dev.example.com.  IN  CNAME   dev.example.com.

These two records make every hostname under dev.example.com resolve to your local machine. This is ideal for Minikube development because all Gateway traffic ultimately ends up on the machine running the minikube tunnel.

Next, update the Traefik Gateway listeners to accept requests for this new wildcard domain. In charts/traefik/values.yaml.gotmpl, add an additional listener:

# charts/traefik/values.yaml.gotmpl
    http-fqdn:
      port: 8000
      protocol: HTTP
      hostname: "*.dev.example.com"
      namespacePolicy:
        from: All

Finally, update the HTTPRoute for the example service so it responds on both hostnames:

# charts/app-example/templates/httproute.yaml
  hostnames:
    - app.localhost
    - app.dev.example.com

🥳 You can now reach the example application at http://app.dev.example.com (via your own wildcard domain)

4. TLS termination and management

Let us start with configuring Cert-Manager and bringing it into the helmfile configurations

Add the jetstack.io to the helmfile repositories section

Security Reminder

This is an example configuration for demonstration purposes. Always follow best practices for managing sensitive information in you environments. Use tools like SOPS to encrypt secrets

# helmfile.yaml
repositories:
  - name: jetstack
    url: https://charts.jetstack.io

Then update thereleases section to include cert-manager and certmanager-issuers sections

# helmfile.yaml

  - name: cert-manager
    namespace: cert-manager
    chart: jetstack/cert-manager
    version: "~1.19.0"
    set:
      - name: crds.enabled
        value: true
    values:
      # if the dns or the challenge isn't resolving, this helps
      - extraArgs:
          - --dns01-recursive-nameservers=8.8.8.8:53,1.1.1.1:53
          - --dns01-recursive-nameservers-only

  - name: certmanager-issuers
    namespace: cert-manager
    chart: ./charts/certmanager-issuers
    needs:
      - "cert-manager"
    values:
      - charts/certmanager-issuers/values.yaml
      - createSecret: false
        createIssuers: true

This configuration file charts/certmanager-issuers/values.yaml is the shared values file for cert-manager issuers.

here we use route53 as the dns provider for the dns-01 challenge. if you have other dns provider please refer to the cert-manager documentation for the correct configuration.

This example uses example.com as the domain, please replace it with your own domain.

# charts/certmanager-issuers/values.yaml
issuers:
  staging:
    name: letsencrypt-staging
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    accountSecret: acme-account-staging
  prod:
    name: letsencrypt-prod
    server: https://acme-v02.api.letsencrypt.org/directory
    accountSecret: acme-account-prod

# these env settings could be moved to the env specific values files
env:
  domain: dev.example.com
  email: noel@rescommunes.ca
  certificate:
    tlsWildCardSecretName: dev-wildcard-tls
    clusterIssuer: letsencrypt-staging

  dns:
    # these are route53  specific values 
    secretName: dns-credentials
    region: your-dns-region
    zoneID: your-dns-zone-id
    accessKeyID: your-dns-access-key

  # !!! be sure to manage your secret properly
  # ie: don't commit unencrypted 
  secrets:
    dnsAccessKey: some-dnsAccessKey

Lets setup the issuers and related resources in cert-manager. Notice that above that we're using letsencrypt-staging for testing purposes. Use letsencrypt-prod for signed certs that work with most browsers.

# charts/certmanager-issuers/templates/issuers.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: {{ .Values.issuers.staging.name }}
spec:
  acme:
    email: {{ .Values.env.email | quote }}
    server: {{ .Values.issuers.staging.server | quote }}
    privateKeySecretRef:
      name: {{ .Values.issuers.staging.accountSecret }}
    solvers:
      - dns01:
          route53:
            hostedZoneID: {{ .Values.env.dns.zoneID | quote }}
            region: {{ .Values.env.dns.region | quote }}
            accessKeyIDSecretRef:
              name: {{ .Values.env.dns.secretName }}
              key: access-key-id
            secretAccessKeySecretRef:
              name: {{ .Values.env.dns.secretName }}
              key: secret-access-key
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: {{ .Values.issuers.prod.name }}
spec:
  acme:
    email: {{ .Values.env.email | quote }}
    server: {{ .Values.issuers.prod.server | quote }}
    privateKeySecretRef:
      name: {{ .Values.issuers.prod.accountSecret }}
    solvers:
      - dns01:
          route53:
            hostedZoneID: {{ .Values.env.dns.zoneID | quote }}
            region: {{ .Values.env.dns.region | quote }}
            accessKeyIDSecretRef:
              name: {{ .Values.env.dns.secretName }}
              key: access-key-id
            secretAccessKeySecretRef:
              name: {{ .Values.env.dns.secretName }}
              key: secret-access-key

We need to store the dns credentials in a secret that cert-manager can use.

# charts/certmanager-issuers/templates/dns-creds-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: {{  .Values.env.dns.secretName | quote }}
type: Opaque
stringData:
  access-key-id:     {{ .Values.env.dns.accessKeyID | quote }}
  secret-access-key: {{ .Values.env.secrets.dnsAccessKey | quote }}

finally, we need to create the certificate resource that cert-manager will use to request the wildcard certificate for our domain (dev.example.com and *.dev.example.com).

# charts/certmanager-issuers/templates/certificate.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  # notice this is deployed to the 'networking' namespace
  namespace: networking
  name: "{{ .Values.env.certificate.tlsWildCardSecretName }}"
spec:
  secretName: "{{ .Values.env.certificate.tlsWildCardSecretName }}"
  issuerRef:
    name: "{{ .Values.env.certificate.clusterIssuer }}"
    kind: ClusterIssuer
  dnsNames:
    - "{{ .Values.env.domain }}"
    - "*.{{ .Values.env.domain }}"
  privateKey:
    rotationPolicy: Always
    algorithm: RSA
    size: 2048

be sure to updaet your dependencies and sync the helmfile 

helmfile deps
helmfile -e dev sync

🥳 now should have a working TLS certificate for your domain. * check kubectl get certificates -A for the ready status it shouldn't take more than 5 mins

5. Update Traefik to use TLS

Finally, we can configure the Traefik Gateway listener for HTTPS to use the certificate we just created.

# charts/traefik/values.yaml.gotmpl

    websecure:
      port: 8443
      protocol: HTTPS
      hostname: "*.dev.example.com"
      mode: Terminate
      certificateRefs:
        - kind: Secret
          name: {{ .Values.env.certificate.tlsWildCardSecretName | quote }}
          group: ""
      namespacePolicy:
        from: All

be sure to updaet your dependencies and sync the helmfile 

helmfile -e dev sync

🥳now you should be able to access the example app securely at https://app.dev.example.com with the staging TLS certificate issued by Let's Encrypt.

Odds and Ends

Https Redirect force-ssl-redirect

Put the following in charts/traefik/templates/

# charts/traefik/templates/Chart.yaml
apiVersion: v2
name: traefik-routes
type: application
version: 0.1.0
appVersion: "1.0.0"

# charts/traefik/templates/httproute.yaml

# this is the default https redirect
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: redirect-http-to-https
  namespace: networking
spec:
  parentRefs:
    - name: traefik-gateway
      namespace: networking
      sectionName: http-fqdn
  hostnames:
    - "{{ .Values.env.domain }}"
    - "*.{{ .Values.env.domain }}"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      filters:
        - type: RequestRedirect
          requestRedirect:
            scheme: https
            statusCode: 301

then update the helm file to include the new chart

# helmfile.yaml
  - name: traefik-routes
    namespace: networking
    chart: ./charts/traefik
    values:
      - env/{{ .Environment.Name }}/values.yaml

change the example app route to use https entrypoint

# charts/app-example/templates/httproute.yaml

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: app-example
spec:
  parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: traefik-gateway
      namespace: networking
      sectionName: websecure
  hostnames:
    - "app.{{ .Values.env.domain }}"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: app-example
          port: 80

be sure to update your dependencies and sync the helmfile 

helmfile -e dev sync

🥳 now you should be redirected from http://app.dev.example.com to https://app.dev.example.com

To do

  • [ ] maintenance mode for traefik?
  • [ ] custom 404 page / maintenance page
  • [ ] rate limiting
  • [ ] WAF
  • [ ] other security stuff (geo,etc)
  • [ ] Add observability
  • [ ] load balancing strategies
  • [ ] Good defaults for logs, Prometheus, and dashboards.