This is an example of how to set up cloudflare-tunnel + ingress-nginx for bombproof (within reason) internet ingress on kubernetes.

Cloudflare-tunnel works by reaching out from your network to cloudflare’s cdn, like a reverse ssh tunnel. In turn, you can run it behind nat, firewalls, dynamic addresses, etc, and you benefit from cloudflare’s robust internet edge - as opposed to needing to run your own HA bgp borders.

Heres a terraform definition of the cloudflare-tunnel helm chart:

resource "helm_release" "cloudflare-tunnel" {
  name       = "cloudflare-tunnel"
  namespace  = "default"
  repository = "https://cloudflare.github.io/helm-charts"
  chart      = "cloudflare-tunnel"
  version    = "0.3.2"
  values = [<<EOT
cloudflare:
  account: asdf
  tunnelName: k8s
  tunnelId: asdf
  secret: 1234
  ingress:
    - hostname: "*.nih.earth"
      service: https://wan-ingress-nginx-controller.default.svc.cluster.local:443
      originRequest:
        noTLSVerify: true

replicaCount: 2

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
            - key: app.kubernetes.io/name
              operator: In
              values:
                - cloudflare-tunnel
        topologyKey: "kubernetes.io/hostname"
EOT
  ]
}

The ingress section behaves like a kubernetes ingress. We can point hostnames to individual services here if we wanted, but we already have ingress-nginx for that, so this is the simplest way to just point everything at ingress-nginx. By using the internal service name here, we are not relying on a LoadBalancer. I quite like this as its one less thing to break. This name resolves to the ClusterIP.

And heres the associated ingress controller:

resource "helm_release" "wan-ingress-nginx" {
  name       = "wan"
  namespace  = "default"
  repository = "https://kubernetes.github.io/ingress-nginx"
  chart      = "ingress-nginx"
  version    = "4.12.1"
  values = [<<EOF
controller:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: app.kubernetes.io/name
                operator: In
                values:
                  - ingress-nginx
          topologyKey: "kubernetes.io/hostname"
  replicaCount: 2
  service:
    type: ClusterIP
EOF
  ]
}

External to all of this, I have wildcard DNS for the domain pointed via CNAME at the cfargotunnel.com path that cloudflare gives you when set up the tunnel in “Zero Trust -> Networks -> Tunnels”. I dont remember what all this part of the setup entailed; I think you pretty much click through the “Create a tunnel” wizard and it ends up giving you the credentials for the new connection.

And with that we have two HA ingress pods, two HA tunnel pods, and wildcard DNS that lets us deploy new subdomain paths with absolutely zero overhead. As I’m just running stateless http services on this cluster, we can easily do rolling reboots and upgrades of all this with effectively no downtime.

~ > kubectl get pods -o wide | egrep 'cloudflare|ingress'
cloudflare-tunnel-7f6b54c898-49rgn                  1/1     Running   0            9h      10.42.3.141   k8s-1c7a   <none>           <none>
cloudflare-tunnel-7f6b54c898-vtclx                  1/1     Running   0            9h      10.42.2.248   k8s-8ae4   <none>           <none>
wan-ingress-nginx-controller-6d5bf67796-6nmrb       1/1     Running   0            9h      10.42.3.139   k8s-1c7a   <none>           <none>
wan-ingress-nginx-controller-6d5bf67796-z8hn6       1/1     Running   0            9h      10.42.2.252   k8s-8ae4   <none>           <none>