Arbitrary k8s CRD Manifests in Terraform

When going about installing something like cert-manager or metallb in a kubernetes cluster, it is a very good idea to use terraform or opentofu so you can version control these critical aspects of the cluster. However, when doing so, you will eventually need to create resources that have no terraform-native resource type. Usually this is the case with CRDs that come along with helm packages. MetalLB is one such package. The following installs metallb using terraform/opentofu:

resource "helm_release" "metallb" {
  name       = "metallb"
  repository = "https://metallb.github.io/metallb"
  chart      = "metallb"
  version    = "0.14.8"
}

However, to actually start using metallb, we still need to create an IPAddressPool and an L2Advertisement. There is no IPAddressPool resource type in terraform, but luckily we can create arbitrary manifests using the kubernetes_manifest module:

resource "kubernetes_manifest" "ipaddresspool" {
  depends_on = [helm_release.metallb]
  manifest = {
    "apiVersion" = "metallb.io/v1beta1"
    "kind"       = "IPAddressPool"
    "metadata" = {
      "name"      = "default"
      "namespace" = "default"
    }
    "spec" = {
      "addresses" = [
        "172.30.190.32-172.30.190.64",
      ]
    }
  }
}

resource "kubernetes_manifest" "l2advertisement" {
  depends_on = [kubernetes_manifest.ipaddresspool]
  manifest = {
    "apiVersion" = "metallb.io/v1beta1"
    "kind"       = "L2Advertisement"
    "metadata" = {
      "name"      = "default"
      "namespace" = "default"
    }
  }
}

Now we’re able to version control 100% of the metallb installation; and importantly; forget about it and come back a year later to update/modify it without worrying about forgotten zombie manifests that turn out to be critical.

Nathan Hensel

on caving, mountaineering, networking, computing, electronics


2024-10-20