tags:
Contents
EKS Addon Configuration via Terraform
This is a quick overview of how to pass custom configuration values to EKS managed addons using Terraform.
Configuration Schema
Before you begin find the addon version you need and describe the available configuration options:
❯ aws eks describe-addon-versions --addon-name coredns --kubernetes-version 1.27 | jq -r '.addons[].addonVersions[] | select(.compatibilities[].defaultVersion == true) | .addonVersion'
v1.10.1-eksbuild.1
❯ aws eks describe-addon-configuration --addon-name coredns --addon-version v1.10.1-eksbuild.1 | jq -r .configurationSchema | jq
{
"$ref": "#/definitions/Coredns",
"$schema": "http://json-schema.org/draft-06/schema#",
"definitions": {
"Coredns": {
"additionalProperties": false,
"properties": {
"affinity": {
"default": {
"affinity": {
"nodeAffinity": {
"requiredDuringSchedulingIgnoredDuringExecution": {
"nodeSelectorTerms": [
{
"matchExpressions": [
{
"key": "kubernetes.io/os",
"operator": "In",
"values": [
"linux"
]
},
{
"key": "kubernetes.io/arch",
"operator": "In",
"values": [
"amd64",
"arm64"
]
}
]
}
]
}
},
"podAntiAffinity": {
"preferredDuringSchedulingIgnoredDuringExecution": [
{
"podAffinityTerm": {
"labelSelector": {
"matchExpressions": [
{
"key": "k8s-app",
"operator": "In",
"values": [
"kube-dns"
]
}
]
},
"topologyKey": "kubernetes.io/hostname"
},
"weight": 100
}
]
}
}
},
"description": "Affinity of the coredns pods",
"type": [
"object",
"null"
]
},
"computeType": {
"type": "string"
},
"corefile": {
"description": "Entire corefile contents to use with installation",
"type": "string"
},
"nodeSelector": {
"additionalProperties": {
"type": "string"
},
"type": "object"
},
"replicaCount": {
"type": "integer"
},
"resources": {
"$ref": "#/definitions/Resources"
},
"tolerations": {
"default": [
{
"key": "CriticalAddonsOnly",
"operator": "Exists"
},
{
"key": "node-role.kubernetes.io/master",
"operator": "NoSchedule"
}
],
"description": "Tolerations of the coredns pod",
"items": {
"type": "object"
},
"type": "array"
}
},
"title": "Coredns",
"type": "object"
},
"Limits": {
"additionalProperties": false,
"properties": {
"cpu": {
"type": "string"
},
"memory": {
"type": "string"
}
},
"title": "Limits",
"type": "object"
},
"Resources": {
"additionalProperties": false,
"properties": {
"limits": {
"$ref": "#/definitions/Limits"
},
"requests": {
"$ref": "#/definitions/Limits"
}
},
"title": "Resources",
"type": "object"
}
}
}
Terraform Configuration
The configuration_values must be in json format. The easiest way to do this is to use the jsonencode() function included with terraform.
data "aws_eks_addon_version" "coredns" {
addon_name = "coredns"
kubernetes_version = module.eks.cluster_version
most_recent = false # use the default version for the cluster rather than latest
}
resource "aws_eks_addon" "coredns" {
cluster_name = module.eks.cluster_id
addon_name = "coredns"
addon_version = data.aws_eks_addon_version.coredns.version
resolve_conflicts = "OVERWRITE"
# This requires the consul-dns svc ClusterIP be set to 172.20.0.5.
# https://github.com/hashicorp/consul-k8s/blob/6e9f4731e76642dbb5d0869cdf1aa3fa40e1681a/charts/consul/values.yaml#L1687-L1691
configuration_values = jsonencode({
corefile = <<-EOT
.:53 {
errors
health
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
forward . /etc/resolv.conf
cache 30
loop
reload
loadbalance
}
consul {
errors
cache 30
forward . 172.20.0.5
}
EOT
tolerations = [
{
effect = "NoSchedule"
key = "node-role.kubernetes.io/master"
},
{
key = "CriticalAddonsOnly"
operator = "Exists"
},
{
key = "role",
value = "karpenter"
effect = "NoSchedule"
}
]
})
}
Final Notes
That basically covers it. A couple notes about the example above. The CoreDNS corefile includes a block to forward lookups for hostnames that end in .consul to the DNS service listening on IP address 172.20.0.5. To facilitate this you must set your consul-dns service to use that IP address for the ClusterIP. This also requires specifying the k8s ipv4 service CIDR to something like 172.20.0.0/16.
Some additional information related to this can be found at the links below.