diff options
Diffstat (limited to 'content/blog/terraform')
-rw-r--r-- | content/blog/terraform/input_validation.md | 122 | ||||
-rw-r--r-- | content/blog/terraform/tofu_for_each_providers.md | 112 |
2 files changed, 234 insertions, 0 deletions
diff --git a/content/blog/terraform/input_validation.md b/content/blog/terraform/input_validation.md new file mode 100644 index 0000000..b352304 --- /dev/null +++ b/content/blog/terraform/input_validation.md @@ -0,0 +1,122 @@ +--- +title: 'Validating JSON or YAML input files with terraform' +description: 'a much anticipated feature' +date: '2025-02-11' +tags: +- OpenTofu +- Terraform +--- + +## Introduction + +I am used to building small abstraction layers over some OpenTofu/Terraform code +via YAML input files. It would be too big an ask to require people (usually +developers) unfamiliar with infrastructure automation to understand the +intricacies of HCL, but filling up YAML (or JSON) files is no problem at all. + +In this article I will explain how I perform some measure of validation on these +input files, as well as handle default values. + +## Input file validation + +I am using two nested modules to abstract this validation away. I name the top +module `input` and its job is to read and decode the input files, then call the +nested `validation` module with them. + +### Input module + +A simplified version of this `input` module contains the following: + +``` hcl +output "data" { + description = "The output of the validation module." + value = module.validation +} + +locals { + input_path = "${path.module}/../../../inputs" +} + +module "validation" { + source = "./validation/" + + teams = yamldecode(file("${local.input_path}/teams.yaml")) + users = yamldecode(file("${local.input_path}/users.yaml")) +} +``` + +There is a single output to expose the validated data. The `input_path` should +obviously point to where your `inputs` data lives. + +### The validation submodule + +The `validation` module does the heavy lifting of validating the input, handling +default values and mangling data in necessary ways. Here is a simplified +example: + +``` hcl +output "aws_iam_users" { + description = "The aws IAM users data." + value = { for user, info in var.users : + user => info if info.admin.aws + } +} + +output "users" { + description = "The users data." + value = var.users +} + +variable "users" { + description = "The yaml decoded contents of the users input file." + nullable = false + type = map(object({ + admin = optional(object({ + aws = optional(bool, false) + github = optional(bool, false) + }), {}) + email = string + github = optional(string, null) + })) + validation { + condition = alltrue([for _, info in var.users : + endswith(info.email, "@adyxax.org") + ]) + error_message = "A user's email must be for the @adyxax.org domain." + } +} +``` + +Here I have two outputs: one that mangles the input data a bit to filter AWS +admin users, and another that simply returns the input data augmented by the +default values. I added a validation block that checks that every users' email +address is on the proper domain. + +### Usage + +Using this input module is as simple as: + +``` hcl +module "input" { + source = "../modules/input/" +} +``` + +With this, you can then do something with `module.input.data.users` or +`module.input.data.aws_iam_users`. A common debugging step can be to run +OpenTofu or Terraform with the `console` command and inspect the resulting input +data. + +## Limitations + +The main limitation of this validation system is that invalid (or misspelled) +keys in the original input file are simply ignored by OpenTofu/Terraform. I did +not find a way around it with just terraform which is frustrating! + +A solution to this particular need that relies on outside tooling is to perform +JSON schema or YAML schema validation. This solves the problem and runs nicely +in a CI environment. + +## Conclusion + +This pattern is really useful, use it without moderation! diff --git a/content/blog/terraform/tofu_for_each_providers.md b/content/blog/terraform/tofu_for_each_providers.md new file mode 100644 index 0000000..4c23f90 --- /dev/null +++ b/content/blog/terraform/tofu_for_each_providers.md @@ -0,0 +1,112 @@ +--- +title: 'Opentofu provider iteration with `for_each`' +description: 'a much anticipated feature' +date: '2025-01-25' +tags: +- AWS +- OpenTofu +--- + +## Introduction + +The latest release of OpenTofu came with a much anticipated feature: provider +iteration with `for_each`! + +My code was already no longer compatible with terraform since OpenTofu added the +much needed variable interpolation in provider blocks feature, so I was more +than ready to take the plunge. + +## Usage + +A good example will be to rewrite the lengthy code from my [Securing AWS default +vpcs]({{< ref "blog/aws/defaults.md" >}}#iterating-through-all-the-default-regions) +article a few months ago. It now looks like: + +``` hcl +locals { + aws_regions = toset([ + "ap-northeast-1", + "ap-northeast-2", + "ap-northeast-3", + "ap-south-1", + "ap-southeast-1", + "ap-southeast-2", + "ca-central-1", + "eu-central-1", + "eu-north-1", + "eu-west-1", + "eu-west-2", + "eu-west-3", + "sa-east-1", + "us-east-1", + "us-east-2", + "us-west-1", + "us-west-2", + ]) +} + +provider "aws" { + alias = "all" + default_tags { tags = { "managed-by" = "tofu" } } + for_each = concat(local.aws_regions) + profile = "common" + region = each.key +} + +module "default" { + for_each = local.aws_regions + providers = { aws = aws.all[each.key] } + source = "../modules/defaults" +} +``` + +Note the use of the `concat()` function in the `for_each` definition of the +providers block. This is needed to silence a warning that tells you it is a bad +idea to iterate through your providers using the same expression in provider +definitions and module definitions. + +Though I understand the reason (to allow for resources destructions when the +list we are iterating on changes), it is not a bother for me in this case. + +## Modules limitations + +The main limitation at the moment is the inability to pass down the whole +`aws.all` to a module. This leads to code that repeats itself a bit, but it is +still better than before. + +For example, when creating resources for multiple aws accounts, a common pattern +is to have your DNS manged in a specific account (for me it is named `core`) +that you need to pass around. Let's say you have another account named `common` +with for example monitoring stuff and here is how some module invocation can +look like: + +``` hcl +module "base" { + providers = { + aws = aws.all["${var.environment}_${var.region}"] + aws.common = aws.all["common_us-east-1"] + aws.core = aws.all["core_us-east-1"] + } + source = "../modules/base" + + ... +} +``` + +It would be nice to be able to just pass down aws.all, but alas we cannot yet. + +## Cardinality limitation + +Just be warned that you cannot go too crazy with this mechanism. I tried to +iterate through a cross-product of all AWS regions and a dozen AWS accounts and +it does not go well: OpenTofu slows down to a crawl and it starts taking a dozen +minutes just to instantiate all providers in a folder, before planning any +resources! + +This is because providers are instantiated as separate processes that OpenTofu +then talks to. This model does not scale that well (and consumes a fair bit of +memory), as least for the time being. + +## Conclusion + +I absolutely love this new feature! |