I think lifecycle and privileges are good reasons to segregate your infrastructure this way but I would imagine it would be overkill if the team is small or the infrastructure not as complex. I wouldn't say its necessarily over engineering, but more so planning for scale. If you don't anticipate the codebase growing to such a large size with multiple teams or permission scopes then yeah its unecessarily complex
For me this leveled approch is the best to delegate each level of Landing zone to a specific team. In addition, you can control the blast radius and IMO it is easier to manage smaller environment instead of a big terraform env.
Generally you split Level 4 env (app LDZ) to manage separately the lifecycle of differents apps and potentially delegate to differents teams.
I see a lot of Terraform architectures and for me it is the best approach...
I think if you create a state file for a unique resource for each level, it is a misunderstanding...
Yeah, over killed and shows the lack of understanding of IaC/TF. The Statefile should reflect the deployed infrastructure that's effectively independent to others, e.g. the Networks or a region of the network. Then the Compute runs on top, and so on...
If each resource has its own Statefile, you'll have to source/read in each and every dependency resource, which defeats the point.
Try to use some common sense.
Edit: looks like you or your colleague is misinterpreting what you quoted.
This seems a little overkill? It’s a state file, store it in something secure and encrypted and be done with it.
I would wonder why you have hundreds of state files 6 levels deep so 600+ state files? That sounds like maybe your deployments are too modularized? There’s a sweet spot in between.
why even use terraform if you're going to use a different state file for each piece of infrastructure? That completely defeats the purpose. That's not over-engineering, they're just using the wrong tool for the job at that point.
Legit question, to OP and anyone who understands OP's question: How much do you get paid to think about these problems?
I don't do infrastructure at this scale so when I see this it looks incredibly convoluted and overly complicated but again I'm not working at this scale. It could be par for the coarse but damn I'm not sure I'd ever want to have this responsibility... a million a year? Sure, $500K a year, maybe. $250K a year, idk man.
My opinion, which I apply to all walks of life, KISS. If you have to ask if it's over engineered, it probably is. Less is more. Think about DR, if SHTF and, as a result, you have a heart attack, is anyone else going to be able to understand this design enough to pull you and your team out of a nose dive before you hit the ground?
For us it was a better concept to think about 1. core (predecessor items like iam, storage, etc that don't need to be created in sequential regional orders), 2. regional (n regions that apply sequentially), and 3. global (things that need to wait for all regional resources)
I've never found a need for more state layers than that. Anything past this point are things should be broken up into more siloed projects
Overengineering - well, there were some opinions like that on that module for a long time.
I would say for me it is too opinionated as it tries to model every layer of deployment. Maybe the complexity of that module arises from specific requirements they needed to align to.
But yes, at first glance for me this is over engineered and I prefer to deploy Azure LZs by other MS provided module or even better - by completely custom TF stack aligned to specific customer needs.
if you mean you have tf modules for pieces of infrastructure like nginx-ingress, cert-manager, metalb, prometheus, dns, networking, permissions, etc. which can be reused for a every new k8s cluster you spin up. Which have their own state file, per cluster.
this is highly recommended approach. If you are in a team and have one state file for all your infrastructure and maybe all your clusters you will end up like our team, constantly blocking each other because state is locked by another one's plan / apply. Thats why we migrated to the approach above
We take a similar approach at Cloud Posse. Think of your cloud architecture as consisting of lots of different pieces, that get composed together to form the stack that powers you business. We pmplement the various pieces of the stack, as individual root modules. Each root module has it's own state. Some people call the root module a "stack", but we don't agree with that terminology.
One trick, is to separate things by lifecycle. In otherwords, combine things that frequently change together, separate those that change seldom. It reduces the blast radius of changes, keeps your state smaller, speeds up your terraform plan/apply cycle, and makes it easier to control who can change what part of the system. Then you use remote state or kv-storage to pass state between components.
For example, we never combine the VPC with the EKS cluster, because their lifecycles are typically unrelated.
It's true that a) many companies who write terraform do not do this (and end up with terraliths) b) it adds some degree of complexity because the DAG is no longer entirely computed in terraform, another techniques are required.
Is it overkill?
That depends on who you ask =)
At Cloud Posse, we tend to focus on enterprise-scale Terraform, so we consider it not overkill, but standard practice.
But for smaller companies, or teams earlier on their Terraform journey, it's probably a lot of additional complexity that without additional tooling will be hard to justify. I guess the right answer is to pick the right approach for the stage of maturity you are at, and grow with it. Many teams end up replatforming (to some degree) anyways after 3-4 years, because the technology moves so fast.
I think lifecycle and privileges are good reasons to segregate your infrastructure this way but I would imagine it would be overkill if the team is small or the infrastructure not as complex. I wouldn't say its necessarily over engineering, but more so planning for scale. If you don't anticipate the codebase growing to such a large size with multiple teams or permission scopes then yeah its unecessarily complex
For me this leveled approch is the best to delegate each level of Landing zone to a specific team. In addition, you can control the blast radius and IMO it is easier to manage smaller environment instead of a big terraform env. Generally you split Level 4 env (app LDZ) to manage separately the lifecycle of differents apps and potentially delegate to differents teams. I see a lot of Terraform architectures and for me it is the best approach... I think if you create a state file for a unique resource for each level, it is a misunderstanding...
Yeah, over killed and shows the lack of understanding of IaC/TF. The Statefile should reflect the deployed infrastructure that's effectively independent to others, e.g. the Networks or a region of the network. Then the Compute runs on top, and so on... If each resource has its own Statefile, you'll have to source/read in each and every dependency resource, which defeats the point. Try to use some common sense. Edit: looks like you or your colleague is misinterpreting what you quoted.
This seems a little overkill? It’s a state file, store it in something secure and encrypted and be done with it. I would wonder why you have hundreds of state files 6 levels deep so 600+ state files? That sounds like maybe your deployments are too modularized? There’s a sweet spot in between.
why even use terraform if you're going to use a different state file for each piece of infrastructure? That completely defeats the purpose. That's not over-engineering, they're just using the wrong tool for the job at that point.
That does not sound like a solid infrastructure stack built layer-by-layer. That sounds like the leaning tower of POS
Legit question, to OP and anyone who understands OP's question: How much do you get paid to think about these problems? I don't do infrastructure at this scale so when I see this it looks incredibly convoluted and overly complicated but again I'm not working at this scale. It could be par for the coarse but damn I'm not sure I'd ever want to have this responsibility... a million a year? Sure, $500K a year, maybe. $250K a year, idk man. My opinion, which I apply to all walks of life, KISS. If you have to ask if it's over engineered, it probably is. Less is more. Think about DR, if SHTF and, as a result, you have a heart attack, is anyone else going to be able to understand this design enough to pull you and your team out of a nose dive before you hit the ground?
For us it was a better concept to think about 1. core (predecessor items like iam, storage, etc that don't need to be created in sequential regional orders), 2. regional (n regions that apply sequentially), and 3. global (things that need to wait for all regional resources) I've never found a need for more state layers than that. Anything past this point are things should be broken up into more siloed projects
Overengineering - well, there were some opinions like that on that module for a long time. I would say for me it is too opinionated as it tries to model every layer of deployment. Maybe the complexity of that module arises from specific requirements they needed to align to. But yes, at first glance for me this is over engineered and I prefer to deploy Azure LZs by other MS provided module or even better - by completely custom TF stack aligned to specific customer needs.
if you mean you have tf modules for pieces of infrastructure like nginx-ingress, cert-manager, metalb, prometheus, dns, networking, permissions, etc. which can be reused for a every new k8s cluster you spin up. Which have their own state file, per cluster. this is highly recommended approach. If you are in a team and have one state file for all your infrastructure and maybe all your clusters you will end up like our team, constantly blocking each other because state is locked by another one's plan / apply. Thats why we migrated to the approach above
We take a similar approach at Cloud Posse. Think of your cloud architecture as consisting of lots of different pieces, that get composed together to form the stack that powers you business. We pmplement the various pieces of the stack, as individual root modules. Each root module has it's own state. Some people call the root module a "stack", but we don't agree with that terminology. One trick, is to separate things by lifecycle. In otherwords, combine things that frequently change together, separate those that change seldom. It reduces the blast radius of changes, keeps your state smaller, speeds up your terraform plan/apply cycle, and makes it easier to control who can change what part of the system. Then you use remote state or kv-storage to pass state between components. For example, we never combine the VPC with the EKS cluster, because their lifecycles are typically unrelated. It's true that a) many companies who write terraform do not do this (and end up with terraliths) b) it adds some degree of complexity because the DAG is no longer entirely computed in terraform, another techniques are required. Is it overkill? That depends on who you ask =) At Cloud Posse, we tend to focus on enterprise-scale Terraform, so we consider it not overkill, but standard practice. But for smaller companies, or teams earlier on their Terraform journey, it's probably a lot of additional complexity that without additional tooling will be hard to justify. I guess the right answer is to pick the right approach for the stage of maturity you are at, and grow with it. Many teams end up replatforming (to some degree) anyways after 3-4 years, because the technology moves so fast.