# Keeping it stupidly standardized

How do you manage 20+ kubernetes clusters, 10+ AWS accounts, CI/CD for 1000+ git repositories, for 150+ developers?

Easy - Don't.

It's much more sustainable to manage one kubernetes cluster, two or three AWS accounts, and maybe five or six CI/CD pipelines. But going from one to 20 without losing anything isn't trivial. The automation is easy enough; a set of OpenTofu modules, automated pipelines with state changes across runs and boom - as many clusters as you can come up with names for, and only one thing to manage. But as with anything there **will** be edge cases, and each team of developers will have their own needs and wants. If this is you, or something you're looking at starting, ensuring that those edge cases don't turn into manual interventions is the hard part.

Some of these can be handled easily enough; a set of variables to change security group CIDRs and swapping environment variables to change an AWS account are trivial. But what happens when a user says that their new application uses UDP and every entrypoint to your cluster is an ALB? Or that disk IO is causing huge latency spikes, can we swap to high throughput NVME storage for just this one app on this one cluster? It's easy to spin up a new cluster for a new application that needs to be isolated for compliance purposes, but if each cluster has a fixed cost of ~$100 pretty soon that AWS bill will be in the clouds too.

Identifying these use cases early on makes things a bit easier, but adapting to the business's needs is absolutely critical. You can't just say 'No' to everything that deviates from the decisions you made 5 years ago when you created the base infrastructure; that's a great way to lose out on major competitive advantages or to anger a developer who sees a huge time saver in a new technology. See also: AI acceleration. The core part to recognize is where changes can be made, and when laying the foundation, do it flexibly. Create a set bedrock then allow for building on top of it. 

On the thought of UDP support for an ALB-based cluster: Let's just swap that one cluster's ALB for an NLB; it's just an annotation on the service, a one line change. Easy, done. This seems incredibly obvious right? 

But let's think deeper about what this mental model gets you: If you replace the ALB, you'll need to do a cutover and/or deal with an outage, you lose the benefits of an ALB over NLB like optional path-based routing, and now the entire development team needs to swap their thinking from layer 7 to layer 4. The path-based routing alone is likely to break a lot of your applications unless each one had its own hostname. And if you do it for just the one cluster to minimize the blast radius, now you'll need to remember that until someone accidentally causes an outage because they moved their application from an ALB cluster to the NLB cluster. If you instead decide to do it for all 20+ clusters; you'll be in migration hell for months, and there'll probably still be one or two that can't even be migrated.

A new load balancer is ~$15 a month. If you instead keep the existing ALBs, ensure that they're the default, but add a flag to optionally create a new NLB, this opens a whole new set of doors. By adding to the existing foundation rather than ripping it up, the cost is minimal and lets you keep the last 5 years of work in place. Only the UDP-enabled applications need to differ, and just by changing their (legacy) ingress class or gateway in the manifests. 

One key bit of caution though: Before you go and add a new X for every Y request, it's always best practice to see if a small tweak can fit into your existing configurations. Adding a new flag for every request is how you get a few hundred knobs and now your job is remembering what they all are and how they all affect everything else. You need to ask yourself: If you add this change, how can I do it in such a way that if I need to do it again - and that *is* something that will happen - how do I do it in such a way that it's sustainable? For example, does this new application *actually* need a new, isolated cluster, or could we create a new NLB for TLS passthrough and set up appropriate NetworkPolicies to get the same level of isolation? Is the disk thrashing something that will *actually* be fixed by swapping to NVME storage, or can we bump the RAM a bit and use an in-memory store? If that's not feasible, do we swap the entire cluster's node types over to NVME-enabled instances - one variable you probably already have, but a high cost; or do we create a whole new nodegroup and use a NodeSelector? What about the next guy who needs accelerator-enabled instances? If over a hundred people depend on you, you'll probably have 200 different use cases.

The same applies for your git repositories and CI/CD. Most companies already try to standardize on a few set of core technologies; if you're a C# shop you know it, and probably don't want someone creating a new webapp in C++. Being able to identify the core similarities between these is what separates a senior-level from staff-level platform engineer; and what made it entirely feasible to manage that many repositories with over a hundred people committing per day. It's a core benefit of GitLab's CI/CD as an example. Their components allow you to build different blocks that you can swap in and out, assuming you keep to a few known stages and variables in sync. This allows you to build a catalog of "Deploying to Kubernetes? Use this! Deploying to CloudFront? This one instead!". And one small set of composable blocks is easily manageable; 1000+ bespoke pipelines are much less so. This also fits incredibly well into the developer mindset of composition over inheritance, which the previous 'include/inherit' keywords didn't really fit into. This means your development team can more easily understand it, cutting down on the back and forth questions. 

By having a set of standardized templates, you can ensure they'll work well with that set of standardized Kubernetes clusters inside that set of standardized AWS accounts. Plus, if a team wants, say, a custom Slack message at the end of their pipeline - they can do it just in their repo, allowing for the flexibility that they need.

Let's think about deployments too - before you let the team use Jim in accounting's spare laptop to deploy their freshly built container, let's provide them with the right tool. At first glance, if you have a container and just need a deployment, service, and HTTPRoute, a company helm chart sounds like the perfect plan. Until your payments team needs a custom HPA config because their app scales differently. And then the marketing-focused team needs a second port opened for Prometheus, and the search team needs a sidecar, and the analytics team needs Redis deployed alongside each of their applications for reasons? And those asks will just keep coming as the needs of the business grow. Now you don't just have a helm chart, you've just added a new language that "compiles" into Kubernetes manifests.

Obviously we need something better. We can shoehorn raw manifests into helm - a first thought is to just add a post-helm kubectl apply step in your deploy job. This *technically* works, but is messy and adds a drift problem if someone needs to override the chart itself, and binds each repo's customizations to that helm chart - the smallest change could be breaking. But maybe a second tool can handle what doesn't fit nicely into the company's standard helm chart?

Let's consider adding something like Flux to our repertoire, which decouples the repositories, automatically handles drift and grants full control over the manifests. We can reduce the lift of that though by making it easy to follow what the helm chart is doing. Create a little 'Flux Onboarding' script or Backstage job that pulls from your company's helm chart, then let the teams who need that flexibility make the changes they need to. This'll minimize the cost, and since everyone is starting from that same place, you can rely on people not changing things that they don't need to. 

One thing worth noting about this: Everything mentioned here is open-sourced, self-hostable, or has a drop-in replacement with another cloud provider. It's always worth considering when adding another tool: "What happens if the maintainer/provider wins the lottery?"

Back to deployments: your development teams have enough on their plate with fixing bugs and shipping features.They probably don't want to have to debug and see why whenever they deploy their replica count goes to 1 before shooting back up to 15 (hint: Did you forget to unset the replica count in your deployment?); and for those teams helm works perfectly. But that team of overachievers who want to be able to tweak every small YAML configuration, would be *incredibly* frustrated if all they had was a single helm chart to deploy with, and that's where something like Flux shines. Don't shy away from additional tooling, but just like in this case, make sure it's actually solving a real problem that can't fit into your existing setup.

And that's really the core of it - standardize where you can, be conscious of the tradeoffs where you can't. Focus on keeping that standardization and reducing sprawl to keep the workload reasonable, but allow for the flexibility where it's needed so the business can adapt and grow without needing a team of 15 to support a company of 150.

Interested in how I can help consolidate and make your engineers' lives easier? Let's chat - I love finding these sorts of efficiencies wherever I can.

Oh by the way, did this seem like a lot of dependencies? I'll be doing a deep dive into why that's actually a *good* thing, even with the AI revolution where code is cheap.