Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Who Should Write the Terraform?”

zwischenzugs.com

Post reply on HN
Print view View original

121–130 of 137 posts

Re: “Who Should Write the Terraform?”

#121

Post by physicsguy »

aranelsurion wrote: post #70
theamk wrote: post #15

Re this segment: > "There were endless complaints about the time taken to get ‘central IT’ to do their bidding, and frequent demands for more autonomy and freedom. A cloud project was initiated by the centralised DBA team to enable that autonomy. [...] Cue howls of despair from the development teams that they need a centralised DBA service" Author makes it sound like users didn't know what they wanted. This is not tr…

> _it was a different set of people_ who were complaining before and after. That's something I've observed as well. Seems to me there are (at least) two developer personas, one kind only wants to deliver their task, specialize in what they do well, and generally can't care less if their DB is oversized or has no maintenance windows set or no recovery plans or who has access to it etc. They usually lack the cloud/plat…

I’d probably fall into the latter category of “complainers”, but I actually care very little about doing infrastructure work even though I’m interested in it, I just want a quick turn around on simple requests.

My current place is just awful. Over complicated architecture born from a platform team that couldn’t be less helpful, so people have worked around it with all sorts of hacks.

Re: “Who Should Write the Terraform?”

#122

Post by Halan »

jprd wrote: post #116
Halan wrote: post #78

Earlier quoted context omitted.

Everything should be in code, I feel for sys admins but they had all the time to become software professionals. No more mercy

Yikes! I presume this self-assured statement comes from ignorance and youth? Can you detail the sysadmins that you've worked with that didn't understand code or the underlying infrastructure they are tasked with supporting better than you or your team?

Sure, it comes from ignorance but none of these people are young, myself included. Possessing domain knowledge cannot be enough anymore on its own as it implies making room for limiting solutions like terraform that only exist to make the life easier for those who call themselves tech professionals without knowing how to write code. Everyone should be a software engineer. Infrastructure should be in code, and follow software development practices. I hope that the CDKs will become the norm. This also means that nobody will need to “support” anybody but rather “work with” a teammate because they would all be software engineers, no more devops, qa, software walls. It is possible, anybody can still be a SME in something while also being a software engineer, people are just lazy because the industry pays you 6 digits even if you only know terraform and aws.

Re: “Who Should Write the Terraform?”

#123

Post by Spasnof »

On "how we got here" you use "bulleted list" rather than "numbered list". This is important as "If you rearrange the items in a bulleted list, the list's meaning does not change. "

Credit to https://developers.google.com/tech-writing/one/lists-and-tab... which pointed this out and has stuck in my craw ever since.

Re: “Who Should Write the Terraform?”

#124

Post by readlikeasloth »

solatic wrote: post #94

ITT people arguing for embedding infrastructure engineers into product teams. Ayyyy, dios mio. a) If you need to embed, then actually, you need to embed InfoSec, UX, IT, Customer Success, Product, Compliance, etc. etc. for exactly the same reasons. In today's labor-constrained economy, good luck finding qualified people for every role on every team! And if one of them leaves, who ensured that they documented everythi…

> If you're 100% serverless then this doesn't really apply, but if you're spinning up eight different Kubernetes clusters for eight different teams then you probably need to collaborate a bit better.

This is exactly the situation I´m currently in. Company decided to migrate from big on-prem kubernetes to AWS. Now every team got their own account and well... good luck, you´re on your own now. We´re a small team of three developers. Although we have three certifications under our belts (AWS Dev, CKA, CKAD) it took us almost three months to configure AWS and set up the Terraform pipeline and define processes like "upgrading cluster". The "enabling" part was basically missing in the whole cloud strategy of the company. It was more like: good luck, you´re on your own now.

In fact we made contact with a neighboring team. Only to find out that their use case was so different from ours that collaboration didn´t make any sense. For them Kubernetes was not a good fit, for us it was the way to go.

Speaking of sharing a cluster or AWS ressources: we figured out that it is not allowed due to billing reasons. Company policy is: One product per AWS account.

If you ask me I see a shift of paradigms happening here. Now you hear a lot about "enabling teams" instead a dedicated team for infrastructure providing services (e.g. the Kubernetes podcast from Google). I´m not convinced yet. I think this is more like kicking down responsibility down the chain. And then it feels more like: Someone needs to do the dirty work but nobody wants to do it.

It might work if you don´t have to provide Service-level agreements (in our case: we don´t). For us it is just more work to do. And our work shifts from dev to ops. Instead of writing software we´re mostly busy with configuring cloud resources. This will ease a bit once everything is running. However: I see this whole change more as ... uh, strong word... ideologically motivated. Cui bono? Neither our team, nor our users nor our infrastructure bill.

Re: “Who Should Write the Terraform?”

#125

Post by yarosh »

upupandup wrote: post #51

The answer to this question is that you should never be writing Terraform/CDK from scratch, you are wasting time. 1. Scaffold your infrastructure with simple point & click in web console. 2. Generate terraform/CDK code by scanning your AWS account with typically available tools. 3. Edit an update said Infrastructure as Code as needed, swapping out the parameters with the vectors you need to change according to CI/CD…

> "i want to write infrastructure as code from day 1" is not only stupid , its a waste of resources

I tend to disagree. Depends on the scale... and after you've scaled and grown DevSecOps absence becomes a source of detraction, affect your delivery cycle and indirectly your Sales. Proper DevOps defines some of the business lifecycle operations as well, like BI and A/B testing, which essentially helps in validating pending Business Assumptions. It's something that can help differentiating the market and Validate the actual Product Viability - prove that your MVP actually has any V in it.

Operations wise, First and foremost you have to keep track of the issues that are currently present in AWS solutions and automate workarounds, and there are a lot of security automation and organizational means which can't really be solved with a "Click in Web Console" efficiently.

For instance, setting up a proper EKS cluster by hand, without any hardening, would require at least three hours of clicking through, with all the IRSA roles and EKS specific IAM permissions. While, on the other hand, Terraform automation has ready to use OpenSource modules shipped by both the community and AWS itself (terraform-aws-modules, aws-ia), which introduces some advanced EKS management practices, without any added effort. 10 lines of IaC can easily replace half an hour of click-through.

The cost of Integration is nearly Zero during the product bootstrap phase, but when you're growing integrating proper Organizational Management with AWS Organizations and Control Tower, reordering your AWS Accounts, transferring resources, and hardening security boundaries tends to rise in complexity and cost a lot. Especially if you'll ever want to perform proper security Audits or need some HIPAA/GDPR compliance.

For some Disney companies, for instance, who choose to perform org management by developing custom tools after 5 years of operation, proper integration with AWS Organizations remained a dream, and their unreasonably tight Operational Schedule and On-call deficiency became a source of detraction. The integration cost rose to eight figures.

The cost of DevSecOps hardening basically doubles every quarter, if you're growing fast enough and lack automation.

As for myself, automating everything allowed me manage Kubernetes complexity and develop a fine tuned vertically scalable solution (VPA+HPA on Keda with cluster autoscalers) - about 30 different k8s services deployed in a mix of x86 and Arm instances, with continuous placement and resource limits/requests optimization, completely downscalable. My AWS bill is only 7% of my raw income.

So, if you can hire a DevOps consultancy, and can Actually Measure how much time is wasted during the manual operation compared to the automated one, able to self reflect without a confirmation bias, do that ASAP.

Re: “Who Should Write the Terraform?”

#126

Post by Aeolun »

aranelsurion wrote: post #70
theamk wrote: post #15

Re this segment: > "There were endless complaints about the time taken to get ‘central IT’ to do their bidding, and frequent demands for more autonomy and freedom. A cloud project was initiated by the centralised DBA team to enable that autonomy. [...] Cue howls of despair from the development teams that they need a centralised DBA service" Author makes it sound like users didn't know what they wanted. This is not tr…

> _it was a different set of people_ who were complaining before and after. That's something I've observed as well. Seems to me there are (at least) two developer personas, one kind only wants to deliver their task, specialize in what they do well, and generally can't care less if their DB is oversized or has no maintenance windows set or no recovery plans or who has access to it etc. They usually lack the cloud/plat…

> For them there is nothing worse than relying on another team while they could geek out on the subject themselves and they believe they could do it better.

It may not be optimal, but it’s almost invariably faster.

Personally I like the way my company does it, where people have (more or less) full access to the AWS account, but there’s a lot of automated guardrails/scanning that alerts you when you’ve done something stupid (public S3 buckets etc.)

Re: “Who Should Write the Terraform?”

#127

Post by Aeolun »

jiveturkey wrote: post #68
jesseryoung wrote: post #64

Earlier quoted context omitted.

What exactly is a "Conflicting duty"? What's stopping a company from stating that developing, deploying and supporting software is a single duty?

Nothing ... except compliance. The idea comes from finance -- to require collusion to execute a fraud. It's not perfect, but it's something.

It seems like the description is vague enough that this is entirely up to whoever gives you your certification.

Re: “Who Should Write the Terraform?”

#128

Post by Too »

mountainriver wrote: post #91
cube2222 wrote: post #22

Software Engineer at Spacelift[0] here - a CI/CD specialized for Infra as Code (including Terraform). A pattern we're seeing increasingly commonly are Platform Engineering teams doing the bulk of the work, including all the fundamentals, guidelines, safety railing, and conventions, while Software Engineers only use those, or write their own simple service-specific Terraform Stacks which however extensively use module…

Seen this at a couple companies and it doesn’t work well. The platform team becomes a bottleneck and the devs don’t want to have to deal with or learn the mess that is terraform. It’s time for the ecosystem to move beyond the half baked config language known as HCL

What do you suggest instead?

Re: “Who Should Write the Terraform?”

#129

Post by frederich »

Too wrote: post #128
mountainriver wrote: post #91

Earlier quoted context omitted.

Seen this at a couple companies and it doesn’t work well. The platform team becomes a bottleneck and the devs don’t want to have to deal with or learn the mess that is terraform. It’s time for the ecosystem to move beyond the half baked config language known as HCL

What do you suggest instead?

Have a look at GruCloud, an alternative to Terraform/Pulumi/CDK, which generates the infrastructure code automatically from a live infrastructure. Disclaimer, I am the author.

Re: “Who Should Write the Terraform?”

#130

Post by mountainriver »

Too wrote: post #128
mountainriver wrote: post #91

Earlier quoted context omitted.

Seen this at a couple companies and it doesn’t work well. The platform team becomes a bottleneck and the devs don’t want to have to deal with or learn the mess that is terraform. It’s time for the ecosystem to move beyond the half baked config language known as HCL

What do you suggest instead?

Pulumi seems a lot more sane, trying to back all the complexity of infrastructure into a config language just doesn't add up at the end of the day. This is why we have general purpose languages.

We could also probably use more abstractions similar to Pulumi, theres been talk on HN about storing all of the state for applications like this in the tags of the underlying cloud resources. There are some caveats with this approach, but it would provide a interesting tradeoffs

Post reply on HN