Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Who Should Write the Terraform?”

zwischenzugs.com

Post reply on HN
Print view View original

61–70 of 137 posts

Re: “Who Should Write the Terraform?”

#61

Post by robertlagrant »

jesseryoung wrote: post #56

The author calls out a few reasons why DevOps fails for organizations all of which I agree with - however the one that I've never completely understood: Regulatory reasons for keeping Ops centralized. I work in healthcare which I guess should fall under this rule - but in practice I haven't really seen that impeding DevOps. Teams that have the capabilities to build the full stack get handed a subscription to a cloud…

Same here - made and make health apps, ISO13485. No requirement for a central ops team.

Re: “Who Should Write the Terraform?”

#62

Post by lawnchair »

At Terrateam[0] we specialize in Terraform automation with GitHub Pull Requests and GitHub Actions.

We talk to a lot of Terraform users from a lot of different companies. The most popular way of doing things is having your SRE/DevOps team write the bulk of the Terraform modules for your organization. Other members of engineering then consume these modules to create resources for their platform/application/etc. This code can either live in a Terraform monorepo or inside an application-specific repo. We've seen many approaches.

Scaling Terraform inside your organization is incredibly convenient with Terrateam as we leverage many pieces of GitHub.

[0]: https://terrateam.io

Re: “Who Should Write the Terraform?”

#63

Post by PragmaticPulp »

danwee wrote: post #34
cube2222 wrote: post #22

Software Engineer at Spacelift[0] here - a CI/CD specialized for Infra as Code (including Terraform). A pattern we're seeing increasingly commonly are Platform Engineering teams doing the bulk of the work, including all the fundamentals, guidelines, safety railing, and conventions, while Software Engineers only use those, or write their own simple service-specific Terraform Stacks which however extensively use module…

Having a single "platform" team per company is a bottleneck as soon as the number of product teams is greater than N. > ...you'll probably start having as many approaches to your infrastructure as there are teams, complexity will explode, and implementation/verification of compliance requirements will be a chore. Just a few people responsible for handling this will yield huge benefits. Agree with the centralization o…

> Having a single "platform" team per company is a bottleneck as soon as the number of product teams is greater than N.

This is my experience as well. Having a single platform team has been a great experience for laying foundations, establishing shared architectures, and centralization documentation.

As soon as two or more teams need something from the platform team, it becomes a battle of priorities. A good platform team will recognize this and work on a division of labor and coordination strategy that can start to scale. A bad platform team will treat this as an opportunity to claim the company’s wins for themselves and leverage their bottleneck position for political gain.

The company’s management of the platform team is key. I’ve also seen a single platform team abused as the engineers who are expected to own all the hard work while other teams get to walk all over them with demands. This results in a lot of employee turnover, which is the opposite of what you want on a team tasked with holding the core knowledge of the company’s infrastructure.

Re: “Who Should Write the Terraform?”

#64

Post by jesseryoung »

jgworks wrote: post #58
jesseryoung wrote: post #56

The author calls out a few reasons why DevOps fails for organizations all of which I agree with - however the one that I've never completely understood: Regulatory reasons for keeping Ops centralized. I work in healthcare which I guess should fall under this rule - but in practice I haven't really seen that impeding DevOps. Teams that have the capabilities to build the full stack get handed a subscription to a cloud…

I am not in a regulated industry, but we have recently gone through the process of getting SOC2/ISO27001 certified. This is what was cited for us. ISO27001:2013 A.6.1.2: Segregation of Duties. Conflicting duties and areas of responsibility must be segregated in order to reduce the opportunities for unauthorized or unintentional modification or misuse of any of the organization's assets.

What exactly is a "Conflicting duty"? What's stopping a company from stating that developing, deploying and supporting software is a single duty?

Re: “Who Should Write the Terraform?”

#65

Post by randerson »

jgworks wrote: post #58
jesseryoung wrote: post #56

The author calls out a few reasons why DevOps fails for organizations all of which I agree with - however the one that I've never completely understood: Regulatory reasons for keeping Ops centralized. I work in healthcare which I guess should fall under this rule - but in practice I haven't really seen that impeding DevOps. Teams that have the capabilities to build the full stack get handed a subscription to a cloud…

I am not in a regulated industry, but we have recently gone through the process of getting SOC2/ISO27001 certified. This is what was cited for us. ISO27001:2013 A.6.1.2: Segregation of Duties. Conflicting duties and areas of responsibility must be segregated in order to reduce the opportunities for unauthorized or unintentional modification or misuse of any of the organization's assets.

Surely that means that no one individual can push a change they created without involving someone else, but that it is still fine as long as any two people (even if they're on the same team) are involved? You could solve this by e.g. forcing GitHub to require a review.

Re: “Who Should Write the Terraform?”

#67

Post by jiveturkey »

jesseryoung wrote: post #56

The author calls out a few reasons why DevOps fails for organizations all of which I agree with - however the one that I've never completely understood: Regulatory reasons for keeping Ops centralized. I work in healthcare which I guess should fall under this rule - but in practice I haven't really seen that impeding DevOps. Teams that have the capabilities to build the full stack get handed a subscription to a cloud…

SOD - Segregation of Duties.

Re: “Who Should Write the Terraform?”

#68

Post by jiveturkey »

jesseryoung wrote: post #64
jgworks wrote: post #58

Earlier quoted context omitted.

I am not in a regulated industry, but we have recently gone through the process of getting SOC2/ISO27001 certified. This is what was cited for us. ISO27001:2013 A.6.1.2: Segregation of Duties. Conflicting duties and areas of responsibility must be segregated in order to reduce the opportunities for unauthorized or unintentional modification or misuse of any of the organization's assets.

What exactly is a "Conflicting duty"? What's stopping a company from stating that developing, deploying and supporting software is a single duty?

Nothing ... except compliance.

The idea comes from finance -- to require collusion to execute a fraud. It's not perfect, but it's something.

Re: “Who Should Write the Terraform?”

#69

Post by unethical_ban »

In my old employer, "Ops" and "Security" still had role for managing fundamental components of the system. In AWS terms, that means deploying AWS accounts in the organization, automation for best practices and compliance detection, IAM roles, VPCs, etc.

Security and network teams also built custom terraform modules which the deve teams were forced to use that were guardrails. You didn't use aws_s3_bucket, you used custom_aws_s3_bucket that mandated certain fields and prerequisites. This was the compromise struck to allow devs otherwise to go ham in their own AWS accounts and self-manage their deploys, databases, and so on.

Re: “Who Should Write the Terraform?”

#70

Post by aranelsurion »

theamk wrote: post #15

Re this segment: > "There were endless complaints about the time taken to get ‘central IT’ to do their bidding, and frequent demands for more autonomy and freedom. A cloud project was initiated by the centralised DBA team to enable that autonomy. [...] Cue howls of despair from the development teams that they need a centralised DBA service" Author makes it sound like users didn't know what they wanted. This is not tr…

> _it was a different set of people_ who were complaining before and after.

That's something I've observed as well. Seems to me there are (at least) two developer personas, one kind only wants to deliver their task, specialize in what they do well, and generally can't care less if their DB is oversized or has no maintenance windows set or no recovery plans or who has access to it etc. They usually lack the cloud/platform skills as well, and won't develop much in that regard because they don't care to. Even if they did, they're unlikely to get rewarded much for that effort. They are easy to make happy, and you rarely hear from them other than the occasional "thanks" in some Slack channel.

The other kind is either internally very curious about the subject, or already has the experience, or at least they think they have it. They want to have full access, invent things anew in the "right way" they believe. For them there is nothing worse than relying on another team while they could geek out on the subject themselves and they believe they could do it better. Sometimes they're right about it, and other times they're either oversimplifying the work needed, or optimizing locally around themselves/their team/their task. There seems to be no way to make them really happy other than giving them complete freedom to do whatever.

Seems to me most developers (I've worked with) are of the first kind, and they can be made happy after some level of maturity is reached within the company, but the second kind is way more vocal and they won't ever be happy with whatever a central team builds.

More and more I'm getting convinced that the only way to really win both personas is to build two products instead of one. So you build the golden path, the Helm chart or the portal or whatever for the first kind, and give ownership and loosely govern with compliance/policy tooling with the second kind.

This optimizes for the short/mid-term satisfaction, but of course it can also go wrong since team compositions are not set in stone and what one builds may not be maintained properly by the other, and there'll be some duplication of efforts and quality of solutions built might vary between the teams. I guess for some companies this is acceptable, and for others it won't ever be.

tl;dr ¯\_(ツ)_/¯

Post reply on HN