Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

System Initiative: Second Wave DevOps

systeminit.com

Post reply on HN
Print view View original

41–50 of 80 posts

Re: System Initiative: Second Wave DevOps

#41

Post by dipperdottydoo »

chologrande wrote: post #19
totallywrong wrote: post #13

Earlier quoted context omitted.

> Devops doesn't have to be hard. Infrastructure doesn't have to be complex That's simply not true for anything larger than a few services and a small dev team. The cloud is very complex to do right when you focus on security, performance, and scalability. And Terraform invariably devolves into a nightmare when you have a ton of resources with dependencies between them.

I'm definitely not google scale, but we're global, in over 300 cities spanning ~30 countries. On an avg day we process well over 25k rps on multiple services. Simple architecture and IaC like terraform is exactly how we manage the dependencies. It's the solution, not the problem.

You think you have simple architecture when you’ve introduced Terraform to what is, based on your statistics, a two server use case. A PlayStation is capable of 25 kRPS and probably its data iops, too. Buy another one and you’re HA.

You’re trapped in the complexity of the method and think you’ve achieved nirvana. This comment reminds me of those demos when Hadoop was the rage, where people would do a $4 million Hadoop ETL on their laptop and shut up a room.

Re: System Initiative: Second Wave DevOps

#42

Post by jedberg »

JohnMakin wrote: post #32
chologrande wrote: post #7

After reading this post, I've browsed the site. I'm not sure how this is anything but significantly worse than the current model? I've been around long enough to know that any "no code" style interface or GUI are typically the _problem_ not the solution. Regardless of the code they export, you end up with fat fingers, misclicks, forgotten UI paths to follow... Taking a software eng approach to shipping infra is a sta…

Well said. Click ops is the root of all evil, IME, unless you're a very small shop.

Click ops is great for most shops, as long as it has advanced configs available and you have at least one expert on staff for when those configs are needed.

At Netflix our goal was always to build tools where the majority of devs just check into source control and click a few buttons, but could go as far as configuring kernel tunables if necessary (but also making that as unnecessary as possible).

Re: System Initiative: Second Wave DevOps

#43

Post by pmoriarty »

esafak wrote: post #36
pmoriarty wrote: post #18

Earlier quoted context omitted.

"Deploying every day isn't _that_ difficult" Just don't ever ask to roll back...

Why? Keep build artifacts and deploy any build you like. Do you have a state or dependency problem?

It's quite common for one service to depend on another (or multiple others), on the network/firewall state, on configurations that might affect or be affected by other services, etc.

What looks simple when you're the king of your own little kingdom suddenly doesn't seem as simple when that fantasy meets the reality of sharing the world with others.

Re: System Initiative: Second Wave DevOps

#44

Post by anotherhue »

We could begin by accepting that operations work often deals with far more complex problems than feature work.

Yes, the tooling is bad, all tooling is bad, but hearing the same old 'Infrastructure should "just work"' trope is getting old. Such developers should stop grandstanding and roll up their sleeves. Learning about TCP isn't beneath you.

Re: System Initiative: Second Wave DevOps

#45

Post by midasuni »

chologrande wrote: post #19
totallywrong wrote: post #13

Earlier quoted context omitted.

> Devops doesn't have to be hard. Infrastructure doesn't have to be complex That's simply not true for anything larger than a few services and a small dev team. The cloud is very complex to do right when you focus on security, performance, and scalability. And Terraform invariably devolves into a nightmare when you have a ton of resources with dependencies between them.

I'm definitely not google scale, but we're global, in over 300 cities spanning ~30 countries. On an avg day we process well over 25k rps on multiple services. Simple architecture and IaC like terraform is exactly how we manage the dependencies. It's the solution, not the problem.

25krps? As in requests per second? I.e one request every 3 seconds?

Re: System Initiative: Second Wave DevOps

#46

Post by hinkley »

convolvatron wrote: post #39
hinkley wrote: post #27

Earlier quoted context omitted.

That’s definitely part of the dynamic. My biggest regret with automated testing is that software used to be a triumvirate of Quality, Dev, and Management, and when dev was fucking around they had two teams hitting them, and when Management was out of control, they had everyone mad at them. Get rid of QA and it’s Us vs Them and that worked briefly at the dawn of Agile but they got wise. OP’s should replace QA at that…

I agree. but having a QA group was hardly a magic bullet. most of the really talented contributors eschewed QA - even if they enjoyed it - because it meant being identified as a QA person for the rest of their career. even if you could staff a competent group, they are often left with nothing to do while the devs 'work their magic'. and suddenly its 3 months past the original 'functionally complete' deadline, and QA…

I would definitely say that some of the finest and worst people I’ve worked with were in QA. Mercifully usually not at the same job. It can be a great place to push ideals, or to hide from them.

I know why it’s gone, I’m just not happy about it.

Re: System Initiative: Second Wave DevOps

#47

Post by Mutlut »

chologrande wrote: post #7

After reading this post, I've browsed the site. I'm not sure how this is anything but significantly worse than the current model? I've been around long enough to know that any "no code" style interface or GUI are typically the _problem_ not the solution. Regardless of the code they export, you end up with fat fingers, misclicks, forgotten UI paths to follow... Taking a software eng approach to shipping infra is a sta…

Great that your setup work, i personally hate terraform and try to avoid it.

k8s is also KISS but it brings even more 'out of the box' like logging and monitoring, would highly recommend you to take a look perhaps you like it.

Terraforms state management is bad and a lot of people don't get that you store secrets in them. Bootstrapping this securly already needs infrastructure like remote stores.

Jenkins is fine i would say but with argocd you actually gain real insight. Argocd is also IaC and you can manage argocd through argocd.

The adoption of argocd in the platforms i have build, is great. Developer teams love it, get used to it very fast and don't need cluster access/ (in your case vm access).

With k8s you also get zero downtime deployment, blue / green basically for free.

Re: System Initiative: Second Wave DevOps

#49

Post by dmlittle »

midasuni wrote: post #45
chologrande wrote: post #19

Earlier quoted context omitted.

I'm definitely not google scale, but we're global, in over 300 cities spanning ~30 countries. On an avg day we process well over 25k rps on multiple services. Simple architecture and IaC like terraform is exactly how we manage the dependencies. It's the solution, not the problem.

25krps? As in requests per second? I.e one request every 3 seconds?

How do you go from "25,000 requests per second" (25krps) to "one request every 3 seconds"?

Re: System Initiative: Second Wave DevOps

#50

Post by Mutlut »

I'm very curious about this as normally all tools i know still have a higher entry point than i realize.

My current setup is 'get a k8s cluster spup up and configured properly as fast and easy as possible' and than just use argocd. Argocd is by far the best tool i have been using in the last 15 years: It does exactly what it should do (syncing and showing me k8s insight vs my git repo), can manage itself through the same mechanism (IaC) and people of different backgrouns are very fast in using it.

This tool either might bridge the gap for people and potentially solve problems but i do have to say: argocd.

Even if you think you want to start small and just use kubectl: start with argocd.

Post reply on HN