Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Big DevOps Misunderstanding

wolfoliver.medium.com

Post reply on HN
Print view View original

311–320 of 321 posts

Re: The Big DevOps Misunderstanding

#311

Post by boublepop »

notimrelakatos wrote: post #158
manish_gill wrote: post #54

I had to basically give up my full time role as a Data Engineer who is supposed to solve "big data" problems (Storage, Streaming, Machine Learning pipelines) etc, and had to solely focus on DevOps because there was no other expert in the team. We (a colleague and I) basically learned Terraform, then Kubernetes, and then Helm in an extremely stressful environment. I wouldn't recommend that to most people. (Also, I can…

> it makes the difficult things possible, and that's what we needed Indeed, the problem of just saying "you don't need k8s" is you may accidentally invent an even worse in-house version of k8s.

Though for 80-90% og cases out there you seriously don’t need k8s, and when the small internal IT dept that used to “manage” a dozen virtual machines starts talking about “need to keep up with the times and use new tech” what they are really saying is “let’s gamble infrastructure stability on my personal self education in k8s for my next job” and it’s perfectly reasonable to give them a hard no.

Re: The Big DevOps Misunderstanding

#312

Post by hnthrowaway0315 »

stepbeek wrote: post #306
hnthrowaway0315 wrote: post #101

Earlier quoted context omitted.

OMG, I read the sample chapters from Amazon about the Phoenix project and that was a really really good read, thank you!

It’s an amazingly good read. The follow up - The Unicorn Project - wasn’t as good, but definitely worth reading.

Thanks, really LOL a few times during the read.

Re: The Big DevOps Misunderstanding

#313

Post by ayberk »

deanCommie wrote: post #298
ayberk wrote: post #216

Earlier quoted context omitted.

> "Real engineers"? This is a Googleism and their caste system of "real engineers" and "software reliability engineers". The rest of the industry doesn't coddle their precious "real engineers". There's no such caste system at Google. SREs have the same bar as the SWEs, and plenty of people switch roles from one to other. SREs are also well-respected and I don't think anyone would consider them as _not_ real engineers…

I can't find the post, but I base this on rachelbythebay's writing - who got hired as an SRE, and then tried to become an SWE and was told she couldn't just convert. But fair enough, I don't work at Google, so I'll withdraw the point. Having said that, knowing that AWS engineers do oncall, and GCP don't (letting the SREs do it) makes me still think there is some sort of two-tier system.

Before you withdraw your point, I guess I have to say my experience is limited to my org and maybe one or two more. Google is a big company, so I can see that happening. I'd still presume it's not the general attitude towards SREs, though.

re: Oncall, we have two rotations:

- one manned (personed?) by SWEs, 9-5, responsible for dealing with customer issues and mandatory.

- one is mix of SREs and SWEs, 24/7, responsible for prod issues

I believe SREs also have their own rotation, but that's 9am-9pm because they are always spread among two timezones. Overall, this is muuuuuch better for everyone involved compared to the AWS oncalls. I remember barely sleeping for a week being the norm on one of the teams I worked at. Our cries for another team in a different timezone, similar to Google SREs, were shut down every single time. "Customer obsession" at AWS means delivering stuff as fast as possible and then throwing the engineers under the bus. I still remember the days I had to wake up multiple times a night to run a command manually (literally 5 minutes) because engineers couldn't take the extra 4 months to do it right.

Thanks, but no thanks. I was at a great team at AWS for ~2 years with great engineering culture and little operational load, but unlike Google, that's rare.

Re: The Big DevOps Misunderstanding

#314

Post by dvtrn »

mrweasel wrote: post #245
Pxtl wrote: post #172

Earlier quoted context omitted.

> And through this I've noticed that developers can be just as bad at communicating issues I swear to god the next guy who sends me the screenshot of an error log...

The number of developers who believe that operations is a support channel for development is truly amazing. How am I suppose to know why your software is broken? You wrote it. In reality it's easy enough most of the time, if you can read logs. Still, there's a weird expectation that operations is a bunch of elite developers, knowledgeable about all programming language, who just choose to do operations because they e…

The number of developers who believe that operations is a support channel for development is truly amazing.

The amount of engineering leaders that allow it to happen is far more frustrating, at least IMO it is.

Re: The Big DevOps Misunderstanding

#315

Post by ayberk »

amzn-throw wrote: post #301
ayberk wrote: post #296

Earlier quoted context omitted.

It really isn't. It's just the cultural difference between the two companies. I have been told that GCP oncalls are a lot busier than rest of the Google, but it's still nowhere close to the suffering I had at AWS. It's an organizational pain and comes from the mindset AWS has towards software development and their engineers.

AWS's mindset is software is useless without customers. Google doesn't seem to have the same care - it's a charity for academic software engineers to spend AdSense revenue on abstract high level computer science problems not caring about practical applications. It's why most Google X ideas flame out. It's why there's a new chat app every year. But - it's a net positive for humanity. Google publishes a lot of papers,…

I'm sure you're more than enough smart to be at Google :) It's just practice and luck.

I completely agree that people who prefer to ship fast would be happier at AWS. I've heard all the horror stories about how slow Google is, but I think GCP has a great balance between speed and quality. I'm definitely happier here, but I understand why some people would be happier at AWS.

Re: The Big DevOps Misunderstanding

#316

Post by IanSanders »

xxpor wrote: post #302
hcrean wrote: post #276

Earlier quoted context omitted.

His point is that Ops time is more expensive than Dev time, they get paid more. The Dev teams should really fix their own bugs if they happen in the wee-small-hours.

Not just that (since there's a lot of situations you may not be able to actually fix bugs in the moment, just work around them), but it also aligns incentives and enforces lessons learned. If you know you're gonna be on the other end of that pager, maybe that hack won't see the light of day. For more senior folks, pattern recognition around how things were built and what issues they led to are incredibly valuable. Be…

All they said is they want to keep work within contracted hours.

> waste my already precious *freetime* on fixing bugs in prod

It doesn't mean bugs cannot be raised, planned and addressed through the normal development process.

Also, abundance of bugs in production could indicate cutting costs on testing.

Re: The Big DevOps Misunderstanding

#317

Post by clh1126 »

throwaway984393 wrote: post #206

"Things get much easier if you-" You can stop right there. There is no easy way. That is the DevOps misunderstanding. To understand why, look at your car. Your car is a complex system that appears simple. You just purchase it, put gas in it, turn a key, shift a gear, press a pedal, and it goes. And goes, and goes, and goes. But over time the tires will go bald, the brakes will wear out, you will run out of coolant an…

+1 this, and Devops can also be strongly traced back to Deming/Japan. Developers not caring to learn about ops or how their code functions in actual production are the ones who could care less about the long term impact or quality of their work.

Re: The Big DevOps Misunderstanding

#318

Post by smus »

dijksterhuis wrote: post #291
smus wrote: post #174

Earlier quoted context omitted.

I'm glad there's a dichotomy. I (a dev) never signed up to be on call 24/7 for production and it's ridiculous to just assume that I should be. I have a life outside of my work. If the system needs 24/7 uptime, they better be paying someone other than me to ensure that's the case, because my time is too valuable to waste my already precious freetime on fixing bugs in prod.

> If the system needs 24/7 uptime, ..., because my time is too valuable to waste my already precious freetime on fixing bugs in prod. As an ops-sided person, this is the exact reason why ops folks always said no to releases on a Friday afternoon. Ops people would rather be in the pub enjoying their bug free weekend by not risking pushing buggy broken code from developers, in spite of the desperate pleas and guarantee…

Totally, and I'd never suggest deploying on a Friday afternoon

Re: The Big DevOps Misunderstanding

#319

Post by smus »

xxpor wrote: post #207
smus wrote: post #174

Earlier quoted context omitted.

I'm glad there's a dichotomy. I (a dev) never signed up to be on call 24/7 for production and it's ridiculous to just assume that I should be. I have a life outside of my work. If the system needs 24/7 uptime, they better be paying someone other than me to ensure that's the case, because my time is too valuable to waste my already precious freetime on fixing bugs in prod.

Who wrote those bugs in the first place?

Good point, devs should either just not write bugs OR be on call 24/7. That's a totally reasonable way of running things.

I understand if people's lives are on the line. Otherwise, recognize that bugs happen, and either be ok with that or come up with a process (QA) that finds these bugs before deployment or that allows you to roll back to a more stable version easily.

Re: The Big DevOps Misunderstanding

#320

Post by dijksterhuis »

smus wrote: post #318
dijksterhuis wrote: post #291

Earlier quoted context omitted.

> If the system needs 24/7 uptime, ..., because my time is too valuable to waste my already precious freetime on fixing bugs in prod. As an ops-sided person, this is the exact reason why ops folks always said no to releases on a Friday afternoon. Ops people would rather be in the pub enjoying their bug free weekend by not risking pushing buggy broken code from developers, in spite of the desperate pleas and guarantee…

Totally, and I'd never suggest deploying on a Friday afternoon

You've completely missed the point of what I was saying and/or are unwilling to see what my point was.

That will only work if you never need to release anything over a weekend... and some of us actually need/want to be able to do that -- safely. As a result we work in a different team structure, with different tooling to mitigate risks and we share the responsibility as best we can (no, not everyone is expected to be 24/7 on call for production).

Post reply on HN