Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All of us test in production all the time (2019)

increment.com

Post reply on HN
Print view View original

61–70 of 164 posts

Re: All of us test in production all the time (2019)

#61

Post by vinceguidry »

Nbox9 wrote: post #37
mandelbrotwurst wrote: post #30

Earlier quoted context omitted.

He mentioned that he had a job doing that, but the company moved away from the framework without technical justification. He could find another, but there would be no guarantee that the same thing wouldn't happen again.

> the company moved away from the framework without technical justification I think this alone is enough justification to look for a new place to work, but I’m just a job hopping millennial.

I did. Asked to be moved to a different client. Still there.

Re: All of us test in production all the time (2019)

#62

Post by vinceguidry »

tluyben2 wrote: post #36
vinceguidry wrote: post #15

A lot of this boils down to the constant churn in frameworks, instrumentation, programming languages. Developers don't have time to master one way of doing things, it's just a constant lava layer of crap. I took the time to learn how to build and maintain Ruby on Rails systems, hoping it would be the ticket to a fun, manageable career. Any project I worked on was an island of sustainable, fast development where the t…

I had the same issue with Rails as I have with Node : the churn is detriment to long standing projects. Most projects I do run for years and even decades; you go run an update for security for Rails code that is 10 years old. It is a nightmare. I did not have issues like that with php, asp.net or spring.

I loathe NodeJS with the passion of a thousand suns. When Rails projects churn, at least you can rely on the stability of Ruby. With Node, the number of times I had to work around inconsistencies with the framework and even language were maddening. Babel is a horrible horrible thing, and Javascript was never meant to run on a server.

Maybe it's better now. But I won't ever do serious work again with the stack so I'll never find out.

Re: All of us test in production all the time (2019)

#63

Post by mr_tristan »

danpozmanter wrote: post #42

"Engineers should be on call for their own code." - Would you rather work someplace you are expected to be on call 24/7, or a company that doesn't require that? It isn't the norm, and it isn't competitive. It's just more "always on" culture in the workplace - and that's not healthy. A company should understand workers need real breaks - and being on call is not a real break.

I've found the problem isn't being on call, it's being on call PLUS not having control over priorities.

When doing it right, owning the on call experience can be a valuable learning experience. But this usually means having extra time to do things like develop integration and performance testing environments. And access to make whatever changes needed to happen, happen.

But a lot of places are like "nah, you're being a perfectionist". And then expect you to magically respond to issues with vague descriptions and no diagnostics. And yeah, that sucks.

Re: All of us test in production all the time (2019)

#64

Post by bob1029 »

danpozmanter wrote: post #42

"Engineers should be on call for their own code." - Would you rather work someplace you are expected to be on call 24/7, or a company that doesn't require that? It isn't the norm, and it isn't competitive. It's just more "always on" culture in the workplace - and that's not healthy. A company should understand workers need real breaks - and being on call is not a real break.

The whole point of putting engineers on call (in this context) is to encourage them to make good technology choices and to take some ownership for their products. If there was no counter-pressure with pager duty or other threats to peaceful existence, then most developers would just pick whatever technology they personally enjoy using the most and expect that someone else will fix their special shitpile for them at 3am. Someone is always going to get screwed in this equation, at least make it an equitable screwing.

Being on-call doesn't just apply to code either. Would you be OK if no one tried to fix your broken water pipes or electricity until the following business day? Do we turn off the global internet at bed time?

At some point people are going to have to do shitty work to keep this world running. The best you can do is rotate the shitty work around so that everyone can help out. Automate what you can, share the load for what you cannot. If everyone does their part, it is a lot less painful all around.

Re: All of us test in production all the time (2019)

#65

Post by rybosworld »

vinceguidry wrote: post #62
tluyben2 wrote: post #36

Earlier quoted context omitted.

I had the same issue with Rails as I have with Node : the churn is detriment to long standing projects. Most projects I do run for years and even decades; you go run an update for security for Rails code that is 10 years old. It is a nightmare. I did not have issues like that with php, asp.net or spring.

I loathe NodeJS with the passion of a thousand suns. When Rails projects churn, at least you can rely on the stability of Ruby. With Node, the number of times I had to work around inconsistencies with the framework and even language were maddening. Babel is a horrible horrible thing, and Javascript was never meant to run on a server. Maybe it's better now. But I won't ever do serious work again with the stack so I'll…

Node is getting better each year imo. I've been pretty happy with ES6.

What makes you say Babel is horrible? Not disagreeing but I haven't heard anyone say that before.

Re: All of us test in production all the time (2019)

#66

Post by binsh »

TedDoesntTalk wrote: post #47
tluyben2 wrote: post #35

Earlier quoted context omitted.

> Sometimes what works now won't be competitive in the near future. What do you mean by that? I can understand in terms of hiring devs, this is true, but any other reason I would be curious about. Because I work with/in many companies, I see a lot of different tech and competitive is usually just what the team knows best. So unless you are making a new team, I cannot phantom why you would switch. Java with Servlets/S…

Perhaps he means that you won’t be able to attract great talent. A lot of extremely talented engineers don’t want to work on code bases and technologies that are 15 years old.

Definitely would like a source for this because I feel like I hear it a lot and I don't believe it's true.

You can be talented and not care what you work on. And if all you care about is what tech is new and shiny, how talented can you be? I don't want you rewriting my product every year so you can use the next framework du jour.

Maybe talent means something else when nothing is at stake, but I'd argue that a talented engineer knows that when a product works and makes money, and the tech stack isn't prohibitive of those goals, it doesn't really matter what it's written in. In fact, I see trend-chasing developers as liabilities, because they're the ones who will replace everything every six months while you bleed money.

Re: All of us test in production all the time (2019)

#67

Post by ratww »

_diq5 wrote: post #55
danpozmanter wrote: post #42

"Engineers should be on call for their own code." - Would you rather work someplace you are expected to be on call 24/7, or a company that doesn't require that? It isn't the norm, and it isn't competitive. It's just more "always on" culture in the workplace - and that's not healthy. A company should understand workers need real breaks - and being on call is not a real break.

Want to jump in here - I have worked at a company where engineers are not on call for their code, and it was a living nightmare. _You_ might not be on call for your code, but _somebody_ will be. Often some poor SRE/ops person that has absolutely no idea what the app is doing/or why it's failing in production. Not being on-call makes engineers complicit. I've seen it all, known memory leaks shipped into production, ap…

I'm always happy to help some poor SRE in the middle of the night, and I once even drove to the office in a rainy Sunday, in the middle of my vacation, to access IP-restricted stuff because a support intern messaged me on Instagram.

...but with that said: I'm glad I only worked in countries where work is properly regulated and "on call" means "I'm getting fucking paid every cent for each hour I _must_ answer that goddamn phone". Which in practice means there's no PagerDuty.

The unpaid on-call culture is bullshit. The company can either pay me or go fuck itself.

Re: All of us test in production all the time (2019)

#68

Post by danpozmanter »

bob1029 wrote: post #64
danpozmanter wrote: post #42

"Engineers should be on call for their own code." - Would you rather work someplace you are expected to be on call 24/7, or a company that doesn't require that? It isn't the norm, and it isn't competitive. It's just more "always on" culture in the workplace - and that's not healthy. A company should understand workers need real breaks - and being on call is not a real break.

The whole point of putting engineers on call (in this context) is to encourage them to make good technology choices and to take some ownership for their products. If there was no counter-pressure with pager duty or other threats to peaceful existence, then most developers would just pick whatever technology they personally enjoy using the most and expect that someone else will fix their special shitpile for them at 3…

There's more going on and worth exploring if engineers are so unattached to outcomes they pick technology in a vacuum. Pager duty is a heavy stick - it's important not to avoid root cause analysis. So if your engineers are making bad choices - what's really going on?

That assumes engineers are even empowered to make technology choices. At many companies they are not (whether by dint of organizational structure or the roadmap not allowing a major technology shift from whatever "shitpile" you and your team have inherited).

Having clear escalation strategies (and knowing when escalation to the original engineers behind a project is even appropriate) is often lacking. I wouldn't want to call engineers in at 3am for a problem that can be fixed by following a documented devops process. Plus - what happens when the engineer you need to reach is unavailable? They are sick, or don't wake up, or their phone died?

What happens when business pressure says "we're ok with calling engineers twice a week as long as the roadmap moves"?

"You built it you're on call" is a fragile way to handle problems in more ways than one.

Which isn't to say there shouldn't be shared responsibility. Of course there should. But responsibility without power is toxic. At the very least it increases flight risk - but in practice often has a far wider reaching deleterious effect than just that.

Re: All of us test in production all the time (2019)

#69

Post by toast0 »

danpozmanter wrote: post #42

"Engineers should be on call for their own code." - Would you rather work someplace you are expected to be on call 24/7, or a company that doesn't require that? It isn't the norm, and it isn't competitive. It's just more "always on" culture in the workplace - and that's not healthy. A company should understand workers need real breaks - and being on call is not a real break.

Being on call sucks, clearly. But, the benefit of engineers oncall for their code is that it makes a pretty effective feedback loop --- the person who breaks the thing fixes the thing, and learns to break the thing less often, or to break it earlier in the business day so as not to ruin their evening, or to make it run better in degraded modes so it's ok to be broken for longer and alerts can be acknowledged amd dealt with later.

I like working in small teams because there's less required communication. Having the oncall be the engineer means the oncall doesn't have to communicate with the engineer, they're always up to date because it's one person (subject to sleep deprevation issues).

It's certainly not good for work/life balance though. Some production issues are unavoidable, automation for the common ones can help.

Edit to add: if you're oncall and your alerts are mostly because your dependencies are bad at their job, and you aren't empowered to do anything about that; having the engineer oncall isn't useful. It's only useful if the engineer is in a place to make changes to reduce future alerts.

Re: All of us test in production all the time (2019)

#70

Post by Lammy »

toast0 wrote: post #69
danpozmanter wrote: post #42

"Engineers should be on call for their own code." - Would you rather work someplace you are expected to be on call 24/7, or a company that doesn't require that? It isn't the norm, and it isn't competitive. It's just more "always on" culture in the workplace - and that's not healthy. A company should understand workers need real breaks - and being on call is not a real break.

Being on call sucks, clearly. But, the benefit of engineers oncall for their code is that it makes a pretty effective feedback loop --- the person who breaks the thing fixes the thing, and learns to break the thing less often, or to break it earlier in the business day so as not to ruin their evening, or to make it run better in degraded modes so it's ok to be broken for longer and alerts can be acknowledged amd deal…

And then that one person gets hit by a bus and you go out of business. Very-interconnected large-scale systems rarely have failure modes that are as simple as something the dev did/didn't do.
Post reply on HN