Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All of us test in production all the time (2019)

increment.com

Post reply on HN
Print view View original

51–60 of 164 posts

Re: All of us test in production all the time (2019)

#51

Post by throwlogon »

cagenut wrote: post #43
KaiserPro wrote: post #39

Earlier quoted context omitted.

Take a new site for example Its just text and pictures on a web page. The new york times re-invented a database out of kafka to power theirs (mind bogglingly stupid) The Guardian re-wrote their CMS at least 4 times, at a cost of well over 50 million, if not more. Thats before they got to re-doing the layout. The FT spent 6(!) years rebuilding their entire stack, (150 tech staff for 6 years. a good percentage contract…

can confirm. media companies spent the last decade alternating between redesigns and cms migrations in a circular path that went nowhere while fb & goog ate not just their lunch but their breakfast and dinner too. the vast majority of it was driven by engineering-management-career resume-building and was actively detrimental to the editorial/content-production/journalism side of the house.

> [T]he vast majority of it was driven by engineering-management-career resume-building and was actively detrimental to the editorial/content-production/journalism side of the house.

Why didn't the non-technology side prevent this if it was so clearly detrimental? I can't imagine that the technology side of a media concern has that much influence over the company's overall priorities.

Re: All of us test in production all the time (2019)

#52

Post by mgkimsal »

aunty_helen wrote: post #32

>Nobody invests in their “test in prod” tooling. Firstly, what is logging then? How is this not tooling to ensure things are running smoothly? `less +F` anyone? Secondly, if you're running an aws/azure/gcp based server you now have a ridiculous amount of tooling for production diag, analytics and tracing.

"Firstly, what is logging then?"

"Secondly, if you're running an aws/azure/gcp based server you now have a ridiculous amount of tooling for production diag, analytics and tracing."

This presume that you, the person expected to fix problem X actually have access to logs and the servers where problems are happening. I've been at multiple projects/companies where this simply Isn't Allowed(tm).

"You wrote this code, you need to fix it! It's broken!"

"Let me get on the server and take a look at the logs to see what's going on".

"That violates our security policy! You can't do that!"

I was tasked with 'finding a problem' and was told to look in the logs. We knew what day the problem was, but I couldn't get anyone to confirm if the log files were in UTC or something else (turns out it was something else).

HOWEVER, I never actually got the actual log files. I didn't have access directly (that takes about a week to go through the chain of command to approve), so someone just sent me small snippets from where they thought the problem might be.

So even orgs that tick all their checkboxes of stats/analytics/logging... sometimes seem to forget that access to the collected info is a requirement too.

Re: All of us test in production all the time (2019)

#53

Post by mxz3000 »

danpozmanter wrote: post #42

"Engineers should be on call for their own code." - Would you rather work someplace you are expected to be on call 24/7, or a company that doesn't require that? It isn't the norm, and it isn't competitive. It's just more "always on" culture in the workplace - and that's not healthy. A company should understand workers need real breaks - and being on call is not a real break.

When you're Oncall you aren't Oncall 24/7 forever. It usually rotates amongst the engineers in the team. I'm on a team with about 10 engineers in it so you're on call about a week every two months. I call that manageable.

Engineers should 100% be responsible for owning their code, and fixing any issues that arises from it. After all they're the ones that wrote it, aren't they the best people to fix it when it breaks?

Re: All of us test in production all the time (2019)

#54

Post by KaiserPro »

throwlogon wrote: post #51
cagenut wrote: post #43

Earlier quoted context omitted.

can confirm. media companies spent the last decade alternating between redesigns and cms migrations in a circular path that went nowhere while fb & goog ate not just their lunch but their breakfast and dinner too. the vast majority of it was driven by engineering-management-career resume-building and was actively detrimental to the editorial/content-production/journalism side of the house.

> [T]he vast majority of it was driven by engineering-management-career resume-building and was actively detrimental to the editorial/content-production/journalism side of the house. Why didn't the non-technology side prevent this if it was so clearly detrimental? I can't imagine that the technology side of a media concern has that much influence over the company's overall priorities.

because editorial say: "we want the website to do x" the tech team say ok we'll need 100 staff, and six months.

then, after delivery, the tech team "we need 50% more staff to do this thing, its really good, it'll save 25% in our hosting costs"

Editorial then say: "sure, deliver this feature as well"

This continues, and then deadlines are missed. So tech say: "We have so much tech debt we need to re-architect" which allows ego/CV driven design.

Re: All of us test in production all the time (2019)

#55

Post by _diq5 »

danpozmanter wrote: post #42

"Engineers should be on call for their own code." - Would you rather work someplace you are expected to be on call 24/7, or a company that doesn't require that? It isn't the norm, and it isn't competitive. It's just more "always on" culture in the workplace - and that's not healthy. A company should understand workers need real breaks - and being on call is not a real break.

Want to jump in here - I have worked at a company where engineers are not on call for their code, and it was a living nightmare.

_You_ might not be on call for your code, but _somebody_ will be. Often some poor SRE/ops person that has absolutely no idea what the app is doing/or why it's failing in production.

Not being on-call makes engineers complicit. I've seen it all, known memory leaks shipped into production, apps where half the endpoints couldn't even be compiled, code dumping the production redis at 1AM ... and every time the pain just felt on deaf ears.

If your code is what wakes you up in the middle of the night, you have: - Incentive to fix/mitigate as soon as possible. - No blame game to play. Either the error was made by you, or someone on your team. It doesn't have to go up 3 rungs on the ladder then back down again.

I don't think the author was suggesting that everyone should always be on call, just that you _must_ be responsible for your own code in production

Re: All of us test in production all the time (2019)

#56

Post by dmurray »

chrischattin wrote: post #48
tluyben2 wrote: post #36

Earlier quoted context omitted.

I had the same issue with Rails as I have with Node : the churn is detriment to long standing projects. Most projects I do run for years and even decades; you go run an update for security for Rails code that is 10 years old. It is a nightmare. I did not have issues like that with php, asp.net or spring.

My experience has been the opposite. The Rails ecosystem makes updates straight forward and is very well documented. Been running various Rails apps for almost a decade now.

It's pretty good for incremental updates. It only asks a few hours a year to manage those.

Not so good when you want to upgrade a 5-year-old system. I inherited a reasonably well-built Rails app deployed with Ruby 2.2.1 and would like to update it, if only for security reasons. I can't even run that on a recent Ubuntu. On a system where I can run it, I run into dependency hell with the gems - need to upgrade some, pin lower versions of others. Possibly that could have been handled with pinned dependency versions the first time round, but that breaks the regular upgrade process.

Re: All of us test in production all the time (2019)

#57

Post by Lammy »

chrischattin wrote: post #48
tluyben2 wrote: post #36

Earlier quoted context omitted.

I had the same issue with Rails as I have with Node : the churn is detriment to long standing projects. Most projects I do run for years and even decades; you go run an update for security for Rails code that is 10 years old. It is a nightmare. I did not have issues like that with php, asp.net or spring.

My experience has been the opposite. The Rails ecosystem makes updates straight forward and is very well documented. Been running various Rails apps for almost a decade now.

Straight-forward and documented is the minimum. Not having to think about it at all ever would be ideal. At some (huge) scale it will eventually become a thing that takes a year and a half and gets blog posts written about it, and I never enjoy losing a weekend to a microcosm of that: https://github.blog/2018-09-28-upgrading-github-from-rails-3...

Re: All of us test in production all the time (2019)

#58

Post by ratww »

nthj wrote: post #33
Nbox9 wrote: post #23

Earlier quoted context omitted.

If you like Ruby on Rails so much, why don’t you find a job programming Ruby on Rails? There are so many.

My experience is that nobody actually does things The Rails Way, which is where most of the benefits come in. No, they’ll start out in Rails such that their underlying data model evolves into a huge mess (because Rails guards against most problems in the code layer, so the data model problems don’t really appear as bugs yet), and then some resume-driven developer will push for serverless or microservices to solve sca…

The biggest waste of time I've ever seen in my career was when a team put all their backend efforts into migrating their app from The Rails Way to Services, then from Services to Trailblazer, then from Trailblazer to ActiveInteraction. All in the span of two years and without delivering a finished product.

In the end the company just bought a competitor and gave up on the app.

100% agree that doing things The Rails Way is where you reap the benefits.

Re: All of us test in production all the time (2019)

#59

Post by bob1029 »

tluyben2 wrote: post #36
vinceguidry wrote: post #15

A lot of this boils down to the constant churn in frameworks, instrumentation, programming languages. Developers don't have time to master one way of doing things, it's just a constant lava layer of crap. I took the time to learn how to build and maintain Ruby on Rails systems, hoping it would be the ticket to a fun, manageable career. Any project I worked on was an island of sustainable, fast development where the t…

I had the same issue with Rails as I have with Node : the churn is detriment to long standing projects. Most projects I do run for years and even decades; you go run an update for security for Rails code that is 10 years old. It is a nightmare. I did not have issues like that with php, asp.net or spring.

This is the primary reason why I love the .NET stack so much. Not only is a security update almost universally a non-event, but so are major framework revisions.

Updating .NET applications for security concerns in many cases boils down to letting windows update do the thing it should already be doing automatically. In a much smaller portion of cases for .NET Core, a security update is a matter of rebuilding a self-contained deployment and pushing it out to production. Neither of these cases requires editing any code or configuration.

Only in the transition from .NET Framework to .NET Core did we find any appreciable difficulty, and even then it was still within the realm of reasonable. Most of the pain here boiled down to usages of System.Drawing and DirectoryServices, both of which have alternatives that are supported on all platforms. We opted to use the compatibility pack instead, but would be OK if that was removed at some point and had to replace with the other implementations.

We have lots of sourcecode that has withstood the test of time because of these technology choices. Many files are completely unchanged since .NET Framework 4.5 was released and now running flawlessly on top of .NET Core 3.1. This is the only reason we are still in business right now.

I ended my journey with NodeJS (and AngularJS on the same day) when I attempted to update all of my packages after just a month of inactivity. Seeing the resulting pile of nonsensical dependency graph trash in my console may have radicalized me into using more "enterprisey" solutions when writing software that I might get put on pager duty for.

Re: All of us test in production all the time (2019)

#60

Post by vinceguidry »

Nbox9 wrote: post #23
vinceguidry wrote: post #15

A lot of this boils down to the constant churn in frameworks, instrumentation, programming languages. Developers don't have time to master one way of doing things, it's just a constant lava layer of crap. I took the time to learn how to build and maintain Ruby on Rails systems, hoping it would be the ticket to a fun, manageable career. Any project I worked on was an island of sustainable, fast development where the t…

If you like Ruby on Rails so much, why don’t you find a job programming Ruby on Rails? There are so many.

After it happened twice, I just gave in to the inevitable and decided to value jobs based on other factors.
Post reply on HN