All of us test in production all the time (2019) • Hacker News Board

Re: All of us test in production all the time (2019)

#31

Post by bdcravens » Sun, Jul 19, 2020, 4:45 PM UTC

hn_throwaway_99 wrote: post #2

This article is inane to me. Someone decided to write a lengthy blog post on the deliberate and obvious misinterpretation of a meme. DevOps and Observability are well described disciplines. I don't know anyone who has uttered the meme "I don't always test, but when I do, I test in production" who meant that monitoring, observing and reviewing prod behavior is bad. They invariably meant that they were rushed to releas…

I didn't get that. It seems to me the point was there's too many developers feel their responsibility ends when they hit deploy.

Re: All of us test in production all the time (2019)

#32

Post by aunty_helen » Sun, Jul 19, 2020, 4:46 PM UTC

>Nobody invests in their “test in prod” tooling.

Firstly, what is logging then?

How is this not tooling to ensure things are running smoothly? `less +F` anyone?

Secondly, if you're running an aws/azure/gcp based server you now have a ridiculous amount of tooling for production diag, analytics and tracing.

Re: All of us test in production all the time (2019)

#33

Post by nthj » Sun, Jul 19, 2020, 4:47 PM UTC

Nbox9 wrote: post #23

vinceguidry wrote: post #15

A lot of this boils down to the constant churn in frameworks, instrumentation, programming languages. Developers don't have time to master one way of doing things, it's just a constant lava layer of crap. I took the time to learn how to build and maintain Ruby on Rails systems, hoping it would be the ticket to a fun, manageable career. Any project I worked on was an island of sustainable, fast development where the t…

If you like Ruby on Rails so much, why don’t you find a job programming Ruby on Rails? There are so many.

My experience is that nobody actually does things The Rails Way, which is where most of the benefits come in. No, they’ll start out in Rails such that their underlying data model evolves into a huge mess (because Rails guards against most problems in the code layer, so the data model problems don’t really appear as bugs yet), and then some resume-driven developer will push for serverless or microservices to solve scalability problems (when the actual bottleneck is the database and the data model, not Rails), and now you’re left managing this Frankensteinish “rails” SPA beast that could be rewritten as a proper database schema and a 5K LOC Rails app, but there’s too much change control process to actually achieve that.

Re: All of us test in production all the time (2019)

#34

Post by 29athrowaway » Sun, Jul 19, 2020, 4:49 PM UTC

I usually try to make all my development environments the same. Make them differ only in terms of infrastructure, not code.

The more code differences you have between environments, the less valuable pre-production testing becomes.

Re: All of us test in production all the time (2019)

#35

Post by tluyben2 » Sun, Jul 19, 2020, 4:54 PM UTC

throwlogon wrote: post #29

vinceguidry wrote: post #15

A lot of this boils down to the constant churn in frameworks, instrumentation, programming languages. Developers don't have time to master one way of doing things, it's just a constant lava layer of crap. I took the time to learn how to build and maintain Ruby on Rails systems, hoping it would be the ticket to a fun, manageable career. Any project I worked on was an island of sustainable, fast development where the t…

> I guess stable just wasn't good enough I suspect that's meant to be rhetorical, but from a business perspective, was stable good enough? Sometimes what works now won't be competitive in the near future.

> Sometimes what works now won't be competitive in the near future.

What do you mean by that? I can understand in terms of hiring devs, this is true, but any other reason I would be curious about. Because I work with/in many companies, I see a lot of different tech and competitive is usually just what the team knows best. So unless you are making a new team, I cannot phantom why you would switch.

Java with Servlets/Spring and C# with asp.net have been competitive for almost 2 decades; maybe some people can shave off a few minutes here and there but in the grand scheme of a serious company with a serious project (500k loc+) it is not going to matter. There are exceptions as for instance machine learning but those gaps close and this was about web frameworks.

Re: All of us test in production all the time (2019)

#36

Post by tluyben2 » Sun, Jul 19, 2020, 4:58 PM UTC

vinceguidry wrote: post #15

A lot of this boils down to the constant churn in frameworks, instrumentation, programming languages. Developers don't have time to master one way of doing things, it's just a constant lava layer of crap. I took the time to learn how to build and maintain Ruby on Rails systems, hoping it would be the ticket to a fun, manageable career. Any project I worked on was an island of sustainable, fast development where the t…

I had the same issue with Rails as I have with Node : the churn is detriment to long standing projects. Most projects I do run for years and even decades; you go run an update for security for Rails code that is 10 years old. It is a nightmare. I did not have issues like that with php, asp.net or spring.

Re: All of us test in production all the time (2019)

#37

Post by Nbox9 » Sun, Jul 19, 2020, 4:59 PM UTC

mandelbrotwurst wrote: post #30

Nbox9 wrote: post #23

Earlier quoted context omitted.

If you like Ruby on Rails so much, why don’t you find a job programming Ruby on Rails? There are so many.

He mentioned that he had a job doing that, but the company moved away from the framework without technical justification. He could find another, but there would be no guarantee that the same thing wouldn't happen again.

> the company moved away from the framework without technical justification

I think this alone is enough justification to look for a new place to work, but I’m just a job hopping millennial.

Re: All of us test in production all the time (2019)

#38

Post by 0xdeadbeefbabe » Sun, Jul 19, 2020, 5:00 PM UTC

> We’re a startup. Startups don’t tend to fail because they moved too fast. They tend to fail because they obsess over trivialities that don’t actually provide business value. It was important that we reach a reasonable level of confidence, handle errors, and have multiple levels of fail-safes (i.e., backups)."

Do established companies, or stopdown, fail because they moved too fast? Can we dismiss Charity's astonishingly good advice over perceived startupiness? That's dumb.

Re: All of us test in production all the time (2019)

#39

Post by KaiserPro » Sun, Jul 19, 2020, 5:06 PM UTC

throwlogon wrote: post #29

vinceguidry wrote: post #15

A lot of this boils down to the constant churn in frameworks, instrumentation, programming languages. Developers don't have time to master one way of doing things, it's just a constant lava layer of crap. I took the time to learn how to build and maintain Ruby on Rails systems, hoping it would be the ticket to a fun, manageable career. Any project I worked on was an island of sustainable, fast development where the t…

> I guess stable just wasn't good enough I suspect that's meant to be rhetorical, but from a business perspective, was stable good enough? Sometimes what works now won't be competitive in the near future.

Take a new site for example

Its just text and pictures on a web page. The new york times re-invented a database out of kafka to power theirs (mind bogglingly stupid)

The Guardian re-wrote their CMS at least 4 times, at a cost of well over 50 million, if not more. Thats before they got to re-doing the layout.

The FT spent 6(!) years rebuilding their entire stack, (150 tech staff for 6 years. a good percentage contractors. > £70million)

All of it is just fucking text on a fucking web page. All that effort to add fancy gizmos for hosting, optimising for framework x, which has a half life of 6 months, adding animation to drop downs. Just a spectacular waste. They are still re-platforming every 9 months, each of the 5 major teams.

All of that money could have been put into content, advertising for customers, collaborations & events.

Most news sites can get away with a static, highly cached system (I mean look at the Daily mail, the page is slow, static and looks like shit, yet its the biggest new paper site online)

So form a business perceptive, its spaffing money up the wall for no real gain.

Re: All of us test in production all the time (2019)

#40

Post by cagenut » Sun, Jul 19, 2020, 5:08 PM UTC

burntoutfire wrote: post #20

vinceguidry wrote: post #15

A lot of this boils down to the constant churn in frameworks, instrumentation, programming languages. Developers don't have time to master one way of doing things, it's just a constant lava layer of crap. I took the time to learn how to build and maintain Ruby on Rails systems, hoping it would be the ticket to a fun, manageable career. Any project I worked on was an island of sustainable, fast development where the t…

> After it happened twice, well, might as well go into devops. My observation of the devops world is that the rotation of technologies and frameworks there is faster than in backend development.

True but imho its somewhat of a fan-out or at the very least knock-on effect. The explosion in devops complexity is a necessary byproduct of the explosion of projects that adopt new languages, runtimes, datastores, etc. No one ever fully deletes and retires the old system, so the overall environmental complexity grows exponentially. The tools it takes to solve those exponentially larger problems (k8s) look ridiculous when you consider them in the context of "a rails app with some node stuff". But they make perfect sense when you say "every quarter or two, for a decade, a dev team has adopted a new component or two into the stack. less than three of which have ever been deleted."