Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All of us test in production all the time (2019)

increment.com

Post reply on HN
Print view View original

151–160 of 164 posts

Re: All of us test in production all the time (2019)

#151

Post by eitland »

sagichmal wrote: post #147
UweSchmidt wrote: post #93

Earlier quoted context omitted.

As QA I think testing, test environments, test ressources should be first class concepts in software. Building whatever "modern" architecture twice, and simulating traffic is not easy but possible, at least to an extent. You need to get that extra license for any piece of software for testing during procurement, and provide means to create test ressources, like typically test users as needed. Sounds trivial, but is o…

Modern architectures look like this: https://github.com/donnemartin/system-design-primer Building this out twice is either actually impossible, or so cost prohibitive as to be practically infeasible.

> Modern architectures look like this:

A lot of modern architecture is not like that.

And even the diagram there is something should be possible to replicate if the organization values it.

Today it doesn't even need to be too expensive as it can be deployed with terraform and torn down an hour later when the full system tests are finished qnd go back to running the integration tests.

Re: All of us test in production all the time (2019)

#152

Post by cagenut »

throwlogon wrote: post #51
cagenut wrote: post #43

Earlier quoted context omitted.

can confirm. media companies spent the last decade alternating between redesigns and cms migrations in a circular path that went nowhere while fb & goog ate not just their lunch but their breakfast and dinner too. the vast majority of it was driven by engineering-management-career resume-building and was actively detrimental to the editorial/content-production/journalism side of the house.

> [T]he vast majority of it was driven by engineering-management-career resume-building and was actively detrimental to the editorial/content-production/journalism side of the house. Why didn't the non-technology side prevent this if it was so clearly detrimental? I can't imagine that the technology side of a media concern has that much influence over the company's overall priorities.

I've tried to reply to this three times and its hard to without writing a book.

The shortest possible answer is that investors/owners/boards were watching what happened to sites like reddit/huffpo/bi/instagram/tumblr and salivating out of control at the word "billion". They believed that if they starved their editorial operations and bet most of the cashflow on growing their tech teams that "user generated content" would make up the difference. It was very simple math that said "media companies are valued at 10x profits, tech companies are valued at 10x revenue, lets do whatever we can to give the appearance of being a tech company not a media one".

Within those companies tech departments it became open season for CTOs, VPEngs, DevOps Directors and sr/lead/architect devs to engage in a peer-to-peer-ddos of architectural and tech-fad-chasing ones-up-manship. Because the only penalty for it all turning out to be too much was having to grow your headcount.

In 2016/2017 it all finally collapsed. Most of the companies have faded to layoffs and/or acquisitions. Many have basically frozen and given up on their tech stacks in-place. A large number have abandoned ship to wordpress (where imho they should have been all along).

Sometimes I wonder what the general state of "the media" or "journalism" would be now if 80% of that money had gone to hiring content creators, or even just paying the same number of content creators the 30% more it would take for it to be an adult job not a early-20-something job. An awful lot of the salaciousness and outrage-stoking in media right now is a byproduct of the actual jobs being very young people sitting in place all day garnering all their information from social media feeds and trying to hit their 2 - 5 posts (sorry 'articles') a day quota. We all think we're critiquing journalistic institutions when we point it out but really we're just yelling at kids. Blaming them for not being able to achieve the standards we're used to with 3 hour to deadline in their permalance gig, when what we got used to was written by mid-career people with benefits who had all day and sometimes week.

Media isn't stoking outrage culture, its clinging to it as the last source of pennies left before the nothing.

Re: All of us test in production all the time (2019)

#153

Post by sagichmal »

eitland wrote: post #151
sagichmal wrote: post #147

Earlier quoted context omitted.

Modern architectures look like this: https://github.com/donnemartin/system-design-primer Building this out twice is either actually impossible, or so cost prohibitive as to be practically infeasible.

> Modern architectures look like this: A lot of modern architecture is not like that. And even the diagram there is something should be possible to replicate if the organization values it. Today it doesn't even need to be too expensive as it can be deployed with terraform and torn down an hour later when the full system tests are finished qnd go back to running the integration tests.

It's not possible to replicate the CDN, or DNS, or the message queue if it's hosted, or the database of it's big enough, or etc. etc. The differences between what you're able to create in a staging environment and what exists in prod are significant enough that testing against the staging environment doesn't build much more confidence than testing with mocks.

Testing in prod means accepting this reality, avoiding the largely unproductive toil of building and maintaining a staging env, and making the production system observable, resilient, and operationally agile enough that you can deploy changes that can trigger unpredictable and emergent system behaviors while managing risk appropriately.

Re: All of us test in production all the time (2019)

#154

Post by sagichmal »

hampfi wrote: post #150
sagichmal wrote: post #145

Earlier quoted context omitted.

The size of the codebase is orthogonal to the size/complexity of the production environment. Modern architectures look like this: https://github.com/donnemartin/system-design-primer It is generally not feasible to recreate all of the elements of this design in a totally separate testing environment.

I know this very well and i'm stlil not seeing an issue. We have 2 GCP accounts. One for production and one for test. Same Setup, not an issue due to terraform and the cost is reasonable for a test env.

Same network rules and configuration? Same CDN settings? Same DNS? Same data in the databases? Same volume of traffic on the queues?

(Rhetorical questions. Of course the answer is no.)

All of these things have huge impact on the actual similarity of the systems, and consequently how they behave.

Re: All of us test in production all the time (2019)

#155

Post by m1117 »

site-packages1 wrote: post #19
m1117 wrote: post #14

Test in prod is terrible. Write unit and integration tests, test on staging, and then monitor Prod.

Sorry to make the assumption that you didn't read the article, but it really sounds like you didn't and are just making a comment about the headline. I would recommend reading the article.

Yes, I think the headline must reflect the content of the article, otherwise it confuses me and make me write false comments and wastes my time.

Re: All of us test in production all the time (2019)

#156

Post by mickotron »

Once upon a time I was a tester. Now I am a sys admin/DevOps engineer (what is a name, really?).

You would not believe the amount of bugs I would find from both vendor code and internal code. Bugs that require dynamic execution and context-spexific scenarios to reveal themselves. Then you had misinterpretations of requirements, which is not something the developers would find, considering they wrote the code and believed it met the requirements.

I worked for the police force of my region as a software tester for all their critical systems. I once developed a test tool that created packets to mimic mobile phone calls to police dispatch. I tested every combination of the data spec, including the very last one, where there was a critical bug. The vendor code had implemented the earlier version of the spec, not the latest. The vendor's test tool created the wrong packets as it targeted the wrong spec. Mine caught the bug cause it creates the correct packets.

As much as I agree with the author's ideas of familiarity with Production, proper unbiased and independent testing is still something this industry needs, even if it's unfashionable.

Re: All of us test in production all the time (2019)

#157

Post by _diq5 »

ratww wrote: post #67
_diq5 wrote: post #55

Earlier quoted context omitted.

Want to jump in here - I have worked at a company where engineers are not on call for their code, and it was a living nightmare. _You_ might not be on call for your code, but _somebody_ will be. Often some poor SRE/ops person that has absolutely no idea what the app is doing/or why it's failing in production. Not being on-call makes engineers complicit. I've seen it all, known memory leaks shipped into production, ap…

I'm always happy to help some poor SRE in the middle of the night, and I once even drove to the office in a rainy Sunday, in the middle of my vacation, to access IP-restricted stuff because a support intern messaged me on Instagram. ...but with that said: I'm glad I only worked in countries where work is properly regulated and "on call" means "I'm getting fucking paid every cent for each hour I _must_ answer that god…

I unfortunately work in a place where on-call is unpaid. I'm an SRE stuck in the 90s.

The policy states that only the Operations team gets paid on-call, because I guess in the old days they would be the expected to deal with production.

Fast forward to today, and the Operations folks are a small team managing 2 datacentres, and all on-call rotations between SREs and developers are considered unofficial and therefore not eligible to be paid.

One of our Sr. Managers tried to take this up the chain, but then got reprimanded for putting developers on-call.

Re: All of us test in production all the time (2019)

#158

Post by nitrogen »

lmm wrote: post #125
josephg wrote: post #124

Earlier quoted context omitted.

When I’ve gone full TDD on projects, I’m always surprised to find there’s a power law distribution or something on failing tests. Most of the tests I write never catch a bug in the life of the software. Something like 90% of the value of a test suite could be achieved with only about 10% of the tests. Of course, the trick is figuring out which tests are going to repeatedly fail ahead of time. But there’s something in…

You'd get a lot of the value by just being willing to delete tests once it's clear they're not serving any purpose. Sadly a lot of people would rather see a high test coverage percentage than have an effective test suite.

What is the value of deleting a test once it's written, though?

Re: All of us test in production all the time (2019)

#159

Post by lmm »

nitrogen wrote: post #158
lmm wrote: post #125

Earlier quoted context omitted.

You'd get a lot of the value by just being willing to delete tests once it's clear they're not serving any purpose. Sadly a lot of people would rather see a high test coverage percentage than have an effective test suite.

What is the value of deleting a test once it's written, though?

Tests impose a maintenance burden like any other lines of code. Just making it harder to navigate to relevant code (including useful tests) is a significant cost.

Re: All of us test in production all the time (2019)

#160

Post by hampfi »

sagichmal wrote: post #154
hampfi wrote: post #150

Earlier quoted context omitted.

I know this very well and i'm stlil not seeing an issue. We have 2 GCP accounts. One for production and one for test. Same Setup, not an issue due to terraform and the cost is reasonable for a test env.

Same network rules and configuration? Same CDN settings? Same DNS? Same data in the databases? Same volume of traffic on the queues? (Rhetorical questions. Of course the answer is no.) All of these things have huge impact on the actual similarity of the systems, and consequently how they behave.

Obviously but this is not a black and white thing.

If it wouldn't make sense for us to have test and prod, we wouldn't do it but i can't imagine a scenario where this is NOT beneficial.

Post reply on HN