Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All of us test in production all the time (2019)

increment.com

Post reply on HN
Print view View original

141–150 of 164 posts

Re: All of us test in production all the time (2019)

#141

Post by Olreich »

faizshah wrote: post #118
siliconc0w wrote: post #89

So it's a fair point that the fidelity of non-prod environments are inherently limited and you still need a bunch of other stuff like canaries, automated canary analysis, automated rollbacks, zone fault-tolerance, feature-flags, chaos engineering, server and client-side instrumentation, but this is generally 'next-level' stuff when most shops aren't even getting the basics right. For the overwhelming majority of non-…

Wholeheartedly agree to everything you’ve said. I have worked at a lot of places where automated testing is almost non existent. Instead they use staging/testing environments and manual testing. For all these projects they frequently have downtime or errors in prod where something was uncaught. It takes ages to release a fix because they have to carefully research how a change affects other code instead of running a…

Self-documenting code is all code. The only thing that speaks to what a piece of code does is the code. Comments aren’t syntax, so they don’t help you there. I’ve never seen a codebase that does a good job of documenting the system architecture, expected usage patterns, and why the code is built the way it is. And I don’t know that having all that be inline to the code would be helpful anyway.

I find it far less error prone to just read the code and determine its purpose and allowable modifications from that. I am thinking about what a citation or footnote system might look like for code now though. That has the potential to provide context on-demand without needing to separate out all the pieces of the code into little silos that are hard to piece together.

Re: All of us test in production all the time (2019)

#142

Post by hampfi »

anon9001 wrote: post #120
faizshah wrote: post #118

Earlier quoted context omitted.

Wholeheartedly agree to everything you’ve said. I have worked at a lot of places where automated testing is almost non existent. Instead they use staging/testing environments and manual testing. For all these projects they frequently have downtime or errors in prod where something was uncaught. It takes ages to release a fix because they have to carefully research how a change affects other code instead of running a…

> They are convinced that unit tests are unnecessary extra work for their project but they don’t realize they are losing time by having to manually test and research changes. If people are finding tests unnecessary extra work, maybe they're right for their project. Most jobs are just "ship feature asap, try not to break anything, repeat". Most managers don't care about code quality as much as speed of delivery. Most…

Yeah no.

The biggest issue in all my dev work so far was a really shitty codebase which would work somehow and had and still have hidden big issues going on.

You just don't see it today perhaps and delivered it in 1 day instead of 3. But you will fix the issue in production half a year later and either have lost data, corrupted data or you have to correct data.

Not doing any tests, is literally shit.

Missing Unit tests also often means no proper CI/CD pipeline. Because yeah why would you need a proper build if you don't even execute tests anyway right?

IT IS always a clearly ignored requirement of doing proper work. Quality work. Do less, do better. Its the fault of the manager to not think about it and enforce it and its the fault of the developer to build shit.

Re: All of us test in production all the time (2019)

#143

Post by hampfi »

sagichmal wrote: post #88
bluedino wrote: post #73

Most places that I've worked, that "test in prod", do so for one reason. It all comes down to laziness. The first scenario is they have never set up a test environment in the first place. They're either too lazy to do so, or too lazy to look into how to do it. Often confused with being 'too busy to do it'. The second scenario is that they do have a testing environment-however, for some reason, it's broken. Some chang…

> The first scenario is they have never set up a test environment in the first place. They're either too lazy to do so, or too lazy to look into how to do it. Often confused with being 'too busy to do it'. One lesson of modern architectures (i.e. anything more recent than the LiveJournal-style Web/App/DB 3-tier stack) is that it is literally impossible to create and maintain a test environment that has enough similar…

Not sure where you are working but not a lot of people work at/on a huge code base.

Most codebases are build and operated by 1-3 teams and if you are not able to reproduce your prod env as a test env, your architecture is broken.

Re: All of us test in production all the time (2019)

#144

Post by hampfi »

joshuamorton wrote: post #117
ianamartin wrote: post #97

I love articles like this because it's so easy to just add that company to a list of places to never ever work. I did read the whole article, btw. It's an absolute clickbait title that the author doesn't really mean, and after the article spends a lot of time diffusing the clickbait title it really boils down to, "This is hard, so I give up." It's true that many--if not most--companies operate this way without ever a…

This works great if you're building something with a tightly controlled API. If, however, your configuration space grows to an even middling size, it no longer becomes feasible to do much of this validation across the configuration space. A good example is any system where the user can customize system aspects. Do you run all of your integration tests across the full configuration space? Additionally managing configu…

Thats a weird point you are making.

Yes i would test basic / standard customizations a customer would do.

I would test the customization system itself.

If it is to complex, your cusotmiztaion system will bring you much more issues later on.

Re: All of us test in production all the time (2019)

#145

Post by sagichmal »

hampfi wrote: post #143
sagichmal wrote: post #88

Earlier quoted context omitted.

> The first scenario is they have never set up a test environment in the first place. They're either too lazy to do so, or too lazy to look into how to do it. Often confused with being 'too busy to do it'. One lesson of modern architectures (i.e. anything more recent than the LiveJournal-style Web/App/DB 3-tier stack) is that it is literally impossible to create and maintain a test environment that has enough similar…

Not sure where you are working but not a lot of people work at/on a huge code base. Most codebases are build and operated by 1-3 teams and if you are not able to reproduce your prod env as a test env, your architecture is broken.

The size of the codebase is orthogonal to the size/complexity of the production environment.

Modern architectures look like this: https://github.com/donnemartin/system-design-primer

It is generally not feasible to recreate all of the elements of this design in a totally separate testing environment.

Re: All of us test in production all the time (2019)

#146

Post by sagichmal »

baddox wrote: post #101
sagichmal wrote: post #88

Earlier quoted context omitted.

> The first scenario is they have never set up a test environment in the first place. They're either too lazy to do so, or too lazy to look into how to do it. Often confused with being 'too busy to do it'. One lesson of modern architectures (i.e. anything more recent than the LiveJournal-style Web/App/DB 3-tier stack) is that it is literally impossible to create and maintain a test environment that has enough similar…

Can you explain that more? Aren’t most of the complex modern tools specifically designed to automate provisioning servers and deploying code?

Modern architectures look like this: https://github.com/donnemartin/system-design-primer

All of the pieces have complex and emergent behaviors as they interact with each other. Many (most?) are hosted, not even fully in control of the team using them. Recreating this environment in a hermetic way for testing is either actually impossible, or so cost prohibitive as to be practically infeasible.

Re: All of us test in production all the time (2019)

#147

Post by sagichmal »

UweSchmidt wrote: post #93
sagichmal wrote: post #88

Earlier quoted context omitted.

> The first scenario is they have never set up a test environment in the first place. They're either too lazy to do so, or too lazy to look into how to do it. Often confused with being 'too busy to do it'. One lesson of modern architectures (i.e. anything more recent than the LiveJournal-style Web/App/DB 3-tier stack) is that it is literally impossible to create and maintain a test environment that has enough similar…

As QA I think testing, test environments, test ressources should be first class concepts in software. Building whatever "modern" architecture twice, and simulating traffic is not easy but possible, at least to an extent. You need to get that extra license for any piece of software for testing during procurement, and provide means to create test ressources, like typically test users as needed. Sounds trivial, but is o…

Modern architectures look like this: https://github.com/donnemartin/system-design-primer

Building this out twice is either actually impossible, or so cost prohibitive as to be practically infeasible.

Re: All of us test in production all the time (2019)

#148

Post by TedDoesntTalk »

ratww wrote: post #78
TedDoesntTalk wrote: post #74

Earlier quoted context omitted.

> the age of the codebase is normally inversely proportional to its code quality. What??? You are saying that simply because a code base is old, it is of poor quality. There are numerous public examples contradicting this: Linux, many Apache projects such as httpd, etc. I emphatically disagree and would like to know why you think that.

I said "normally". I don't think anyone considers Linux or httpd "old tech", neither they are "legacy software" that experienced developers are running away from. C might be old as a language, but there are still modern things being built with it. I also said that the issue is not the age itself or the tech stack.

You didn't use the word legacy in your post. You used the word old. Quite different. Something can be old and still maintained (httpd, etc).

Re: All of us test in production all the time (2019)

#149

Post by faizshah »

Olreich wrote: post #141
faizshah wrote: post #118

Earlier quoted context omitted.

Wholeheartedly agree to everything you’ve said. I have worked at a lot of places where automated testing is almost non existent. Instead they use staging/testing environments and manual testing. For all these projects they frequently have downtime or errors in prod where something was uncaught. It takes ages to release a fix because they have to carefully research how a change affects other code instead of running a…

Self-documenting code is all code. The only thing that speaks to what a piece of code does is the code. Comments aren’t syntax, so they don’t help you there. I’ve never seen a codebase that does a good job of documenting the system architecture, expected usage patterns, and why the code is built the way it is. And I don’t know that having all that be inline to the code would be helpful anyway. I find it far less erro…

Self-documenting code is a total myth. Instead of the author documenting what a piece of code does, why it does it that way and how to use it everyone who works on that code has to spend hours looking through the commit history, merge requests, design docs, wiki history, asking around etc piecing together the history and conversations around the code.

Sometimes these artifacts don’t exist so then you have to go do the same research the author did to write the code just to figure out why the code exists...

Note that none of those activities are reading the code itself because you can do that in both scenarios.

Re: All of us test in production all the time (2019)

#150

Post by hampfi »

sagichmal wrote: post #145
hampfi wrote: post #143

Earlier quoted context omitted.

Not sure where you are working but not a lot of people work at/on a huge code base. Most codebases are build and operated by 1-3 teams and if you are not able to reproduce your prod env as a test env, your architecture is broken.

The size of the codebase is orthogonal to the size/complexity of the production environment. Modern architectures look like this: https://github.com/donnemartin/system-design-primer It is generally not feasible to recreate all of the elements of this design in a totally separate testing environment.

I know this very well and i'm stlil not seeing an issue.

We have 2 GCP accounts. One for production and one for test. Same Setup, not an issue due to terraform and the cost is reasonable for a test env.

Post reply on HN