Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All of us test in production all the time (2019)

increment.com

Post reply on HN
Print view View original

111–120 of 164 posts

Re: All of us test in production all the time (2019)

#111

Post by isbvhodnvemrwvn »

baddox wrote: post #101
sagichmal wrote: post #88

Earlier quoted context omitted.

> The first scenario is they have never set up a test environment in the first place. They're either too lazy to do so, or too lazy to look into how to do it. Often confused with being 'too busy to do it'. One lesson of modern architectures (i.e. anything more recent than the LiveJournal-style Web/App/DB 3-tier stack) is that it is literally impossible to create and maintain a test environment that has enough similar…

Can you explain that more? Aren’t most of the complex modern tools specifically designed to automate provisioning servers and deploying code?

I'm not the person you're responding to, but having the same software and hardware is possible (most of the time), but production environment is a living thing. To simulate production you need the same data and interaction you have in production, and that's quite difficult to do in many systems, if only for regulatory reasons (PII, PCI and friends).

Re: All of us test in production all the time (2019)

#112

Post by inopinatus »

dang wrote: post #96
contingencies wrote: post #81

Earlier quoted context omitted.

We all test in prod

Ok, there's actually a phrase from the article that says that, so we'll put that up there and see what happens.

For us curious latecomers, what was the original title?

Re: All of us test in production all the time (2019)

#113

Post by perl4ever »

jagraff wrote: post #108
mgkimsal wrote: post #107

Earlier quoted context omitted.

what I learned was that the logs are timestamped to whatever machine they're on, wherever it is. The servers in London - GMT. The servers in west coast - Pacfic time. Servers in DC - Eastern time. Not ideal.

Right - assuming the server IP, or at least some other network identifying factor, is in the log, you could write some sort of regex to parse the logs and identify the correct time. Of course, that depends on someone being able to actually parse the logs with your regex

I once found a solution to determining the timezone when it wasn't available - a field with only a date was unnecessarily and incorrectly converted to UT, meaning that the zone could be derived from the offset from midnight and applied to other fields.

If enough things are screwed up, you can possibly find the solution to one issue in another.

Re: All of us test in production all the time (2019)

#114

Post by table-delete »

smoe wrote: post #110
siliconc0w wrote: post #89

So it's a fair point that the fidelity of non-prod environments are inherently limited and you still need a bunch of other stuff like canaries, automated canary analysis, automated rollbacks, zone fault-tolerance, feature-flags, chaos engineering, server and client-side instrumentation, but this is generally 'next-level' stuff when most shops aren't even getting the basics right. For the overwhelming majority of non-…

Agreed I like the view i picked up somewhere but don't remember from whom: Tests are not there to prove something is working, but to prove something is not working. They are a fairly cheap way to tell me during development that this ain't going to work no matter what. And if everything compiles and the tests and reviews pass, I think this has a good shot at working out in production, but there still might well be cas…

Glenford Myers in "The Art of Software Testing" said: "Testing is the process of executing a program with the intent of finding errors."

The preface to this is even more enlightening:

"When you test a program, you want to add some value to it. Adding value through testing means raising the quality or reliability of the program. Raising the reliability of the program means finding and removing errors.

Therefore, don’t test a program to show that it works; rather, you should start with the assumption that the program contains errors (a valid assumption for almost any program) and then test the program to find as many of the errors as possible."

Re: All of us test in production all the time (2019)

#115

Post by inopinatus »

Apropos of this in the context of hiring, one of the biggest green flags I'll attach to someone's CV is any combination of infrastructure/operations, and application development, at the same job.

Since the 1990s I've worked around ISP, hosting, and cloud firms. Many have a core of general purpose people that can't help themselves but have one foot in both graves (we call this DevOps or SRE now, but those are new labels for a long-standing viewpoint). This often correlates to someone who embodies the mindset in the article, viz. that they they will gladly and actively defend any ditch we dig together. They always get on my interview list (the green flag). Very often the interviews are a wide-ranging, free-flowing, and in-depth discussion of multilateral & cross-functional technology/process interactions.

Many of the people I've met with this combination will progress, either immediately or eventually, to become very effective CTOs, tech co-founders, or the highest levels of IC at larger tech firms.

Interestingly, and I say this purely anecdotally because I am not actually qualified to make the diagnosis, some of them also appear to me to have an attentional difference, or present from an unprivileged background, and may not have followed a standard educational path as a result of either. Which is to say that I usually delete "must have bachelors degree" from any JD that HR ask me to authorize.

Re: All of us test in production all the time (2019)

#117

Post by joshuamorton »

ianamartin wrote: post #97

I love articles like this because it's so easy to just add that company to a list of places to never ever work. I did read the whole article, btw. It's an absolute clickbait title that the author doesn't really mean, and after the article spends a lot of time diffusing the clickbait title it really boils down to, "This is hard, so I give up." It's true that many--if not most--companies operate this way without ever a…

This works great if you're building something with a tightly controlled API.

If, however, your configuration space grows to an even middling size, it no longer becomes feasible to do much of this validation across the configuration space. A good example is any system where the user can customize system aspects. Do you run all of your integration tests across the full configuration space?

Additionally managing configuration skew between a dev and prod environment is not simple. Simply claiming that there should be no skew doesn't work. Often you want the prod and dev environments to run as different users, and you certainly want them to have different acls (your dev environment should not have access to your production database).

So you now have to, across your configuration space, validate that only the things that are "supposed" to be different differ, and that the things that aren't don't. Which maybe works for a while, but your prod configuration may also differ across parts of prod if, for example, a change is being canaried or incrementally deployed.

I've spent a non-trivial amount of effort on trying to solve the one problem of configuration skew between dev and prod for one real system. It's ultimately not worth it. The effort expended to "fix" that would be more work, than not. And I mean that in the long term, the effort to maintain and follow the rules that such a system would impose is more effort than dealing with the annoyances of unintended skews.

Systems are hard because systems are hard. There's no good company that doesn't, test/experiment in production. All of them do.

Re: All of us test in production all the time (2019)

#118

Post by faizshah »

siliconc0w wrote: post #89

So it's a fair point that the fidelity of non-prod environments are inherently limited and you still need a bunch of other stuff like canaries, automated canary analysis, automated rollbacks, zone fault-tolerance, feature-flags, chaos engineering, server and client-side instrumentation, but this is generally 'next-level' stuff when most shops aren't even getting the basics right. For the overwhelming majority of non-…

Wholeheartedly agree to everything you’ve said.

I have worked at a lot of places where automated testing is almost non existent. Instead they use staging/testing environments and manual testing.

For all these projects they frequently have downtime or errors in prod where something was uncaught. It takes ages to release a fix because they have to carefully research how a change affects other code instead of running a test suite.

They are convinced that unit tests are unnecessary extra work for their project but they don’t realize they are losing time by having to manually test and research changes. I think a lot of the focus on fancy testing practices has lead some programmers to think testing is a load of extra work when things will fail in prod anyway. We need to push back and say automated tests are a way to codify assumptions about the code and help speed up the prototype-build-test-release loop.

Side-note: self documenting code is a myth but I digress.

Re: All of us test in production all the time (2019)

#119

Post by faizshah »

table-delete wrote: post #114
smoe wrote: post #110

Earlier quoted context omitted.

Agreed I like the view i picked up somewhere but don't remember from whom: Tests are not there to prove something is working, but to prove something is not working. They are a fairly cheap way to tell me during development that this ain't going to work no matter what. And if everything compiles and the tests and reviews pass, I think this has a good shot at working out in production, but there still might well be cas…

Glenford Myers in "The Art of Software Testing" said: "Testing is the process of executing a program with the intent of finding errors." The preface to this is even more enlightening: "When you test a program, you want to add some value to it. Adding value through testing means raising the quality or reliability of the program. Raising the reliability of the program means finding and removing errors. Therefore, don’t…

I like this quote from Kent Beck:

“ It is impossible to test absolutely everything, without the tests being as complicated and error-prone as the code. It is suicide to test nothing (in this sense of isolated, automatic tests). So, of all the things you can imagine testing, what should you test?

You should test things that might break. If code is so simple that it can't possibly break, and you measure that the code in question doesn't actually break in practice, then you shouldn't write a test for it...

Testing is a bet. The bet pays off when your expectations are violated [when a test that you expect to pass fails, or when a test that you expect to fail passes]... So, if you could, you would only write those tests that pay off. Since you can't know which tests would pay off (if you did, then you would already know and you wouldn't be learning anything), you write tests that might pay off. As you test, you reflect on which kinds of tests tend to pay off and which don't, and you write more of the ones that do pay off, and fewer of the ones that don't.”

Source: https://softwareengineering.stackexchange.com/a/244709

Re: All of us test in production all the time (2019)

#120

Post by anon9001 »

faizshah wrote: post #118
siliconc0w wrote: post #89

So it's a fair point that the fidelity of non-prod environments are inherently limited and you still need a bunch of other stuff like canaries, automated canary analysis, automated rollbacks, zone fault-tolerance, feature-flags, chaos engineering, server and client-side instrumentation, but this is generally 'next-level' stuff when most shops aren't even getting the basics right. For the overwhelming majority of non-…

Wholeheartedly agree to everything you’ve said. I have worked at a lot of places where automated testing is almost non existent. Instead they use staging/testing environments and manual testing. For all these projects they frequently have downtime or errors in prod where something was uncaught. It takes ages to release a fix because they have to carefully research how a change affects other code instead of running a…

> They are convinced that unit tests are unnecessary extra work for their project but they don’t realize they are losing time by having to manually test and research changes.

If people are finding tests unnecessary extra work, maybe they're right for their project.

Most jobs are just "ship feature asap, try not to break anything, repeat". Most managers don't care about code quality as much as speed of delivery. Most coders don't have much "skin in the game" and can always get another job when the project fails.

Writing tests is a sign that you know what you're building, you have a methodology for building it, and you have a commitment to keeping it working in the future. That's simply not the requirements for most programming work, and it can be a disservice to both the coders and the management to push for testing that won't help achieve the business goals. If a business needs to iterate fast and reliably, write tests. If a business needs to iterate fast and is ok with breaking things, skip the tests and have a rollback plan instead.

Post reply on HN