Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All of us test in production all the time (2019)

increment.com

Post reply on HN
Print view View original

101–110 of 164 posts

Re: All of us test in production all the time (2019)

#101

Post by baddox »

sagichmal wrote: post #88
bluedino wrote: post #73

Most places that I've worked, that "test in prod", do so for one reason. It all comes down to laziness. The first scenario is they have never set up a test environment in the first place. They're either too lazy to do so, or too lazy to look into how to do it. Often confused with being 'too busy to do it'. The second scenario is that they do have a testing environment-however, for some reason, it's broken. Some chang…

> The first scenario is they have never set up a test environment in the first place. They're either too lazy to do so, or too lazy to look into how to do it. Often confused with being 'too busy to do it'. One lesson of modern architectures (i.e. anything more recent than the LiveJournal-style Web/App/DB 3-tier stack) is that it is literally impossible to create and maintain a test environment that has enough similar…

Can you explain that more? Aren’t most of the complex modern tools specifically designed to automate provisioning servers and deploying code?

Re: All of us test in production all the time (2019)

#102

Post by tluyben2 »

ratww wrote: post #75
rybosworld wrote: post #65

Earlier quoted context omitted.

Node is getting better each year imo. I've been pretty happy with ES6. What makes you say Babel is horrible? Not disagreeing but I haven't heard anyone say that before.

People don't say it directly, but all the constant complaining about the bloat and complexity of the JS ecosystem and the size of node_modules folder boils down mostly to two tools: Babel and Webpack.

But that is a sign is it not? If tools as badly designed as webpack is the standard, is that not saying there is an issue with the entire ecosystem?

Re: All of us test in production all the time (2019)

#103

Post by pydry »

ratww wrote: post #75
rybosworld wrote: post #65

Earlier quoted context omitted.

Node is getting better each year imo. I've been pretty happy with ES6. What makes you say Babel is horrible? Not disagreeing but I haven't heard anyone say that before.

People don't say it directly, but all the constant complaining about the bloat and complexity of the JS ecosystem and the size of node_modules folder boils down mostly to two tools: Babel and Webpack.

When people make fun of leftpad they are not talking about babel and webpack.

The bloat is a lot about the lack of a standard library in JS. Or rather, too many of the damn things. That and an inconsistent and unpredictable type system.

Re: All of us test in production all the time (2019)

#104

Post by tluyben2 »

dmurray wrote: post #56
chrischattin wrote: post #48

Earlier quoted context omitted.

My experience has been the opposite. The Rails ecosystem makes updates straight forward and is very well documented. Been running various Rails apps for almost a decade now.

It's pretty good for incremental updates. It only asks a few hours a year to manage those. Not so good when you want to upgrade a 5-year-old system. I inherited a reasonably well-built Rails app deployed with Ruby 2.2.1 and would like to update it, if only for security reasons. I can't even run that on a recent Ubuntu. On a system where I can run it, I run into dependency hell with the gems - need to upgrade some, pi…

That is indeed what I meant; I had to create debootstraps to even get it running on my laptop. And many of the gem updates broke the apis completely for no reason (move fast and don’t care?) I do not know: I remember a security update where the new version renamed most of the api methods while there was no reason to do so. I understand you can add new methods to keep backward compat; here they actually renamed functions so it was not backwards compatible and you had to change everything.

Simply put; we can cry about java or c# or php (raw php: not laravel; it has the same issues) but I can run 15 year old code on my current Ubuntu and it works fine. Maybe I need to change a few things but I do not have to rewrite a lot of it; it is more find/replace than anything. The feeling that software has to work in 50 years is important. Incremental updates are expensive and not needed for most companies, so software needs to support major version updates over decades. We know that now...

Re: All of us test in production all the time (2019)

#105

Post by ZephyrBlu »

ratww wrote: post #75
rybosworld wrote: post #65

Earlier quoted context omitted.

Node is getting better each year imo. I've been pretty happy with ES6. What makes you say Babel is horrible? Not disagreeing but I haven't heard anyone say that before.

People don't say it directly, but all the constant complaining about the bloat and complexity of the JS ecosystem and the size of node_modules folder boils down mostly to two tools: Babel and Webpack.

I don't think it's the tools' fault. They were built to solve problems, and they solve those problems quite well IMO.

Re: All of us test in production all the time (2019)

#106

Post by sli »

pydry wrote: post #103
ratww wrote: post #75

Earlier quoted context omitted.

People don't say it directly, but all the constant complaining about the bloat and complexity of the JS ecosystem and the size of node_modules folder boils down mostly to two tools: Babel and Webpack.

When people make fun of leftpad they are not talking about babel and webpack. The bloat is a lot about the lack of a standard library in JS. Or rather, too many of the damn things. That and an inconsistent and unpredictable type system.

> The bloat is a lot about the lack of a standard library in JS.

Which is why someone created core-js, which a bunch of libraries use but, in my experience, never update. Every React project will have multiple libraries complaining that core-js 2 is deprecated.

So basically even the "fix" has all those problems.

> That and an inconsistent and unpredictable type system.

Agreed and honestly, I don't think TypeScript's solution for third party types is any better, either. I recently ended up in a situation where `some-library` and `@types/some-library` got out of sync and I simply had to try versions of `@types/some-library` until it would compile again. This happened because typedef versions are not pinned to library version in any way, they are versioned just like any other library, and to the best of my knowledge you can't simply look up this information to figure out which @types release you need for a library. You can only enumerate released versions and hope the latest of each work fine. Was there a breaking change that the typedefs haven't accounted for yet? Well guess what, you can't update that library, yet, even if there were security fixes. Your code won't compile.

This means you could potentially have `some-library` at version 1.2.1 and `@types/some-library` at version 1.1.0. There's no relationship there. This happens the moment there's a revision in a library that doesn't require a revision in the typedefs.

Sorry to ramble, I'm just really unhappy with the state of our two major ES languages.

Re: All of us test in production all the time (2019)

#107

Post by mgkimsal »

jagraff wrote: post #90
aunty_helen wrote: post #82

Earlier quoted context omitted.

Feel your pain I know what it's like to ask someone in charge of 8k people to personally approve my server permission escalation even though he's never met me, doesn't know anything about the IT systems, has never visited the site I work at or probably even been in the country I reside. Anyway, a good thing on this front now is that (speaking from Azure exp.), you can dump logs directly to blob storage (s3, whatever…

The problem that the above user had was that the logs were already written without tz info, so they had no way to know. Assuming it was set to the machine's tz producing the logs, you could probably figure out the tz from the machine's ip address (assuming that's produced in the logs as well..)

what I learned was that the logs are timestamped to whatever machine they're on, wherever it is. The servers in London - GMT. The servers in west coast - Pacfic time. Servers in DC - Eastern time. Not ideal.

Re: All of us test in production all the time (2019)

#108

Post by jagraff »

mgkimsal wrote: post #107
jagraff wrote: post #90

Earlier quoted context omitted.

The problem that the above user had was that the logs were already written without tz info, so they had no way to know. Assuming it was set to the machine's tz producing the logs, you could probably figure out the tz from the machine's ip address (assuming that's produced in the logs as well..)

what I learned was that the logs are timestamped to whatever machine they're on, wherever it is. The servers in London - GMT. The servers in west coast - Pacfic time. Servers in DC - Eastern time. Not ideal.

Right - assuming the server IP, or at least some other network identifying factor, is in the log, you could write some sort of regex to parse the logs and identify the correct time. Of course, that depends on someone being able to actually parse the logs with your regex

Re: All of us test in production all the time (2019)

#109

Post by Lammy »

toast0 wrote: post #91
Lammy wrote: post #70

Earlier quoted context omitted.

And then that one person gets hit by a bus and you go out of business. Very-interconnected large-scale systems rarely have failure modes that are as simple as something the dev did/didn't do.

It seems like about half of the postmortems I've seen (public ones for high profile things and private ones where I've worked) have the incident start either when someone pushed a change, or sometime after the change was pushed when the change blew up; this is why change moratoriums are so effective --- when people stop messing with the system, it becomes stable. Another large portion is power transfer switches faili…

Arguing in favor of more than one person per project is not "obsessing" over bus factor lol. I want to be able to take days off, and I want my coworkers to enjoy the same.

The kind of takeaway I'd want to see from your first example is less like "don't do the things we know will cause breakage when we can't tolerate breakage" and more like "develop runtime-gating of new features and a way of sampling or shadowing production traffic onto n+1 builds before they are eligible to become the released build".

I've also had many issues with dodgy hardware of all types forever-circling repair queues in large fleets and never had a satisfying outcome for it either. Hopefully one of these days.

Re: All of us test in production all the time (2019)

#110

Post by smoe »

siliconc0w wrote: post #89

So it's a fair point that the fidelity of non-prod environments are inherently limited and you still need a bunch of other stuff like canaries, automated canary analysis, automated rollbacks, zone fault-tolerance, feature-flags, chaos engineering, server and client-side instrumentation, but this is generally 'next-level' stuff when most shops aren't even getting the basics right. For the overwhelming majority of non-…

Agreed

I like the view i picked up somewhere but don't remember from whom: Tests are not there to prove something is working, but to prove something is not working.

They are a fairly cheap way to tell me during development that this ain't going to work no matter what. And if everything compiles and the tests and reviews pass, I think this has a good shot at working out in production, but there still might well be cases and circumstances I haven't or coulnd't have considered.

No setup will cover 100%, but with not that much work you can already get quite far. The tricky thing in my opinions is more on deciding which kind of tests, processes and automations are actually worth implementing and maintaining at a given stage of a project.

Post reply on HN