Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Servers Are Burning

logicmag.io

Post reply on HN
Print view View original

101–110 of 158 posts

Re: The Servers Are Burning

#101

Post by nitrogen »

toss1 wrote: post #84
gred wrote: post #75

> My first employer, the online dating site OkCupid, didn’t harp on testing either. In part, this was because our company was so small. Only seven fellow engineers and I maintained all the code running on our servers—and making tests work was time-consuming and error-prone. “We can’t sacrifice forward momentum for technical debt,” then-CEO Mike Maxim told me, referring to the cost of engineers building behind-the-sce…

Yup, classic. "...no time/resources to do it right in the first place, but plenty of time/resources to fix it when the customers complain..." I always thought it better to find the bugs in-house before shipping, but so many others don't see it...

The logic kind of makes sense if you consider that customers will only find a small fraction of the bugs that your team will. Sometimes the cost of removing all bugs is greater than the cost of losing customers to the few unlucky bugs that get found.

But you really need experienced people deciding where that tradeoff lies for each company and project.

Re: The Servers Are Burning

#102

Post by viraptor »

tekstar wrote: post #56
fhood wrote: post #40

Earlier quoted context omitted.

You can't possibly say that with any confidence. I literally just finished reproducing a memory leak caused by an insane combination of circumstances. This was not something valgrind or any similar tool would find. It was not something that any rational human being would have thought to write tests for. Once code is no longer synchronous, and tasks are being juggled around and swapped between, it becomes virtually im…

What's an example of a memory leak that can't be found by valgrind?

One happening in an embedded language with its own GC. Unless you can switch all allocations to explicit management for the tests, it's almost impossible to differentiate between live objects, free lists, cycles with deferred collection, and arenas that are supposed to be freed on exit.

Re: The Servers Are Burning

#103

Post by kbenson »

chrisweekly wrote: post #97
kbenson wrote: post #57

Earlier quoted context omitted.

Tests only catch what you test for. In this case "memory leak" can be replaced with "site functions as expected", and the same argument you made applies, and yet it's still useful . Just because you can't determine the exact cause of the leak, or even detect all of them, doesn't mean that having some indication of memory usage initially and under test loads of different (reproducible) amounts at different intervals a…

> "Tests only catch what you test for." Er, sort of. Integration and e2e tests often catch problems nobody specifically, explicitly envisioned. That's part of what makes them so valuable! Right? https://m.youtube.com/watch?v=0GypdsJulKE

Sure. I think we're just working on slightly different meanings of what I said, which wasn't meant in the exact way it's often rolled out as a criticism. Part of the reason you do different, vague tests is the hope you'll catch some odd errors someone missed. So you are testing, vaguely and imperfectly but over a much wider area, in the hopes of finding things you can't specifically think of.

Sort of like if you are planning to drive some car you recently did a bunch of work yourself to on a long trip. You've probably already tested all the things you think might be wrong, but that doesn't mean it's not worth taking it for a drive for an hour around town and on the freeway to see if the unexpected happens, so you can deal with it close to home. You're testing the car, just in a vague "shake it and see what falls out" way. You are specifically testing for that, it's just not guaranteed to find any or all problems. It's still better than nothing most times.

Re: The Servers Are Burning

#104

Post by kbenson »

shady-lady wrote: post #99
chrisweekly wrote: post #96

Earlier quoted context omitted.

Thanks for the rec! Just bought for $5 (kindle ed).

what site? i can only see manning for $36. not on amazons.

Here[1] it is on Amazon, which is cheaper, and seems to have used copies for ~$18, but I can't find anything for $5. Maybe it was a typo?

1: https://www.amazon.com/Specification-Example-Successful-Deli...

Re: The Servers Are Burning

#105

Post by always_good »

nradov wrote: post #77
1_800_UNICORN wrote: post #7

If you're an early-stage startup that's still determining product-market fit, then it's completely fair to eschew tests and accumulate tech debt; you're at a higher risk of running out of runway before you've even proven that your business works. As soon as you have enough users that an outage poses a significant risk to your business, you need to invest in either refactoring to reduce tech debt, or a rewrite. And yo…

Skipping tests generally doesn't allow for creating a working MVP any faster. It's just an illusion.

Only if you are referring to perfect tests which cost zero time to write/maintain and only the tests written that eventually will catch an issue.

Otherwise, tests have a cost just like any other code. You can see this by looking at both extremes: perfect tests (described above) and useless tests. For example, tests that make your codebase too brittle, tests that don't actually test anything useful, spending too much time writing tests, spending a lot of time writing tests for precisely the code with high disposability, tests that are too coupled with the code, stupid tests that should be removed but won't be because tests tend to be append-only, etc.

Testing is an advanced topic where every +1 unit you spend on testing doesn't mean your application is now +1 unit more robust. +1 unit of time spent in tests can even mean your application is -2 units worse because testing is a trade-off.

So tests are more than capable of hamstringing your unlaunched MVP. They're also capable of being the reason you launched your MVP sooner than later. But the costs of testing are no illusion.

Re: The Servers Are Burning

#106

Post by 08-15 »

Is this really about testing?

> If (the database throws an error) { do nothing }

It's paraphrased, the original was probably the C++ anti pattern of

> try { /some code/ } catch {...} {}

Whoever wrote this should be disciplined. This kind of code doesn't do anything, it just makes a bad situation worse. (It's also never tested, because who writes tests that provoke failures?) This kind of idiocy has no technical solution, unless the two-by-four counts.

If you're ever tempted to write something like this... don't. Especially in C++, the correct approach is to do nothing. Someone else will handle the exception. Code that isn't written doesn't cause trouble and doesn't need to be tested either.

Re: The Servers Are Burning

#107

Post by zbentley »

auganov wrote: post #91
mreome wrote: post #20

Earlier quoted context omitted.

While I can understand some of the benefits of functional programming within a single sub-system, or within academic research, I struggle to understand how pure functional programming can address the need for the large amounts of state information and abstraction layers required for a large scale system. I work with systems where (large/numerous) sub components require very specialized domain knowledge to understand…

Before you spend some time doing mostly functional it's easy to ignore just how much meaningless state you might have floating around. You're right that any non-trivial endeavour will necessitate quite a lot of information being kept around. But in my experience most of the state you produce in languages that don't support functional patterns isn't even related to the domain. You're often forced to put thing into var…

The original article wasn't talking about issues caused by storing too much program state in in-memory structures (variables/objects) though. It was talking about issues introduced by interactions with an external stateful system--a database.

Bugs related to in-memory state that gets out of whack are definitely a hassle--especially in multithreaded situations--but are, in my experience, only the tip of the iceberg. Once you have that kind of issue under control (either by choice of platform, discipline, linting, or unit testing) there is a huge category of problems that can arise from the remaining surface, categorized loosely as "as soon as your code leaves its own memory space, it's part of a distributed system, with all the hassles that entails". Proclaiming that FP techniques resolve or even significantly ease that category of issue seems debatable at best, and a false promise at worst.

Re: The Servers Are Burning

#108

Post by nandemo »

stephengillie wrote: post #48

> “Users don’t care.” This is the counter to every technical bikeshedding post. Either it creates revenue by being what users want, or it's a waste of time. And people seem to love wasting time instead of delivering.

In a surprising turn of events, the user showed they did care.

> “@okcupid how am I supposed to get my daily dose of crushing rejection and emotional humiliation if your site is down????”

Re: The Servers Are Burning

#109

Post by bunderbunder »

zbentley wrote: post #92
bunderbunder wrote: post #44

> in order to write effective tests, a programmer had to know all of the ways that a piece of software could fail in order to write tests for those cases No. In order to write effective tests, a programmer has to think of the piece of software's entire input domain, carve it up into a set of equivalence classes, and then determine what the expected behavior should be for a piece of input from each of those classes. W…

> In order to write effective tests, a programmer has to think of the piece of software's entire input domain, carve it up into a set of equivalence classes, and then determine what the expected behavior should be for a piece of input from each of those classes. OK, so let's revise the statement to "in order to write effective tests, you must know how your software should behave in the face of all kinds of inputs". I…

Well, that's the idea behind breaking it into equivalence classes. Even if the input space is infinite - say, the set of all tuples of two natural numbers and a string - you should be able to determine that the domain can be carved into only a relatively small number of sets that are interesting.

If you find you can't, well, that's what's great about figuring out the test cases first - because now you know that you've got something complicated, and you have a chance to figure out how to make it simpler.

Then, ideally, you don't have to even worry about coming up with a mess of individual test cases. You arm yourself with a good QuickCheck style testing framework, so that all you have to do is write down your invariants, your equivalence classes, and what sort of output to expect for each one, and you're mostly done. Just a few hand-coded unit tests to explicitly call out the edge cases, stuff like that.

Obviously this s a much bigger job if you're limiting yourself to hand-coding an explicit test method for every single test case. Don't do that. There are so many higher-leverage tools - data-driven tests, property tests, fuzz tests, approval tests, etc. - to use in addition to the basics. It's very much worthwhile to get good at them. Even if you don't always use them, learning how to is good practice for learning how to carve a big problem up into manageable pieces.

Re: The Servers Are Burning

#110

Post by ww520 »

Yep. I had caused a similar problem in the past that brought down a live site. It's a cascade failure on error handling that caused an avalanche of retrying requests piling up that eventually more and more servers failed under the load. Not fun. Luckily we had well defined deployment and rollback procedures and was able to roll back the change easily.
Post reply on HN