11–19 of 19 posts
Engineering is about handling what goes wrong, not what goes right. It's about handling the errors, changes, misuse, etc. It isn't about the techniques per say, as much as the mindset of living in an imperfect world.
[Edit: Fixed a typo.]
However, I do agree that handling the huge and complex range of inputs, not only the expected ones, is a great beginning to the process, one that is often overlooked. And same goes for internal monitoring, to make sure your system is still functioning as designed.
One question this raised (and I don't mean this as a gotcha): why could a flood to the error-reporting servers take down all of the applications? I expected the primary fix to be to decouple the work so it could continue with no error reporting server. (But I'm not familiar with Zookeeper or any of the other work the author's doing, beyond reading some post on Storm.)
It's a convenience thing so that users can quickly see if there are any errors happening in their applications. While you could provide hooks to integrate the error stream with some external error reporting system, you also want something that just works out of the box. Zookeeper is the only place that Storm can store state, Zookeeper is good at storing small amounts of data, and the recent errors are a small amount…
Earlier quoted context omitted.
Zookeeper is a distributed coordination service. Think of it as an extremely robust reliable datastore for handling small amounts of data. It provides that robustness by using an expensive synchronization protocol. When you try and slam it with large volumes of data, zookeeper falls over. And Storm relies on Zookeeper for basic functioning, so without a running zookeeper ensemble, the associated Storm cluster will di…
That makes sense. It's not clear to me though why error logging should belong to it.
Everything is fine as long as exceptions are infrequent.
One question this raised (and I don't mean this as a gotcha): why could a flood to the error-reporting servers take down all of the applications? I expected the primary fix to be to decouple the work so it could continue with no error reporting server. (But I'm not familiar with Zookeeper or any of the other work the author's doing, beyond reading some post on Storm.)
Zookeeper is a distributed coordination service. Think of it as an extremely robust reliable datastore for handling small amounts of data. It provides that robustness by using an expensive synchronization protocol. When you try and slam it with large volumes of data, zookeeper falls over. And Storm relies on Zookeeper for basic functioning, so without a running zookeeper ensemble, the associated Storm cluster will di…
All software has bugs and these specific problems have been fixed in newer versions, but they are super scary issues to run into with your distributed coordination service.
Great stuff, and I love the concrete example of the ZK failure due to error logging -- a classic cascading failure mode. While it's true that I'm an inveterate disaster porn addict[1] and would therefore love this regardless, I think that Nathan's piece serves as a model in that it speaks to learning from failure rather than gloating about nascent success -- we collectively need much more of this! I also like that Na…
Great stuff, and I love the concrete example of the ZK failure due to error logging -- a classic cascading failure mode. While it's true that I'm an inveterate disaster porn addict[1] and would therefore love this regardless, I think that Nathan's piece serves as a model in that it speaks to learning from failure rather than gloating about nascent success -- we collectively need much more of this! I also like that Na…
I've enjoyed your talk, thanks for posting. One thing I'd like to know though: As someone who's optimizing his debugging skills and environment so thoroughly as you, it surprised me that you love javascript. Don't get me wrong, obviously it has some of the best tooling thanks to its abundance, but doesn't it bug you that it tends to fail silently? I feel that there are quite a few error classes that need to be caught…
In the end, its a wash.
I think this quote is magic, "Software engineering is a constant battle against uncertainty – uncertainty about your specs, uncertainty about your implementation, uncertainty about your dependencies, and uncertainty about your inputs." Engineering is about handling what goes wrong, not what goes right. It's about handling the errors, changes, misuse, etc. It isn't about the techniques per say, as much as the mindset…
There are a certain series of things you have to hit in a fairly hyper-dimensional world, dodging constraints, hurdling uncertainty and taking risk in your stride as you struggle to make products that work, delight consumers and make bank.
It's like a complex and exquisite ballet really, with suppliers, manufacturers, producers and designers all coming together to make extraordinary products that astonish the world.
Ah, I love engineering.
[1] https://news.ycombinator.com/item?id=4238984
> designing a rocket engine is a massive game of high dimensional parameter whack-a-mole, it's very difficult to get a passable configuration without a lot of iteration and forwards-backwards passes
Super interesting post! I'd have mentioned Unit tests as another measure to tackle uncertainty. Simple, boring unit tests (reminds me of this post[1]). Maybe he just assumes those will exist when professional engineers write code. [2] [1] http://robertheaton.com/2013/04/01/check-youre-wearing-trous... [2] http://www.amazon.com/Clean-Coder-Conduct-Professional-Progr...