Principles of Software Engineering, Part 1
nathanmarz.com
Principles of Software Engineering, Part 1
1–10 of 19 posts
Re: Principles of Software Engineering, Part 1
#2Great stuff, and I love the concrete example of the ZK failure due to error logging -- a classic cascading failure mode. While it's true that I'm an inveterate disaster porn addict[1] and would therefore love this regardless, I think that Nathan's piece serves as a model in that it speaks to learning from failure rather than gloating about nascent success -- we collectively need much more of this! I also like that Nathan doesn't romanticize other engineering domains, as naive software engineers are wont to do; other engineering domains also struggle with failure -- it's just that their failures are so much more public (and so much more likely to involve loss of property and/or life) that they cannot evade collective introspection the way software engineering so frequently seems to. Very much looking forward to Part 2!
[1] http://www.infoq.com/presentations/Debugging-Production-Syst...
Re: Principles of Software Engineering, Part 1
#3There is a fine line between industry (cost center vs generating revenue) and startups when it comes to discussing the term software engineering.
I see a large amount of legacy maintenance in cost center based programming. Revenue generating industry channels seem to favor the enterprise aspect of software engineering. Startups attempt to just build, and fix as necessary (cowboy). Yet, each has their own facet of software engineering.
I am still trying to draw the line between too-enterprisy, too-maintenancy, and too-cowboy. At my current job, we assume everything is certain. The uncertainties are not coded for, because everything is internal. This bothers me to a large extent. I love coding for the uncertain. Giving more control to the user and automating a whole department is right up my court. Sadly, it is hard to convert people. Only the 'RU' in CRUD is in the user's hands most of the time. It is pure legacy fear.
The removing cascading failures part needs more emphasis. Remove portions from your cycle/automation/jobs. What happens? I also agree with the measure and monitor portion. Waiting to create analyzers and looking at metrics once the program starts breaking in production is too late.
Looking forward to the next posts.
Re: Principles of Software Engineering, Part 1
#4Super interesting post!
I'd have mentioned Unit tests as another measure to tackle uncertainty. Simple, boring unit tests (reminds me of this post[1]).
Maybe he just assumes those will exist when professional engineers write code. [2]
[1] http://robertheaton.com/2013/04/01/check-youre-wearing-trous... [2] http://www.amazon.com/Clean-Coder-Conduct-Professional-Progr...
Re: Principles of Software Engineering, Part 1
#5One question this raised (and I don't mean this as a gotcha): why could a flood to the error-reporting servers take down all of the applications? I expected the primary fix to be to decouple the work so it could continue with no error reporting server. (But I'm not familiar with Zookeeper or any of the other work the author's doing, beyond reading some post on Storm.)
Re: Principles of Software Engineering, Part 1
#6One question this raised (and I don't mean this as a gotcha): why could a flood to the error-reporting servers take down all of the applications? I expected the primary fix to be to decouple the work so it could continue with no error reporting server. (But I'm not familiar with Zookeeper or any of the other work the author's doing, beyond reading some post on Storm.)
Zookeeper is a distributed coordination service. Think of it as an extremely robust reliable datastore for handling small amounts of data. It provides that robustness by using an expensive synchronization protocol. When you try and slam it with large volumes of data, zookeeper falls over. And Storm relies on Zookeeper for basic functioning, so without a running zookeeper ensemble, the associated Storm cluster will die too.
Re: Principles of Software Engineering, Part 1
#7Super interesting post! I'd have mentioned Unit tests as another measure to tackle uncertainty. Simple, boring unit tests (reminds me of this post[1]). Maybe he just assumes those will exist when professional engineers write code. [2] [1] http://robertheaton.com/2013/04/01/check-youre-wearing-trous... [2] http://www.amazon.com/Clean-Coder-Conduct-Professional-Progr...
Yes, that's absolutely true. Unit tests are super important in order to guard against future mistakes. Or put another way, they guard against the uncertainty of the code maintaining its current functionality in the future.
Re: Principles of Software Engineering, Part 1
#8One question this raised (and I don't mean this as a gotcha): why could a flood to the error-reporting servers take down all of the applications? I expected the primary fix to be to decouple the work so it could continue with no error reporting server. (But I'm not familiar with Zookeeper or any of the other work the author's doing, beyond reading some post on Storm.)
Zookeeper is a distributed coordination service. Think of it as an extremely robust reliable datastore for handling small amounts of data. It provides that robustness by using an expensive synchronization protocol. When you try and slam it with large volumes of data, zookeeper falls over. And Storm relies on Zookeeper for basic functioning, so without a running zookeeper ensemble, the associated Storm cluster will di…
That makes sense. It's not clear to me though why error logging should belong to it.
Re: Principles of Software Engineering, Part 1
#9One question this raised (and I don't mean this as a gotcha): why could a flood to the error-reporting servers take down all of the applications? I expected the primary fix to be to decouple the work so it could continue with no error reporting server. (But I'm not familiar with Zookeeper or any of the other work the author's doing, beyond reading some post on Storm.)
It's a convenience thing so that users can quickly see if there are any errors happening in their applications. While you could provide hooks to integrate the error stream with some external error reporting system, you also want something that just works out of the box. Zookeeper is the only place that Storm can store state, Zookeeper is good at storing small amounts of data, and the recent errors are a small amount of data (as long as things are properly throttled). Hence, the design.
Re: Principles of Software Engineering, Part 1
#10Earlier quoted context omitted.
Zookeeper is a distributed coordination service. Think of it as an extremely robust reliable datastore for handling small amounts of data. It provides that robustness by using an expensive synchronization protocol. When you try and slam it with large volumes of data, zookeeper falls over. And Storm relies on Zookeeper for basic functioning, so without a running zookeeper ensemble, the associated Storm cluster will di…
That makes sense. It's not clear to me though why error logging should belong to it.
Indeed.
All our error logging is entirely away from the live kit to prevent shit like this happening.