Simple Systems Have Less Downtime • Hacker News Board

Re: Simple Systems Have Less Downtime

#221

Post by hartzell » Tue, Mar 03, 2020, 8:57 PM UTC

jniedrauer wrote: post #189

monus wrote: post #7

Usually, being easy is confused with being simple. Easy to use systems usually tend to be very complex but they'd only hide it from the average user. Once you go one step ahead, do something outside of quick start, you find yourself helpless. In that sense, the article is on point. Not only the management of the ship is simple, machinery behind the scenes is simple as well. This applies to programming languages, too.…

In real life, there are monsters lurking behind Go's "simplicity." An example: errors as values. It forces you to deal with errors as they come up, right? Not exactly... People just start ignoring errors. The extremely opinionated linter doesn't care if you just assign an error to _, or even just don't handle the return values at all. And it's not something you can easily spot in a pull request either. Then you end u…

> Go just hides the complexity from initial inspection.

If you (parent, other readers) haven't seen it, Rob Pike's "Simplicity is complicated" is a good discussion of just this point.

video: https://youtu.be/rFejpH_tAHM

slides: https://talks.golang.org/2015/simplicity-is-complicated.slid...

Re: Simple Systems Have Less Downtime

#222

Post by hartzell » Tue, Mar 03, 2020, 9:01 PM UTC

monus wrote: post #7

Usually, being easy is confused with being simple. Easy to use systems usually tend to be very complex but they'd only hide it from the average user. Once you go one step ahead, do something outside of quick start, you find yourself helpless. In that sense, the article is on point. Not only the management of the ship is simple, machinery behind the scenes is simple as well. This applies to programming languages, too.…

> [...] easy is confused with being simple [...]

The first half of Rich Hickey's "Simple Made Easy" presentation does a great job of defining easy/hard and simple/complex axes and distinguishing them.

video: https://www.infoq.com/presentations/Simple-Made-Easy/

It has been discussed before on Hacker News:

https://news.ycombinator.com/item?id=4173854

Re: Simple Systems Have Less Downtime

#223

Post by mdonahoe » Tue, Mar 03, 2020, 9:07 PM UTC

klysm wrote: post #210

throw0101a wrote: post #202

See Gall's Law: > A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.[9] * https://en.wikipedia.org/wiki/John_Gall_(author)#Gall's_law

I find it mildly annoying when people state such nuggets of knowledge as “laws”. Sure, it’s generally true that you can’t go for the most sophisticated and powerful system at the first go, but sometimes the minimal working system is complex no? Anyway, it’s certainly not a “law” of the universe.

Klysm's Law:

Nuggets of knowledge are not laws. Referring to them as such annoys Klysm.

Re: Simple Systems Have Less Downtime

#224

Post by vthriller » Tue, Mar 03, 2020, 9:10 PM UTC

throw0101a wrote: post #214

klysm wrote: post #211

Earlier quoted context omitted.

rsync.net is built on ZFS which is anything but simple. Having worked with multi-petabyte ZFS systems, you can run into some seriously hard to track down issues.

Is ZFS more or less complicated than mdraid+LVM+ext4?

I'd say it's more complicated even for simple ZFS pools without any redundancy or fancy features. My favorite illustration on this topic would be figure 2 from "Reliability Analysis of ZFS" (PDF): http://pages.cs.wisc.edu/~kadav/zfs/zfsrel.pdf

I still miss some of the features that are hard (if not impossible) to replicate in layered storage systems, like Merkle tree checksumming, but near-nightmare that low-level debugging of ZFS was actually turned me away from that filesystem.

Re: Simple Systems Have Less Downtime

#225

Post by courtf » Tue, Mar 03, 2020, 9:11 PM UTC

jniedrauer wrote: post #189

monus wrote: post #7

Usually, being easy is confused with being simple. Easy to use systems usually tend to be very complex but they'd only hide it from the average user. Once you go one step ahead, do something outside of quick start, you find yourself helpless. In that sense, the article is on point. Not only the management of the ship is simple, machinery behind the scenes is simple as well. This applies to programming languages, too.…

In real life, there are monsters lurking behind Go's "simplicity." An example: errors as values. It forces you to deal with errors as they come up, right? Not exactly... People just start ignoring errors. The extremely opinionated linter doesn't care if you just assign an error to _, or even just don't handle the return values at all. And it's not something you can easily spot in a pull request either. Then you end u…

I think there are multiple levels to this phenomenon. With Go in particular, after having used it in some high performance scenarios (10k request per second for images that are rendered on the fly by GPUs, on systems holding 200GB of binary radar imagery in memory, that also need to ingest 50MB per second of new radar data), I've seen that the simplicity of the language is not a hindrance in these areas.

Edit: if you find yourself spending most of your time trying to come up with the perfect abstraction, Go may really piss you off, I won't deny that, it's not a strength. You have to be satisfied with 'good enough' and move on, solving edge if/when they arise. Go often encourages moving toward the concrete, and you can solve generic problems in really basic ways sometimes. As an example, I was writing a DAG server last year, and coming up with ways to move data between vertices in the graph in a general way. Rather than getting out my abstractions, I just pass around []byte, and leave the interpretation of those bytes up to each vertex (often just a type cast). I personally find this refreshing, and while there are costs to doing it this way, with a few basic helper funcs, you can get 95% of what you want from a generic server like this without doing a lot of modeling.

Re: Simple Systems Have Less Downtime

#226

Post by K0SM0S » Tue, Mar 03, 2020, 9:14 PM UTC

sev0 wrote: post #215

klysm wrote: post #210

Earlier quoted context omitted.

I find it mildly annoying when people state such nuggets of knowledge as “laws”. Sure, it’s generally true that you can’t go for the most sophisticated and powerful system at the first go, but sometimes the minimal working system is complex no? Anyway, it’s certainly not a “law” of the universe.

"Law" can also be used for pithy statements that don't have to be particularly rigorous. C.f. Murphy's Law.

This statement generalizes quite well all around the world to the other kind of law, that's voted in parliament/congress... haha! ;-)

More seriously, I think the word OP is looking for is theory. Nuggets aren't theory, they're really empirical by essence — observations.

Re: Simple Systems Have Less Downtime

#227

Post by Smoosh » Tue, Mar 03, 2020, 9:23 PM UTC

mdonahoe wrote: post #223

klysm wrote: post #210

Earlier quoted context omitted.

I find it mildly annoying when people state such nuggets of knowledge as “laws”. Sure, it’s generally true that you can’t go for the most sophisticated and powerful system at the first go, but sometimes the minimal working system is complex no? Anyway, it’s certainly not a “law” of the universe.

Klysm's Law: Nuggets of knowledge are not laws. Referring to them as such annoys Klysm.

An aphorism being called a law is called a klysmism.

Re: Simple Systems Have Less Downtime

#228

Post by SahAssar » Tue, Mar 03, 2020, 9:27 PM UTC

vntok wrote: post #51

ailideex wrote: post #21

Earlier quoted context omitted.

Linux is incredibly simple compared to Windows once you actually your computer as a computer instead of as a really poorly made gaming console.

Try talking Grandma over the phone to help her share her photo directory with Aunt Jeanine, on the same PC. No command line allowed, and you know she can only focus on that type of task for 4, maybe 5 minutes top.

I think you're confusing easy and simple. Windows is easier (for most people), but linux is certainly simpler.

Re: Simple Systems Have Less Downtime

#229

Post by marcantonio » Tue, Mar 03, 2020, 9:30 PM UTC

klysm wrote: post #210

throw0101a wrote: post #202

See Gall's Law: > A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.[9] * https://en.wikipedia.org/wiki/John_Gall_(author)#Gall's_law

I find it mildly annoying when people state such nuggets of knowledge as “laws”. Sure, it’s generally true that you can’t go for the most sophisticated and powerful system at the first go, but sometimes the minimal working system is complex no? Anyway, it’s certainly not a “law” of the universe.

In other words: salt tastes salty.

Re: Simple Systems Have Less Downtime

#230

Post by rsync » Tue, Mar 03, 2020, 9:31 PM UTC

klysm wrote: post #211

rsync wrote: post #143

I gave a talk on this subject at CU last year and have, one way or another, spent my entire professional life thinking about this topic. I agree wholeheartedly that simple systems have less downtime. I would like to add a line from the talk that I give: Simple systems fail in boring ways. Complex systems fail in fascinating, unexpected ways. rsync.net storage arrays typically have multi-hundred day uptimes. But acros…

rsync.net is built on ZFS which is anything but simple. Having worked with multi-petabyte ZFS systems, you can run into some seriously hard to track down issues.

"rsync.net is built on ZFS which is anything but simple. Having worked with multi-petabyte ZFS systems, you can run into some seriously hard to track down issues."

Obviously I have a few things to say about this ...

First, zfs was considered production worthy on FreeBSD and put into fairly widespread use as early as ... 2008 ? 2009 ? We could have made very good use of it as we were running into all kinds of limits and corner cases with UFS2 and very large filesystems with hundreds of millions of inodes.

But we waited until late 2012 to do our first deployment and it took us about six years to finally deprecate the last UFS2 systems. We did this out of an abundance of caution and a desire to see things shake themselves out over several major releases of FreeBSD.

As for the complexity of ZFS, our previous architecture had 3ware RAID cards and all of their firmware and complexity sitting between the drives and the OS. In a way there was a beautiful elegance (in my opinion) in giving the OS a single drive:

    newfs /dev/da0

.... and that single drive just happened to be 40 TB in size and the OS has no idea what's going on underneath ... but there is a lot of complexity inside a full-blown RAID card and a lot of ways for drives to interact weirdly with it - especially when the drive is X years newer than the latest firmware for the card.

I find that on the very deepest, hardware level, handing over raw disks to ZFS and letting it manage them is simpler. We remove a fairly complex piece of hardware and accompanying firmware (since our HBAs are "degraded" to dumb "IT" mode).

Further, all of the "bolt-ons" of UFS2 that we absolutely relied on, such as quotas and snapshots, are elegantly built into the filesystem from the very lowest levels.

We ran rsync.net (and its predecessor) on UFS2 for 11-12 years and I can tell you from the perspective of both day to day management and middle of night firefighting, ZFS has made our lives much simpler.