Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Simple Systems Have Less Downtime

gkogan.co

Post reply on HN
Print view View original

231–240 of 271 posts

Re: Simple Systems Have Less Downtime

#231

Post by rsync »

sysashi wrote: post #201
rsync wrote: post #143

I gave a talk on this subject at CU last year and have, one way or another, spent my entire professional life thinking about this topic. I agree wholeheartedly that simple systems have less downtime. I would like to add a line from the talk that I give: Simple systems fail in boring ways. Complex systems fail in fascinating, unexpected ways. rsync.net storage arrays typically have multi-hundred day uptimes. But acros…

sorry for offtopic. there a broken link that I was interested to read on. its on https://www.rsync.net/resources/howto/rsync.html ctrl+f "rsync snapshots are detailed here" I think given the topic, someone is expected to figure it out on their own (remember linux cake?), but I feel a bit lazy after a work day.. sorry

Sorry - that is in purpose ...

The definitive page for rsync snapshots has been, and always will be, here:

http://www.mikerubel.org/computers/rsync_snapshots/

... we actually don't want people to do rsync snapshots anymore because ZFS snapshots are much more efficient and use up less of their rsync.net account.

If you change one bit of a file, the rsync snapshot method will cost the entire size of that file, since it's a hard link and is either broken or not broken.

But if you just do a "dumb" sync to us and change one bit in the file, your ZFS snapshot will take up just one more bit.

So not only do you get a much simpler backup script - you can really just do a dumb sync to us and let us rotate your snapshots - but you get more efficient space usage for changed files.

Re: Simple Systems Have Less Downtime

#232

Post by closeparen »

sgift wrote: post #145
nickbauman wrote: post #111

Earlier quoted context omitted.

But microservices are an example of simpler systems. Each microservice does far less than the whole monolith does. You can read all the code in ~15 minutes. I've worked at companies that have monoliths that are 50x more difficult to work on because of the size. Some of them millions of lines of code. Nobody really knows how they work anymore.

> But microservices are an example of simpler systems. Each microservice does far less than the whole monolith does. You can read all the code in ~15 minutes. Microservice usually means distributed system. A distributed system is more complex than a non-distributed system since it has to do everything the non-distributed system has to do and, additionally, handle all the distributed problems. Microservices just hide…

Simple example where this is not quite true: moving a slow, fallible operation out of band to a durable queue with a sensible retry policy will tend to make the system simpler and less brittle, even though it becomes distributed.

Re: Simple Systems Have Less Downtime

#233

Post by K0SM0S »

snovv_crash wrote: post #205
justaguyhere wrote: post #192

Earlier quoted context omitted.

typically deploying to production around a hundred times per day That is insane! We make a release once a week at work and things still go wrong sometimes. I am in awe how they are able to pull this off, especially at their scale.

The more frequently you release, the less, or less serious, bugs there typically are, and the earlier you catch them. The faster development cycle also helps people invest in testing infrastructure more effectively.

I'm trying to get a sense of magnitude, here.

Let's say 10-20 commits per day per dev. Over 5~10 hours that's what, on the order of 1 commit-test-release cycle every 15 to 60 minutes? (subjectively for each dev)

What do we actually write in that timeframe on average (thus including the ~90% of time we don't type code but think or read or test)? What's the "unit commit" here?

So I'm thinking... let's take an example: today I'll refactor a few functions to update our model handling; I wish to reflect our latest custom types in the code. So it's a lot of in-place changes, e.g. from some list to tuple or dict; and the syntax that goes with it. No external logic change, but new methods mean slight variations in the details of implementation.

- refactor one function: commit every testable change, like list to tuple? At least, I'm sure I'm not breaking other stuff by running the whole test suite every time it "works for me" in my isolated bubble. So I might commit every 5-10 minutes in that case.

- Now I'm touching the API so I can't break promises to clients: I actually need to test more rigorously anyway. I'm probably taking closer to 20-40 minutes per commit, it's more tedious. Assuming I commit every update of the model, even insignificant, I get immediate feedback (e.g. performance dump), so I know when to stop, backtrack, try again? And it's always just one "tiny" step?

- Later I review some code and have to go through all these changes. I assume it's easier to spot elementary mistakes; but what of the big picture? Sure I can show a diff over the whole process — I assume you'd learn to play with git with such an "extreme" approach.

Am I on the right track, here? I totally get your comment but I'm trying to get a feel for how it works. I typically commit-test-release prod 3-4 times a day at most (on simple projects), and typically more like once every 2-3 days, 2-3 times a week. Which is "agile" enough I reckon... So I'm genuinely interested here. I feel there's untapped power in the method I'm just beginning to grasp.

Re: Simple Systems Have Less Downtime

#234

Post by dang »

harrisreynolds wrote: post #206
dang wrote: post #197

Earlier quoted context omitted.

You're overdoing these links. That's no doubt why you're getting downvoted and flagged. It's fine on HN to post your own work (1) in places where it's relevant, as long as (2) you only do it occasionally and (3) are also participating in the community in the intended ways, submitting interesting articles and having curious conversation. But when users break these rules of thumb (and you've been breaking all three of…

Thanks for the feedback. So far I have been getting positive feedback which is why I have continued to do this. I have been trying to add to the overall value of HN with insight in general and only including a link to my site if it is relevant. But I hear you and will adjust accordingly. I do sincerely appreciate the feedback as I have been steadily gaining karma and then out of no-where it when backwards today.

It's a clear indication from the community that you're overdoing the promotional links. The fact that you haven't really been participating in any other way accentuates that impression. Readers check these things.

Re: Simple Systems Have Less Downtime

#235

Post by rjkennedy98 »

downerending wrote: post #177

As a sub-case of this, almost every HA (high availability) system I've ever seen has been less reliable than the original system was (without HA). It sounds like a good idea, but the extra complexity kills it. One non-software system was a rather expensive UPS/generator. It was meant to trip on a power loss and provide X minutes of stable power. In reality, it was incredibly sensitive to minute power fluctuations and…

Every? that seems a stretch. Plenty have very good and simple HA architectures: Cassandra, Kafka come to mind. Others of course don't.

Re: Simple Systems Have Less Downtime

#236

Post by snovv_crash »

K0SM0S wrote: post #233
snovv_crash wrote: post #205

Earlier quoted context omitted.

The more frequently you release, the less, or less serious, bugs there typically are, and the earlier you catch them. The faster development cycle also helps people invest in testing infrastructure more effectively.

I'm trying to get a sense of magnitude, here. Let's say 10-20 commits per day per dev. Over 5~10 hours that's what, on the order of 1 commit-test-release cycle every 15 to 60 minutes? (subjectively for each dev) What do we actually write in that timeframe on average (thus including the ~90% of time we don't type code but think or read or test)? What's the "unit commit" here? So I'm thinking... let's take an example:…

I think you've got it. Now put 10 people on the project, and have them all working at that pace.

Re: Simple Systems Have Less Downtime

#237

Post by ailideex »

AnIdiotOnTheNet wrote: post #128
ailideex wrote: post #21

Earlier quoted context omitted.

Linux is incredibly simple compared to Windows once you actually your computer as a computer instead of as a really poorly made gaming console.

Bullshit. You're talking about an operating system where the common advice for someone who wants to install up to date software is to fucking compile it from source because the whole community never got their collective shit together enough to allow developers to directly distribute binaries without a gigantic fucking headache. Christ, it's such a fucking mess that one of the most compatible ways to distribute softwa…

> compile it from source

Way simpler to do on Linux than M$ Windows.

> [no way to ] to directly distribute binaries without a gigantic fucking headache.

I mean, flatpak, appimage, docker, etc ...

Re: Simple Systems Have Less Downtime

#238

Post by K0SM0S »

snovv_crash wrote: post #236
K0SM0S wrote: post #233

Earlier quoted context omitted.

I'm trying to get a sense of magnitude, here. Let's say 10-20 commits per day per dev. Over 5~10 hours that's what, on the order of 1 commit-test-release cycle every 15 to 60 minutes? (subjectively for each dev) What do we actually write in that timeframe on average (thus including the ~90% of time we don't type code but think or read or test)? What's the "unit commit" here? So I'm thinking... let's take an example:…

I think you've got it. Now put 10 people on the project, and have them all working at that pace.

Ah, awesome, thanks for the feedback.

Re: Simple Systems Have Less Downtime

#239

Post by downerending »

rjkennedy98 wrote: post #235
downerending wrote: post #177

As a sub-case of this, almost every HA (high availability) system I've ever seen has been less reliable than the original system was (without HA). It sounds like a good idea, but the extra complexity kills it. One non-software system was a rather expensive UPS/generator. It was meant to trip on a power loss and provide X minutes of stable power. In reality, it was incredibly sensitive to minute power fluctuations and…

Every? that seems a stretch. Plenty have very good and simple HA architectures: Cassandra, Kafka come to mind. Others of course don't.

Well, "almost every" and by "seen" I mean in environments that I've had to support, in corporate and academic land.

Perhaps I've just had bad luck, but my impression is that getting this right, both internally and in the field, is a lot harder than it looks. And when it breaks, it can be a real s___show compared to the simple, non-HA version.

Re: Simple Systems Have Less Downtime

#240

Post by downerending »

jniedrauer wrote: post #184
downerending wrote: post #177

As a sub-case of this, almost every HA (high availability) system I've ever seen has been less reliable than the original system was (without HA). It sounds like a good idea, but the extra complexity kills it. One non-software system was a rather expensive UPS/generator. It was meant to trip on a power loss and provide X minutes of stable power. In reality, it was incredibly sensitive to minute power fluctuations and…

This has absolutely not been my experience, at least for well designed cloud infrastructure. High availability doesn't have to be complex or hard to understand. In most cases, automatic failovers and restarts on well designed architecture fix problems with zero end-user impact. It happens so seamlessly that my standard procedure these days is to fail over infrastructure if I want to restart it.

What is this "well-designed infrastructure" of which you speak? ;-)
Post reply on HN