Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Simple Systems Have Less Downtime (2020)

gkogan.co

Post reply on HN
Print view View original

111–120 of 220 posts

Re: Simple Systems Have Less Downtime (2020)

#111

Post by Pokepokalypse »

naasking wrote: post #81
eatonphil wrote: post #74

Earlier quoted context omitted.

The problem is that performance and normalization do not (always) go well together. Let's say you have billions of rows of event data you want to perform summary counts for by a few different key columns. Doing this up front as the events are ingested is going to allow for much more efficient querying on an already grouped table than having to group on your billions of events in each SELECT query. I'm not saying don'…

The question to ask is whether building a cache of normalized data will be more efficient than addressing the complexities of non-normalized data, eg. duplication, renaming, integrity problems, etc.

I've seen the downside of this. And it's having to have an oncall staff of DBA's hand-grooming fragile databases and running deduplication processes that sometimes get so far behind that it's a mathematical impossibility to catch up with new data coming in. (never mind what happens to your maintenance schedule when you have entire teams focused on "fixing this problem" for weeks at a time).

If it were up to me (and it almost never is, because I'm not a DB expert), I would ALWAYS normalize as much as possible when designing a database. I really can't even wrap my brain around why large (and old) databases ever get into this state. But I've seen it at two different employers and it's very painful (and costly) to deal with. My take was that these databases were probably originally set up by people who had no idea what they were doing, and ended up locking the company into a shitty implementation that kept the company crippled 15 years later. But that's just me.

Re: Simple Systems Have Less Downtime (2020)

#112

Post by marcosdumay »

titzer wrote: post #66
kaba0 wrote: post #36

Earlier quoted context omitted.

I think every talk on complexity is meaningless, without taking into account the difference between accidental and essential complexity. There are problem domains where there is simply a minimal complexity required implicitly. You can’t write an insanely simple program that will render vector fonts, simply because the problem inherently has to have a fixed amount of complexity. Anything more (accidental complexity) i…

"Essential" is always relative to requirements. If the requirements include interacting with a poorly-designed, buggy, complicated piece of other software, then yeah, you have essential complexity. Enlightenment is when you realize that you can keep zooming out and dumping what seem like essential requirements but are really just BS that follows from interacting with constantly-changing crapball software stacks.

I'd say "essential" refers to the value statement.

The translation of that into requirements is by itself one of the largest sources of accidental complexity.

And even the value statement sometimes is wrong and leads to unnecessary complexity too.

Re: Simple Systems Have Less Downtime (2020)

#113

Post by zinekeller »

dredmorbius wrote: post #105
zinekeller wrote: post #99

Earlier quoted context omitted.

> I'm told that the UK generally has excellent availability of government publications, though none of https://www.gov.uk , the National Archives, nor the British Library seem to turn up the Factories report presently. Since this is from 1956, its copyrights have already expired (or waived, as the case may be, see https://lists.wikimedia.org/pipermail/wikipedia-l/2005-May/0... ), so scanning it is a fair game, but if…

I need to check with the friend who'd said it was available. I turned up a copy through a friendly online librarian (as with the Economist article). Given the age of both publications and their rather minimal commercial value, the hoop-jumping is annoying. That said, the general availability (if not legality) of information online has absolutely exploded over the past decade or so. Abesence of copyright in published…

Ah, okay. I forgot that bit, I've removed the erroneous section.

Re: Simple Systems Have Less Downtime (2020)

#114

Post by ocdtrekkie »

citrin_ru wrote: post #63
namelosw wrote: post #12

The Author generalized "fewer features lead to less downtime" to "simple systems have less downtime". A simple system such as a hand-written web server is very likely to crash. It's very likely cannot serve many users if we don't make it serve requests concurrently. What's worse, one tiny exception in one request would bring the whole system down. It stops serving any request to any user. A relatively complex system…

> A simple system such as a hand-written web server is very likely to crash. > Kubernetes is complex, and it definitely gives us less downtime than simple scripts we have written before we use Kubernetes There is a point in between 100LoC hand-written web server and K8s cluster. E. g. you need to serve static files and load is small to moderate (say 1. Create a K8s cluster where HTTP traffic will be dynamically proxi…

My single Intel NUC in my basement on residential cable serves cloud services at higher reliability than Office 365.

Sure, it doesn't do it for millions of people, but it's drastically less failure-prone, despite the lack of resilient design.

What boggles the mind is people are often sold the cloud even though their org isn't serving millions of users, and could just as easily operate as a box in a closet.

Re: Simple Systems Have Less Downtime (2020)

#115

Post by LeonB »

eatonphil wrote: post #84
BoiledCabbage wrote: post #82

Earlier quoted context omitted.

> The problem is that performance and normalization do not go well together. > Let's say you have billions of rows of event data you with to perform summary counts for by a few different key columns. Guess what? 99% of people on here don't have billions of rows of event data they need to regularly aggregate. Problem solved. Such a terrible tradition in our industry in focusing on outlier cases, or on what FB/Google/.…

OK well I'm telling you based on my experience and it wasn't at FB or Google. :) Do most people have these problems, no you're right. Should everyone ignore them and be unaware of options when they do have the problem? Probably not?

I had those problems several times within the last few years - having to aggregate and detect various signals from billions of data points - and keep the system flexible enough that different business people could “try out” various ideas and play with the results.

All sorts of different patterns and architectures have to be brought together to make sense of it.

But the technique described above still has its place within such a system. For example - you boil billions of data points down to a handful (several thousand) facts, such as events that occurred or anomalies. Then those several thousand facts can be analysed standalone (as described above) without going back to the source. Etc.

Re: Simple Systems Have Less Downtime (2020)

#116

Post by 10000truths »

Retric wrote: post #92
eatonphil wrote: post #87

Earlier quoted context omitted.

Totally. All I mean to say is that "everything should always be normalized" doesn't necessarily make sense. You need to consider your situation.

Premature optimization is the root of all evil. It’s unlikely normalizing data is going to be an actual problem you can’t solve via caching.

I hate how this quote has been perverted. “Premature optimization” was meant to apply to people making their programs an unreadable/unmaintainable mess to save a couple of CPU instructions. Figuring out how to organize your data is not premature optimization, it’s literally one of the first things you have to tackle when starting a project, because trying to change it down the line is going to be miserable, especially if you can’t afford any downtime.

Re: Simple Systems Have Less Downtime (2020)

#117

Post by rsync »

I gave a talk at CU Boulder[1] in 2019 on complexity in cloud infrastructure and the design of rsync.net.

What I wrote, and then said several times was that:

"Simple systems fail in boring ways. Complex systems fail in fascinating ways."

I gave examples from Chernobyl, Air France Flight 447, etc. as examples of such. It's a topic that fascinates me and I had hoped to give several more such talks - but alas, the pandemic.

In my opinion, the best discussion of this topic is the book _Normal Accidents_ by Charles Perrow[2].

[1] https://www.colorado.edu/libraries/2019/11/08/cyberinfrastru...

[2] https://en.wikipedia.org/wiki/Normal_Accidents

Re: Simple Systems Have Less Downtime (2020)

#118

Post by javajosh »

Sure, but you need a path there. How do you get from complex to simple? That's really hard. It's relatively easier to keep things simple over time.

By default Money does not know or care about conscientious design. It wants functionality and it wants it now, future be damned. This is not irrational. Usually there is no future.

The problem start with your unlikely success, and yes now you have money to throw at it, but ironically the people that are best suited to wrestle complexity to the ground are not super motivated by money. They are motivated by a peculiar kind of beauty that is, by hypothesis, missing in the successful, complex system, and so they will not want to contribute, money or no.

Re: Simple Systems Have Less Downtime (2020)

#119

Post by Pokepokalypse »

chousuke wrote: post #71
Puts wrote: post #25

When people face a new problem they intuitively try to find something they can add as a solution, not thinking about what they can remove to prevent the problem from occurring in the first place. This becomes apparent in the way marketing works. You sell products by promising they will solve a problem. You don't sell things by telling people what they really need to hear, that they just need less of everything. Simpl…

Part of this seems to be that people are afraid of removing parts of a system. It's easier to add stuff because you "know" you aren't removing some important bit, so it's "safer", even though the new stuff is likely to interact with the old stuff in unexpected ways and break things anyway.

This is where the strangler pattern can turn around and strangle you.

At my last job, we hired a new team to transition our stack to a new architecture. They built it, and SOME components of the legacy stack DID get replaced and deprecated. Some did not. Which is why across our two stacks, we used no less that 4 totally different, separate database implementations. 3 different NoSQL cache implementations. 2 different types of kubernetes clusters, (RKE and EKS). 3 totally separate monitoring and alerting systems. (just a few examples, but the whole problem is way too big to list out here). The worst problem was that staff who designed the legacy systems had either left, or didn't remember how they built it or why they made all the choices they did. So the ONLY effective means of knowing whether a legacy component could be replaced, was to simply shut it down and wait to see if anybody complained. (we called it a "scream test"). In some cases, customers would simply stop paying, and nobody (but finance) would notice for MONTHS. We'd still have people logging in from non-paying customers. Or in other cases, the customers had totally shut down their own operations, so there was nobody to pay, and nobody to call us and say "hey, we're not doing business with you anymore". Engineering never wrote a formal way of validating the whole system, so when features would break, the only way we'd know is if a customer noticed (assuming they're still using the feature), and called Support.

Now, this only happened in the 2 years I had been there. In the previous 15 years, I imagine this had happened several times (aborted modernizations/migrations).

In all that time, nobody had been allowed to sit and plan out how the whole system could be simplified, so that the whole organization could even understand what they were deploying and paying AWS for.

Re: Simple Systems Have Less Downtime (2020)

#120

Post by LambdaComplex »

chousuke wrote: post #71
Puts wrote: post #25

When people face a new problem they intuitively try to find something they can add as a solution, not thinking about what they can remove to prevent the problem from occurring in the first place. This becomes apparent in the way marketing works. You sell products by promising they will solve a problem. You don't sell things by telling people what they really need to hear, that they just need less of everything. Simpl…

Part of this seems to be that people are afraid of removing parts of a system. It's easier to add stuff because you "know" you aren't removing some important bit, so it's "safer", even though the new stuff is likely to interact with the old stuff in unexpected ways and break things anyway.

https://fs.blog/2020/03/chestertons-fence/
Post reply on HN