Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Simple Systems Have Less Downtime (2020)

gkogan.co

Post reply on HN
Print view View original

81–90 of 220 posts

Re: Simple Systems Have Less Downtime (2020)

#81

Post by naasking »

eatonphil wrote: post #74
bob1029 wrote: post #65

Earlier quoted context omitted.

> accidental and essential complexity I feel like I am turning into a bot for posting the Out of the Tar Pit paper: http://curtclifton.net/papers/MoseleyMarks06a.pdf This changed my understanding of computer science and our product virtually overnight. We are using a hybrid model of Functional Relational Programming (see section 9 in the paper). This is in production right now and its clearly the right answer for man…

The problem is that performance and normalization do not (always) go well together. Let's say you have billions of rows of event data you want to perform summary counts for by a few different key columns. Doing this up front as the events are ingested is going to allow for much more efficient querying on an already grouped table than having to group on your billions of events in each SELECT query. I'm not saying don'…

The question to ask is whether building a cache of normalized data will be more efficient than addressing the complexities of non-normalized data, eg. duplication, renaming, integrity problems, etc.

Re: Simple Systems Have Less Downtime (2020)

#82

Post by BoiledCabbage »

eatonphil wrote: post #74
bob1029 wrote: post #65

Earlier quoted context omitted.

> accidental and essential complexity I feel like I am turning into a bot for posting the Out of the Tar Pit paper: http://curtclifton.net/papers/MoseleyMarks06a.pdf This changed my understanding of computer science and our product virtually overnight. We are using a hybrid model of Functional Relational Programming (see section 9 in the paper). This is in production right now and its clearly the right answer for man…

The problem is that performance and normalization do not (always) go well together. Let's say you have billions of rows of event data you want to perform summary counts for by a few different key columns. Doing this up front as the events are ingested is going to allow for much more efficient querying on an already grouped table than having to group on your billions of events in each SELECT query. I'm not saying don'…

> The problem is that performance and normalization do not go well together.

> Let's say you have billions of rows of event data you with to perform summary counts for by a few different key columns.

Guess what? 99% of people on here don't have billions of rows of event data they need to regularly aggregate. Problem solved.

Such a terrible tradition in our industry in focusing on outlier cases, or on what FB/Google/... might need when making technical decisions.

Re: Simple Systems Have Less Downtime (2020)

#83

Post by mcphage »

trasz wrote: post #50
tomrod wrote: post #42

Earlier quoted context omitted.

Accidental, of course. There is never a need to introduce latency in text delivery, simply a long line of unmitigated wants.

Thai reduces a number of companies, eg Slack, to an “accidental complexity” then.

Are you saying that you think Slack's primary challenge is font rendering?

Re: Simple Systems Have Less Downtime (2020)

#84

Post by eatonphil »

BoiledCabbage wrote: post #82
eatonphil wrote: post #74

Earlier quoted context omitted.

The problem is that performance and normalization do not (always) go well together. Let's say you have billions of rows of event data you want to perform summary counts for by a few different key columns. Doing this up front as the events are ingested is going to allow for much more efficient querying on an already grouped table than having to group on your billions of events in each SELECT query. I'm not saying don'…

> The problem is that performance and normalization do not go well together. > Let's say you have billions of rows of event data you with to perform summary counts for by a few different key columns. Guess what? 99% of people on here don't have billions of rows of event data they need to regularly aggregate. Problem solved. Such a terrible tradition in our industry in focusing on outlier cases, or on what FB/Google/.…

OK well I'm telling you based on my experience and it wasn't at FB or Google. :) Do most people have these problems, no you're right. Should everyone ignore them and be unaware of options when they do have the problem? Probably not?

Re: Simple Systems Have Less Downtime (2020)

#85

Post by naasking »

dredmorbius wrote: post #76

One of my hobbies is tracking down the origin of concepts and phrases. One such phrase is "complexity is the enemy", which I'd first encountered through the Jargon File, though it appears elsewhere, e.g., http://www.neugierig.org/software/blog/2011/04/complexity.ht... Tracing that trough Google's Ngram Viewer, I found the extended form, "complexity is the enemy of reliability " in a short item in a 1959 issue of The…

> A full definition of "simplicity", as with "complexity", lies somewhat in the eye of the beholder.

Perhaps "somewhat", but not entirely or arguably even mostly:

http://curtclifton.net/papers/MoseleyMarks06a.pdf

Re: Simple Systems Have Less Downtime (2020)

#86

Post by trasz »

mcphage wrote: post #83
trasz wrote: post #50

Earlier quoted context omitted.

Thai reduces a number of companies, eg Slack, to an “accidental complexity” then.

Are you saying that you think Slack's primary challenge is font rendering?

No, it’s providing a web service to do something that would work (and used to work) much better without it.

Their primary challenge is making money, and that part probably is better as a web service, but it makes it worse from technical point of view. Just like with hypothetical FaaS - worse technically, but you can have adds.

Re: Simple Systems Have Less Downtime (2020)

#87

Post by eatonphil »

naasking wrote: post #81
eatonphil wrote: post #74

Earlier quoted context omitted.

The problem is that performance and normalization do not (always) go well together. Let's say you have billions of rows of event data you want to perform summary counts for by a few different key columns. Doing this up front as the events are ingested is going to allow for much more efficient querying on an already grouped table than having to group on your billions of events in each SELECT query. I'm not saying don'…

The question to ask is whether building a cache of normalized data will be more efficient than addressing the complexities of non-normalized data, eg. duplication, renaming, integrity problems, etc.

Totally. All I mean to say is that "everything should always be normalized" doesn't necessarily make sense. You need to consider your situation.

Re: Simple Systems Have Less Downtime (2020)

#88

Post by DanielBMarkham »

kaba0 wrote: post #36
zgniatacz wrote: post #24

“Simplicity is a great virtue but it requires hard work to achieve it and education to appreciate it. And to make matters worse: complexity sells better.” – Edsger W. Dijkstra

I think every talk on complexity is meaningless, without taking into account the difference between accidental and essential complexity. There are problem domains where there is simply a minimal complexity required implicitly. You can’t write an insanely simple program that will render vector fonts, simply because the problem inherently has to have a fixed amount of complexity. Anything more (accidental complexity) i…

>> I think every talk on complexity is meaningless, without taking into account the difference between accidental and essential complexity.

Yeah. This is both true and pointless.

Nobody codes things too complex on-purpose, at least not normal people. So it's not the difference in types of complexity, it's the difference in our ability to understand accidental and essential complexity.

I find that I do a really poor job at this, and I'm the first to stand on the soapbox and rail against systems being too complex. Just like programmers naturally introduce bugs into code without realizing it, programmers naturally introduce complexity into code without realizing it.

We may theoretically be able to talk about the differences in type, or how to manage each, but our real problem begins in our conceptual models inside our brains or various types of problems, and this happens a long time before any code is ever written.

In my personal practice I've come up with a few gimmicks that help me refactor my preconceptions. It is very difficult, however, for people to adopt practices that continue to inform them that they make lots of mistakes. Everybody wants to code as if they're conquering the boss-level at the end of a game. We sell coding, tooling, frameworks, and practices to other coders under the assumption that they're going to have a blast, not that they're going to continuously be reminded that they're screwing things up.

So yes, I agree, but without moving past that statement into something more tractionable, it's more of a truism than a starting point.

Re: Simple Systems Have Less Downtime (2020)

#89

Post by reilly3000 »

eatonphil wrote: post #74
bob1029 wrote: post #65

Earlier quoted context omitted.

> accidental and essential complexity I feel like I am turning into a bot for posting the Out of the Tar Pit paper: http://curtclifton.net/papers/MoseleyMarks06a.pdf This changed my understanding of computer science and our product virtually overnight. We are using a hybrid model of Functional Relational Programming (see section 9 in the paper). This is in production right now and its clearly the right answer for man…

The problem is that performance and normalization do not (always) go well together. Let's say you have billions of rows of event data you want to perform summary counts for by a few different key columns. Doing this up front as the events are ingested is going to allow for much more efficient querying on an already grouped table than having to group on your billions of events in each SELECT query. I'm not saying don'…

Use a columnar database for this. Use materialized views. Don’t do this on the prod DB for the CRUD app. Simple :)
Post reply on HN