Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding Kafka with Factorio (2019)

ruurtjan.medium.com

Post reply on HN
Print view View original

31–40 of 112 posts

Re: Understanding Kafka with Factorio (2019)

#31

Post by titanomachy »

the_af wrote: post #24
cubefox wrote: post #23

Semi related: Are other people also annoyed by how many projects are using names of completely unrelated famous things? I expected to read some wild association between a game and Franz Kafka, but no, it's about a streaming platform which happens to be named "Kafka". This is getting seriously annoying when you google for some, e.g. historical, term and then your search results are littered by some completely unrelate…

Even worse, Kafka and Apache Kafka have almost no meaningful connection. According to Wikipedia, the author was reading (Franz) Kafka, who was a writer, and the software system is "optimized for writing". (Franz) Kafka wasn't "optimized" for anything, so this is just whimsical naming. It could just as well have been named Apache Hemingway, or Apache Tolstoy. Whimsical naming is ok, but can also be confusing and annoy…

Wow, that's so unrelated. I think I'd tacitly assumed that it was named that because adopting an event-driven architecture results in byzantine and overly complicated software, like the bureaucracies in Kafka's novels.

Re: Understanding Kafka with Factorio (2019)

#32

Post by marginalia_nu »

fluoridation wrote: post #26
marginalia_nu wrote: post #22

Earlier quoted context omitted.

I disagree with this definition of horizontal scaling. If you're moving to a bigger computer rather than more computers, then you're scaling vertically and not horizontally. (and fwiw, wikipedia agrees with this definition: https://en.wikipedia.org/wiki/Scalability#Horizontal_(scale_... )

Then it sounds like you have a disagreement of terminology with FTA, since the article is using the terms like I am. Vertical scaling means increasing the serial performance of the system, and horizontal scaling means increasing the parallel performance of the system. In this sense, vertical scaling past a certain point does indeed get exponentially more expensive, while horizontal scaling almost always scales linear…

What I'm commenting on is this phrasing from the article

> Vertical scaling — a bigger, exponentially more expensive server

> Horizontal scaling — distribute the load over more servers

Re: Understanding Kafka with Factorio (2019)

#33

Post by hutzlibu »

golergka wrote: post #28
BSEdlMMldESB wrote: post #4

what about understanding our own imperialistic civilization? crash in planet (or new continent?) and proceed to expand while decimating native life? in any case, good game.

That's what all life forms do — consume everything they can and reproduce as much as they can. So, doing is _is_ life.

That's an oversimplification. If you look closer, you might find that most living things are mostly living in symbiosis with their surroundings. Your human body is an example of cooperation of simple life forms. And when those living cells of your body consume everthing they can and reproduce as much as they can - than this is called cancer.

Re: Understanding Kafka with Factorio (2019)

#34

Post by dekhn »

fluoridation wrote: post #26
marginalia_nu wrote: post #22

Earlier quoted context omitted.

I disagree with this definition of horizontal scaling. If you're moving to a bigger computer rather than more computers, then you're scaling vertically and not horizontally. (and fwiw, wikipedia agrees with this definition: https://en.wikipedia.org/wiki/Scalability#Horizontal_(scale_... )

Then it sounds like you have a disagreement of terminology with FTA, since the article is using the terms like I am. Vertical scaling means increasing the serial performance of the system, and horizontal scaling means increasing the parallel performance of the system. In this sense, vertical scaling past a certain point does indeed get exponentially more expensive, while horizontal scaling almost always scales linear…

The terms are used loosely and it doesn't make a lot of sense to argue about the definitions.

I think it's true to say that vertical scaling normally is done by increasing the RAM and CPU of a single machine with a single address space and switch/bus. While horizontal scaling is normally adding more machines (additional addresses spaces and switch/bus). Historically this is because RAM to CPU performance (throughput and latency) in a single address space and bus greatly exceeds the performance of any NIC connecting machines with distinct address spaces and busses. And it mostly ignores effects like the performance costs of swapping/paging when you don't have enough RAM.

I haven't really seen many systems where horizontal scaling is truly linear, unless the problem is embarassingly parallel, like serving static content.

Re: Understanding Kafka with Factorio (2019)

#35

Post by bee_rider »

titanomachy wrote: post #31
the_af wrote: post #24

Earlier quoted context omitted.

Even worse, Kafka and Apache Kafka have almost no meaningful connection. According to Wikipedia, the author was reading (Franz) Kafka, who was a writer, and the software system is "optimized for writing". (Franz) Kafka wasn't "optimized" for anything, so this is just whimsical naming. It could just as well have been named Apache Hemingway, or Apache Tolstoy. Whimsical naming is ok, but can also be confusing and annoy…

Wow, that's so unrelated. I think I'd tacitly assumed that it was named that because adopting an event-driven architecture results in byzantine and overly complicated software, like the bureaucracies in Kafka's novels.

I’d be shocked if your explanation wasn’t the real one. “Optimized for writing” sounds like the sort of justification you give when your project with a sarcastic self-deprecating name becomes surprisingly successful.

If they wanted to name it after a fast, efficient famous writer it would be Apache Hemingway.

Kafka died before most of his writing was published and most of it was destroyed, which doesn’t seem like something the software would want to be associated with, right?

Re: Understanding Kafka with Factorio (2019)

#36

Post by foota »

vegabook wrote: post #21
dekhn wrote: post #13

Earlier quoted context omitted.

Disagree- typically vertical scaling is lumpy, and even worse- CPU and RAM upgrades are typically not linear, because you're limited by the number of slots/sockets and the manufacturers intentionally charge higher (expoentially) prices for the largest RAM and fastest CPUs.

If they charge these big numbers more it's precisely because they're trying to capture some of the embarrassingly better value you get from vertical scaling. It's a testament to vertical scaling's effectiveness that they _can_ do so.

Sure, but by doing so they consume the effectiveness?

Re: Understanding Kafka with Factorio (2019)

#37

Post by KRAKRISMOTT »

marginalia_nu wrote: post #3

> Vertical scaling — a bigger, exponentially more expensive server This is in practice not true at all. Vertical scaling is typically a sublinear cost increase (up to a point, but that point is a ridiculous beast of a machine), since you're (typically) upgrading just the CPU and/or just the RAM or just the storage; not all of them at once. There are instances where you can get nearly 10x the machine for 2x the cost.

Also beyond a certain point, it makes sense to go straight to dedicated bare metal. The AWS tax is not worth paying if your workload is mostly fixed, somewhat fault tolerant (i.e. failed hardware on the weekends can be replaced on Monday without major interruption to business operations), and CPU bound. Get a high end machine on Hetzner and put everything behind a VPN or API auth and you will save more than 50% in spending.

Re: Understanding Kafka with Factorio (2019)

#38

Post by dekhn »

foota wrote: post #36
vegabook wrote: post #21

Earlier quoted context omitted.

If they charge these big numbers more it's precisely because they're trying to capture some of the embarrassingly better value you get from vertical scaling. It's a testament to vertical scaling's effectiveness that they _can_ do so.

Sure, but by doing so they consume the effectiveness?

No, because you pay a fixed cost to get higher performance and then benefit through the whole lifetime of the product (I'm assuming you are purchasing rationally and keep your machines loaded at 75% or better, and your software is not egregiously wasteful).

Re: Understanding Kafka with Factorio (2019)

#39

Post by golergka »

hutzlibu wrote: post #33
golergka wrote: post #28

Earlier quoted context omitted.

That's what all life forms do — consume everything they can and reproduce as much as they can. So, doing is _is_ life.

That's an oversimplification. If you look closer, you might find that most living things are mostly living in symbiosis with their surroundings. Your human body is an example of cooperation of simple life forms. And when those living cells of your body consume everthing they can and reproduce as much as they can - than this is called cancer.

Those statements do not contradict each other. Life forms consume and reproduce as much as they can, and they also live in cooperation and symbiosis. The latter can often be the most effective strategy to consume and reproduce as much as one wants.

Re: Understanding Kafka with Factorio (2019)

#40

Post by BSEdlMMldESB »

golergka wrote: post #39
hutzlibu wrote: post #33

Earlier quoted context omitted.

That's an oversimplification. If you look closer, you might find that most living things are mostly living in symbiosis with their surroundings. Your human body is an example of cooperation of simple life forms. And when those living cells of your body consume everthing they can and reproduce as much as they can - than this is called cancer.

Those statements do not contradict each other. Life forms consume and reproduce as much as they can, and they also live in cooperation and symbiosis. The latter can often be the most effective strategy to consume and reproduce as much as one wants.

yes, agree

the issue then, is how to decide what "as much as we can" even means; if we do "as much as we can" but this kills us, then it will have turned out that we indeed could not do that much

Post reply on HN