Understanding Kafka with Factorio (2019) • Hacker News Board

Re: Understanding Kafka with Factorio (2019)

#51

Post by teawrecks » Thu, Jul 13, 2023, 8:36 PM UTC

marginalia_nu wrote: post #32

fluoridation wrote: post #26

Earlier quoted context omitted.

Then it sounds like you have a disagreement of terminology with FTA, since the article is using the terms like I am. Vertical scaling means increasing the serial performance of the system, and horizontal scaling means increasing the parallel performance of the system. In this sense, vertical scaling past a certain point does indeed get exponentially more expensive, while horizontal scaling almost always scales linear…

What I'm commenting on is this phrasing from the article > Vertical scaling — a bigger, exponentially more expensive server > Horizontal scaling — distribute the load over more servers

Ok, I see where the lay person would get confused on this. In the context of this article, every core is what Wikipedia calls a "node". There is no difference between a single 32C CPU and 4x 8C CPUs except for their ability to share memory faster. Both are similarly defined as horizontal scaling in the context of this article. You're not going to finish a single workload any faster, but you're going to increase the throughput of finishing multiple workloads in parallel.

The fact that AMD chooses to package the "nodes" together on one die vs multiple doesn't change that.

Re: Understanding Kafka with Factorio (2019)

#52

Post by Terr_ » Thu, Jul 13, 2023, 8:36 PM UTC

hutzlibu wrote: post #33

golergka wrote: post #28

Earlier quoted context omitted.

That's what all life forms do — consume everything they can and reproduce as much as they can. So, doing is _is_ life.

That's an oversimplification. If you look closer, you might find that most living things are mostly living in symbiosis with their surroundings. Your human body is an example of cooperation of simple life forms. And when those living cells of your body consume everthing they can and reproduce as much as they can - than this is called cancer.

Survivorship bias: Mutual benign coexistence and symbiosis within the present are rooted in massacres of the past.

Re: Understanding Kafka with Factorio (2019)

#53

Post by tobbe2064 » Thu, Jul 13, 2023, 8:37 PM UTC

golergka wrote: post #27

jrm4 wrote: post #6

I was 100% expecting something about the writer. :)

One morning, when Gregor Samsa woke from troubled dreams, he found himself transformed into a piece of iron ore on a moving conveyor belt.

That one mad me laugh so much i annoyed my wife

Re: Understanding Kafka with Factorio (2019)

#54

Post by marginalia_nu » Thu, Jul 13, 2023, 8:49 PM UTC

teawrecks wrote: post #51

marginalia_nu wrote: post #32

Earlier quoted context omitted.

What I'm commenting on is this phrasing from the article > Vertical scaling — a bigger, exponentially more expensive server > Horizontal scaling — distribute the load over more servers

Ok, I see where the lay person would get confused on this. In the context of this article, every core is what Wikipedia calls a "node". There is no difference between a single 32C CPU and 4x 8C CPUs except for their ability to share memory faster. Both are similarly defined as horizontal scaling in the context of this article. You're not going to finish a single workload any faster, but you're going to increase the t…

The wikipedia article qualifies what it means with vertical scaling

> typically involving the addition of CPUs, memory or storage to a single computer.

Re: Understanding Kafka with Factorio (2019)

#55

Post by the_af » Thu, Jul 13, 2023, 8:54 PM UTC

bee_rider wrote: post #35

titanomachy wrote: post #31

Earlier quoted context omitted.

Wow, that's so unrelated. I think I'd tacitly assumed that it was named that because adopting an event-driven architecture results in byzantine and overly complicated software, like the bureaucracies in Kafka's novels.

I’d be shocked if your explanation wasn’t the real one. “Optimized for writing” sounds like the sort of justification you give when your project with a sarcastic self-deprecating name becomes surprisingly successful. If they wanted to name it after a fast, efficient famous writer it would be Apache Hemingway. Kafka died before most of his writing was published and most of it was destroyed, which doesn’t seem like som…

Your comment is on point. However...

... if you read Jay Krep's introduction to logs [1] (in the Kafka sense of logs) you can see that while he has a nice sense of humor [2], he felt pretty good about the Log abstraction and about Kafka. In no sense do I get the feeling he thought he was creating a kludge or something bad -- or "kafkaesque". Judging by the article, it might as well been named Apache Tolstoy!

[1] https://engineering.linkedin.com/distributed-systems/log-wha...

[2] "'Each working data pipeline is designed like a log; each broken data pipeline is broken in its own way.' — Count Leo Tolstoy (translation by the author)"

Re: Understanding Kafka with Factorio (2019)

#56

Post by lsaferite » Thu, Jul 13, 2023, 9:00 PM UTC

gjulianm wrote: post #46

haswell wrote: post #44

I recently started playing Factorio, and I kept thinking that this is what "low code" integration/automation tools should look like. Developer tooling with extremely clear visuals, obvious dataflow, endless combinations into which the rigidly defined components can be assembled to do exactly what they do. As opposed to so many takes on "flow based" programming, which present some imperfect nodal representation of the…

> And by the way, be sure you're ready to sink some time in if you're curious about this game...it's just too good, and I've had to consciously reduce the time I'm spending, because I could just keep optimizing...building...expanding...optimizing...it's built in the shape of the reward center of my brain. I feel the same. It scratches so many itches. I made the mistake of installing Space Exploration mod after the fi…

SE is such a deep hole to fall into. I feel for you.

Re: Understanding Kafka with Factorio (2019)

#57

Post by lsaferite » Thu, Jul 13, 2023, 9:01 PM UTC

haswell wrote: post #44

I recently started playing Factorio, and I kept thinking that this is what "low code" integration/automation tools should look like. Developer tooling with extremely clear visuals, obvious dataflow, endless combinations into which the rigidly defined components can be assembled to do exactly what they do. As opposed to so many takes on "flow based" programming, which present some imperfect nodal representation of the…

> it's built in the shape of the reward center of my brain.

This is a very accurate description of how I feel about factorio. Thanks, I'm going to use this going forward.

Re: Understanding Kafka with Factorio (2019)

#58

Post by [deleted] » Thu, Jul 13, 2023, 9:03 PM UTC

[deleted]

Re: Understanding Kafka with Factorio (2019)

#59

Post by defendBanana » Thu, Jul 13, 2023, 9:07 PM UTC

morelisp wrote: post #16

dekhn wrote: post #13

Earlier quoted context omitted.

Disagree- typically vertical scaling is lumpy, and even worse- CPU and RAM upgrades are typically not linear, because you're limited by the number of slots/sockets and the manufacturers intentionally charge higher (expoentially) prices for the largest RAM and fastest CPUs.

Kafka is also a system that can make pretty good general use of more CPUs and more storage, but doesn't have much need for RAM. Tying the CPU and RAM together whether by CPU model or cloud vendor offerings is annoying if you're trying to scale only vertically.

Kafka can keep a decent bit of data in RAM using file system pages. Often times you end up wasting CPUs on kafka nodes, not memory i think.

https://docs.confluent.io/platform/current/kafka/deployment....

Re: Understanding Kafka with Factorio (2019)

#60

Post by defendBanana » Thu, Jul 13, 2023, 9:14 PM UTC

dekhn wrote: post #13

marginalia_nu wrote: post #3

> Vertical scaling — a bigger, exponentially more expensive server This is in practice not true at all. Vertical scaling is typically a sublinear cost increase (up to a point, but that point is a ridiculous beast of a machine), since you're (typically) upgrading just the CPU and/or just the RAM or just the storage; not all of them at once. There are instances where you can get nearly 10x the machine for 2x the cost.

Disagree- typically vertical scaling is lumpy, and even worse- CPU and RAM upgrades are typically not linear, because you're limited by the number of slots/sockets and the manufacturers intentionally charge higher (expoentially) prices for the largest RAM and fastest CPUs.

With clouds this is not true anymore. They are exactly linear. If you ask for a smaller node they are simply propositioning a chunk of a larger machine anyway.

There is a point where the exponential pricing starts, but that point is way out there than most people expect. Probably ~100CPU, ~1TB RAM, >50Gbps network etc.