Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding Kafka with Factorio (2019)

ruurtjan.medium.com

Post reply on HN
Print view View original

71–80 of 112 posts

Re: Understanding Kafka with Factorio (2019)

#71

Post by teawrecks »

marginalia_nu wrote: post #54
teawrecks wrote: post #51

Earlier quoted context omitted.

Ok, I see where the lay person would get confused on this. In the context of this article, every core is what Wikipedia calls a "node". There is no difference between a single 32C CPU and 4x 8C CPUs except for their ability to share memory faster. Both are similarly defined as horizontal scaling in the context of this article. You're not going to finish a single workload any faster, but you're going to increase the t…

The wikipedia article qualifies what it means with vertical scaling > typically involving the addition of CPUs, memory or storage to a single computer.

This is one of those times when I feel like you just didn't read anything I typed. So... I'm just gonna let you be confidently incorrect.

Re: Understanding Kafka with Factorio (2019)

#72

Post by oconnor663 »

cortesoft wrote: post #69
ineedasername wrote: post #62

Earlier quoted context omitted.

>it's built in the shape of the reward center of my brain Yes, it's like a distillation of the feeling I get from the most enjoyable parts of my job, risking productivity loss from real world responsibilities until the complexity rises high enough to require project organization and advanced planning of tasks that are the least enjoyable parts of my job, along the lines of: "Crap, I have a sudden urgent need to deal…

I wonder who the real world equivalent of biters are… sales?

rad hits

Re: Understanding Kafka with Factorio (2019)

#73

Post by dekhn »

defendBanana wrote: post #60
dekhn wrote: post #13

Earlier quoted context omitted.

Disagree- typically vertical scaling is lumpy, and even worse- CPU and RAM upgrades are typically not linear, because you're limited by the number of slots/sockets and the manufacturers intentionally charge higher (expoentially) prices for the largest RAM and fastest CPUs.

With clouds this is not true anymore. They are exactly linear. If you ask for a smaller node they are simply propositioning a chunk of a larger machine anyway. There is a point where the exponential pricing starts, but that point is way out there than most people expect. Probably ~100CPU, ~1TB RAM, >50Gbps network etc.

They're linear... because they're charging you rates based on the cost of the large server, divided down into whatever server you provisioned.

Amusingly, for $94K (probably more like $85K after negotiation) you can buy a white box server: Dual Epyc 9000, 96 core/192thread, 3.5GHz, w/ 3TB RAM, 240T of very fast SSD, and a 10G NIC. The minimum config, Dual Epyc 9124, 16core/32thread, 64GB RAM, and only 4TB of storage is $9K (more like $8K after negotiation). That's "only" a factor of 10 in price for 8X CPUs, 48X the RAM, and 60X the storage.

Re: Understanding Kafka with Factorio (2019)

#74

Post by suremarc »

teawrecks wrote: post #51
marginalia_nu wrote: post #32

Earlier quoted context omitted.

What I'm commenting on is this phrasing from the article > Vertical scaling — a bigger, exponentially more expensive server > Horizontal scaling — distribute the load over more servers

Ok, I see where the lay person would get confused on this. In the context of this article, every core is what Wikipedia calls a "node". There is no difference between a single 32C CPU and 4x 8C CPUs except for their ability to share memory faster. Both are similarly defined as horizontal scaling in the context of this article. You're not going to finish a single workload any faster, but you're going to increase the t…

The ability to “share memory faster” is a bigger distinction than you make it out to be. Distributed applications look quite different from merely multithreaded or multiprocess shared-memory applications due to the unreliability of the network, the increased latency, and other things which some refer to as the fallacies of distributed computing. To me, this is usually what people mean when they talk about “horizontal” vs. “vertical” scaling. With modern language-level support for concurrency, it hurts much more to go from a shared memory architecture to a distributed one than to go from a single-thread architecture to a multithreaded one.

Re: Understanding Kafka with Factorio (2019)

#75

Post by AceJohnny2 »

cubefox wrote: post #23

Semi related: Are other people also annoyed by how many projects are using names of completely unrelated famous things? I expected to read some wild association between a game and Franz Kafka, but no, it's about a streaming platform which happens to be named "Kafka". This is getting seriously annoying when you google for some, e.g. historical, term and then your search results are littered by some completely unrelate…

On the one hand, Apache Kafka is a pretty well-known topic on this hacker/web/computing-centric message board, so for most people it doesn't need elucidating.

On the other hand, Factorio arguably has some Kafka-esque aspects!

Re: Understanding Kafka with Factorio (2019)

#76

Post by CptFribble »

cubefox wrote: post #23

Semi related: Are other people also annoyed by how many projects are using names of completely unrelated famous things? I expected to read some wild association between a game and Franz Kafka, but no, it's about a streaming platform which happens to be named "Kafka". This is getting seriously annoying when you google for some, e.g. historical, term and then your search results are littered by some completely unrelate…

Isn't it because of the beneficial SEO? I'm sure we all well understand the benefits of landing a product on the first results page of a well-known historical figure.

Re: Understanding Kafka with Factorio (2019)

#77

Post by swyx »

lsaferite wrote: post #56
gjulianm wrote: post #46

Earlier quoted context omitted.

> And by the way, be sure you're ready to sink some time in if you're curious about this game...it's just too good, and I've had to consciously reduce the time I'm spending, because I could just keep optimizing...building...expanding...optimizing...it's built in the shape of the reward center of my brain. I feel the same. It scratches so many itches. I made the mistake of installing Space Exploration mod after the fi…

SE is such a deep hole to fall into. I feel for you.

any good explainers of whats fun about SE for those who will never play it?

Re: Understanding Kafka with Factorio (2019)

#78

Post by fardo »

swyx wrote: post #77
lsaferite wrote: post #56

Earlier quoted context omitted.

SE is such a deep hole to fall into. I feel for you.

any good explainers of whats fun about SE for those who will never play it?

It adds hundreds of hours of gameplay, new endings, multiple planets, and much longer progression chains that really push your ability to optimize to the limit.

If the base game isn’t enough to keep you interested anymore because it’s gotten too simple, space exploration is a good hand grenade to mix things up.

Re: Understanding Kafka with Factorio (2019)

#79

Post by p1necone »

mettamage wrote: post #70
cortesoft wrote: post #69

Earlier quoted context omitted.

I wonder who the real world equivalent of biters are… sales?

Security issues in software

This is actually a pretty great analogy. Could be even closer if the biters came in and stole materials, or replaced things on cargo belts as well as just trying to destroy things.
Post reply on HN