Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding Kafka with Factorio (2019)

ruurtjan.medium.com

Post reply on HN
Print view View original

11–20 of 112 posts

Re: Understanding Kafka with Factorio (2019)

#11

Post by teawrecks »

marginalia_nu wrote: post #3

> Vertical scaling — a bigger, exponentially more expensive server This is in practice not true at all. Vertical scaling is typically a sublinear cost increase (up to a point, but that point is a ridiculous beast of a machine), since you're (typically) upgrading just the CPU and/or just the RAM or just the storage; not all of them at once. There are instances where you can get nearly 10x the machine for 2x the cost.

For small consumer products sure, but we're talking at the extreme end of performance and physical capabilities. Sure you can get a 2Ghz CPU for ~2x the price of a 200Mhz CPU, but how much are you going to pay for a 6.0Ghz CPU vs 5.0Ghz? 6.1Ghz vs 6.0Ghz?

Re: Understanding Kafka with Factorio (2019)

#12

Post by Sohcahtoa82 »

teawrecks wrote: post #11
marginalia_nu wrote: post #3

> Vertical scaling — a bigger, exponentially more expensive server This is in practice not true at all. Vertical scaling is typically a sublinear cost increase (up to a point, but that point is a ridiculous beast of a machine), since you're (typically) upgrading just the CPU and/or just the RAM or just the storage; not all of them at once. There are instances where you can get nearly 10x the machine for 2x the cost.

For small consumer products sure, but we're talking at the extreme end of performance and physical capabilities. Sure you can get a 2Ghz CPU for ~2x the price of a 200Mhz CPU, but how much are you going to pay for a 6.0Ghz CPU vs 5.0Ghz? 6.1Ghz vs 6.0Ghz?

Think cores instead of clock speeds.

In the case of cloud instances, doubling cores is frequently less than 100% more expensive.

Re: Understanding Kafka with Factorio (2019)

#13

Post by dekhn »

marginalia_nu wrote: post #3

> Vertical scaling — a bigger, exponentially more expensive server This is in practice not true at all. Vertical scaling is typically a sublinear cost increase (up to a point, but that point is a ridiculous beast of a machine), since you're (typically) upgrading just the CPU and/or just the RAM or just the storage; not all of them at once. There are instances where you can get nearly 10x the machine for 2x the cost.

Disagree- typically vertical scaling is lumpy, and even worse- CPU and RAM upgrades are typically not linear, because you're limited by the number of slots/sockets and the manufacturers intentionally charge higher (expoentially) prices for the largest RAM and fastest CPUs.

Re: Understanding Kafka with Factorio (2019)

#14

Post by The_Colonel »

Sohcahtoa82 wrote: post #12
teawrecks wrote: post #11

Earlier quoted context omitted.

For small consumer products sure, but we're talking at the extreme end of performance and physical capabilities. Sure you can get a 2Ghz CPU for ~2x the price of a 200Mhz CPU, but how much are you going to pay for a 6.0Ghz CPU vs 5.0Ghz? 6.1Ghz vs 6.0Ghz?

Think cores instead of clock speeds. In the case of cloud instances, doubling cores is frequently less than 100% more expensive.

Increasing core count is not really vertical scaling. It's a hybrid between vertical and horizontal scaling, having some characteristics of both. It also tops out quite early (especially its cost-effectiveness for many use cases, but there's an absolute upper limit as well).

Re: Understanding Kafka with Factorio (2019)

#15

Post by morelisp »

Sohcahtoa82 wrote: post #12
teawrecks wrote: post #11

Earlier quoted context omitted.

For small consumer products sure, but we're talking at the extreme end of performance and physical capabilities. Sure you can get a 2Ghz CPU for ~2x the price of a 200Mhz CPU, but how much are you going to pay for a 6.0Ghz CPU vs 5.0Ghz? 6.1Ghz vs 6.0Ghz?

Think cores instead of clock speeds. In the case of cloud instances, doubling cores is frequently less than 100% more expensive.

https://aws.amazon.com/msk/pricing/ prices scale linearly with CPU beginning with m5.large, and I wouldn't really want to run a production Kafka on anything less than m5.xlarge. (They do at least keep linearly scaling all the way up.) Speculating wildly, I could probably have run some of our real clusters on the equivalent of a 8xlarge, but of course 32 core systems were not widely available at that time. The cluster I run today, even a hypothetical 48xlarge would struggle.

YMMV for non-managed stuff, but really, you can only bump cores like 3 times realistically, 4 if you started really shitty, before you start getting into special pricing brackets.

Re: Understanding Kafka with Factorio (2019)

#16

Post by morelisp »

dekhn wrote: post #13
marginalia_nu wrote: post #3

> Vertical scaling — a bigger, exponentially more expensive server This is in practice not true at all. Vertical scaling is typically a sublinear cost increase (up to a point, but that point is a ridiculous beast of a machine), since you're (typically) upgrading just the CPU and/or just the RAM or just the storage; not all of them at once. There are instances where you can get nearly 10x the machine for 2x the cost.

Disagree- typically vertical scaling is lumpy, and even worse- CPU and RAM upgrades are typically not linear, because you're limited by the number of slots/sockets and the manufacturers intentionally charge higher (expoentially) prices for the largest RAM and fastest CPUs.

Kafka is also a system that can make pretty good general use of more CPUs and more storage, but doesn't have much need for RAM. Tying the CPU and RAM together whether by CPU model or cloud vendor offerings is annoying if you're trying to scale only vertically.

Re: Understanding Kafka with Factorio (2019)

#17

Post by marginalia_nu »

teawrecks wrote: post #11
marginalia_nu wrote: post #3

> Vertical scaling — a bigger, exponentially more expensive server This is in practice not true at all. Vertical scaling is typically a sublinear cost increase (up to a point, but that point is a ridiculous beast of a machine), since you're (typically) upgrading just the CPU and/or just the RAM or just the storage; not all of them at once. There are instances where you can get nearly 10x the machine for 2x the cost.

For small consumer products sure, but we're talking at the extreme end of performance and physical capabilities. Sure you can get a 2Ghz CPU for ~2x the price of a 200Mhz CPU, but how much are you going to pay for a 6.0Ghz CPU vs 5.0Ghz? 6.1Ghz vs 6.0Ghz?

You can go from a 8T/16C Epyc 7xxx series CPU to a 32T/64C CPU and not even double the cost.

Re: Understanding Kafka with Factorio (2019)

#18

Post by dang »

Related:

Understanding Kafka with Factorio (2019) - https://news.ycombinator.com/item?id=29304414 - Nov 2021 (72 comments)

Understanding Kafka with Factorio - https://news.ycombinator.com/item?id=20362179 - July 2019 (84 comments)

(Reposts are fine after a year or so; links to past threads are just to satisfy extra-curious readers)

Re: Understanding Kafka with Factorio (2019)

#20

Post by fluoridation »

marginalia_nu wrote: post #17
teawrecks wrote: post #11

Earlier quoted context omitted.

For small consumer products sure, but we're talking at the extreme end of performance and physical capabilities. Sure you can get a 2Ghz CPU for ~2x the price of a 200Mhz CPU, but how much are you going to pay for a 6.0Ghz CPU vs 5.0Ghz? 6.1Ghz vs 6.0Ghz?

You can go from a 8T/16C Epyc 7xxx series CPU to a 32T/64C CPU and not even double the cost.

That's more like horizontal scaling, though. You get more throughput (transactions per second) but not lower latency (seconds per transaction). Though it may be more cost-effective to have a single 32-core machine than two 16-core machines.
Post reply on HN