Understanding Kafka with Factorio (2019)
ruurtjan.medium.com
Understanding Kafka with Factorio (2019)
1–10 of 112 posts
Re: Understanding Kafka with Factorio (2019)
#2Cute, but over years of explaining it I think any explanation of Kafka that presents it as a queue is bound to leave the reader with more misaligned expectations than when they started (while also making them think they learned something, which can be even more dangerous). To keep the Factorio-esque framing, move the consumers, not the messages.
Re: Understanding Kafka with Factorio (2019)
#3> Vertical scaling — a bigger, exponentially more expensive server
This is in practice not true at all. Vertical scaling is typically a sublinear cost increase (up to a point, but that point is a ridiculous beast of a machine), since you're (typically) upgrading just the CPU and/or just the RAM or just the storage; not all of them at once.
There are instances where you can get nearly 10x the machine for 2x the cost.
Re: Understanding Kafka with Factorio (2019)
#4what about understanding our own imperialistic civilization?
crash in planet (or new continent?) and proceed to expand while decimating native life?
in any case, good game.
Re: Understanding Kafka with Factorio (2019)
#5Cute, but over years of explaining it I think any explanation of Kafka that presents it as a queue is bound to leave the reader with more misaligned expectations than when they started (while also making them think they learned something, which can be even more dangerous). To keep the Factorio-esque framing, move the consumers, not the messages.
Agreed. There's an important difference between things like Kafka & Kinesis vs RabbitMQ & SQS. The latter are conceptually queues, and the former are conceptually logs. Logs and queues can both be used in many of the same use cases, but it's important to understand how they are different.
Re: Understanding Kafka with Factorio (2019)
#6I was 100% expecting something about the writer. :)
Re: Understanding Kafka with Factorio (2019)
#7> Vertical scaling — a bigger, exponentially more expensive server This is in practice not true at all. Vertical scaling is typically a sublinear cost increase (up to a point, but that point is a ridiculous beast of a machine), since you're (typically) upgrading just the CPU and/or just the RAM or just the storage; not all of them at once. There are instances where you can get nearly 10x the machine for 2x the cost.
The idea is don't let the logic come in the way of promoting "web scale" software.
Re: Understanding Kafka with Factorio (2019)
#8I was 100% expecting something about the writer. :)
Bad software is way more popular than a good writer.
Re: Understanding Kafka with Factorio (2019)
#9Cute, but over years of explaining it I think any explanation of Kafka that presents it as a queue is bound to leave the reader with more misaligned expectations than when they started (while also making them think they learned something, which can be even more dangerous). To keep the Factorio-esque framing, move the consumers, not the messages.
Agreed. There's an important difference between things like Kafka & Kinesis vs RabbitMQ & SQS. The latter are conceptually queues, and the former are conceptually logs. Logs and queues can both be used in many of the same use cases, but it's important to understand how they are different.
i.e.: Items are frequently removed from queues which have a shared "next item", while logs usually just get longer and each consumer is responsible for keeping track of their own progress or positions.
It's harder to think of factory-game analogies for logs, since they involve copying without altering the original sequence. It would have to involve some kind of moving non-destructive sensor or object-cloner mechanic.
Re: Understanding Kafka with Factorio (2019)
#10> Vertical scaling — a bigger, exponentially more expensive server This is in practice not true at all. Vertical scaling is typically a sublinear cost increase (up to a point, but that point is a ridiculous beast of a machine), since you're (typically) upgrading just the CPU and/or just the RAM or just the storage; not all of them at once. There are instances where you can get nearly 10x the machine for 2x the cost.
The kind of server you'd run Kafka on tends to already be pretty far up the curve. I don't think I can get 10x our default broker for 20x the cost. Maybe 100x the cost. (I could probably get 2x it for 2x the cost but once you value HA the practical inflection point starts below the actual cost intersection.)