Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding Kafka with Factorio

hackernoon.com

Post reply on HN
Print view View original

31–40 of 85 posts

Re: Understanding Kafka with Factorio

#31

Post by GauntletWizard »

saint-loup wrote: post #29
DonHopkins wrote: post #2

When I saw this title, I first thought it was about understanding the cockroach-like aliens in Factorio, but it is actually about queuing and congestion and parallel processing with conveyor belts and factories! With the caveat that Factorio is EXTREMELY addictive, properly described as "programmer crack", here are some other Factorio related discussions: Factorio – a game where you can automate basically anything (f…

Oh my, can someone ping Gwern to add Factorio to this list. https://www.gwern.net/Turing-complete

There's neither surprise nor accident in Factorio's Turing completeness. The developers were aware of and fans of the existence of proofs of games on that list (Transport Tycoon is a particular example that they intentionally modeled train mechanics from)

Re: Understanding Kafka with Factorio

#33

Post by Baeocystin »

DonHopkins wrote: post #2

When I saw this title, I first thought it was about understanding the cockroach-like aliens in Factorio, but it is actually about queuing and congestion and parallel processing with conveyor belts and factories! With the caveat that Factorio is EXTREMELY addictive, properly described as "programmer crack", here are some other Factorio related discussions: Factorio – a game where you can automate basically anything (f…

Pro-tip to others who think 'Programmer Crack' is hyperbole- it is not.

Factorio does have a rather steep learning curve, and the UI is not the most intuitive. But. Once how it works clicks, it clicks, and suddenly morning. And I'm in my 40's, I'd thought my days of unexpected overnights while working on fascinating problems were mostly behind me. Highly recommended, but be aware.

Re: Understanding Kafka with Factorio

#34

Post by drivers99 »

root_axis wrote: post #6
gnulinux wrote: post #3

For those who don't know factorio is truly a masterpiece of a game. It's described as "programming the game". Strongly recommended to everyone here.

And high quality network multiplayer! "Pair programming" was never this much fun!

I used to play this with co-workers and we would end up having a standup meeting about it once we got back to the office in the morning.

Re: Understanding Kafka with Factorio

#35

Post by Baeocystin »

pul wrote: post #20

I wrote this article a couple of months ago and didn't expect it to appear on my news feed all of the sudden. Thanks for the love everyone. Any questions are welcome of course!

How do you like the .17 update? :D

More seriously, I am curious about what people who haven't played Factorio think of the article. Since I've sunk $bignum hours in to it, I found your explanations clear, but I'd like to hear what those who haven't think.

Re: Understanding Kafka with Factorio

#36

Post by seaish »

GauntletWizard wrote: post #31
saint-loup wrote: post #29

Earlier quoted context omitted.

Oh my, can someone ping Gwern to add Factorio to this list. https://www.gwern.net/Turing-complete

There's neither surprise nor accident in Factorio's Turing completeness. The developers were aware of and fans of the existence of proofs of games on that list (Transport Tycoon is a particular example that they intentionally modeled train mechanics from)

Obviously the circuit system is Turing complete on purpose, but I believe the belts and splitters may be Turing complete on their own, which would definitely be accidental and surprising to most.

Re: Understanding Kafka with Factorio

#37

Post by barrkel »

chrischen wrote: post #14

The analogy helped me understand, but in factorio belts are required primarily because everything takes physical space. What are real world use cases for kafka? Is it primarily if your data is too much to simply store in a db?

A database is a huge API to share with another team, and queries from other people add load with a principle-agent problem [1]

Give people a firehouse of events or deltas, though, and if they want to query it, performance is now their problem - they build the database, update it, index it, etc.

This is part of scaling organisationally, not just removing the database as a bottleneck.

Events are also a route to making cache invalidation - one of the hardest problems in CS, as we know - tractable. Build your caches as services that consume the firehouse, and invalidate based on a join between events and the cached data.

[1] https://en.m.wikipedia.org/wiki/Principal%E2%80%93agent_prob...

Re: Understanding Kafka with Factorio

#38

Post by breischl »

jpk wrote: post #25
realityman wrote: post #4

A better way to understand Kafka is to fill a room with rakes, turn of the lights, and walk around.

This sounds like an idea founded on experience, but without including any details about how you arrived at this sentiment the comment doesn't bring any value to the thread.

Not GP, but early versions of Kafka had a lot of poorly-chosen default settings and sharp edges. eg, it was tuned for speed and availability over reliability, so it could lose your data if you weren't careful (not dissimilar to early versions of MongoDB that way). It was possible/easy to set up topics with no replication, so you'd lose data on your first server outage. KStreams had no real error handling - it would just explode and not even really tell you why. I wasn't as close to the operational side, but there were a lot of problems there too.

IME it has gotten better over time, but it's still pretty easy to shoot yourself in the foot if you don't understand what you're doing with it.

Re: Understanding Kafka with Factorio

#39

Post by mabbo »

Baeocystin wrote: post #33
DonHopkins wrote: post #2

When I saw this title, I first thought it was about understanding the cockroach-like aliens in Factorio, but it is actually about queuing and congestion and parallel processing with conveyor belts and factories! With the caveat that Factorio is EXTREMELY addictive, properly described as "programmer crack", here are some other Factorio related discussions: Factorio – a game where you can automate basically anything (f…

Pro-tip to others who think 'Programmer Crack' is hyperbole- it is not. Factorio does have a rather steep learning curve, and the UI is not the most intuitive. But. Once how it works clicks, it clicks, and suddenly morning. And I'm in my 40's, I'd thought my days of unexpected overnights while working on fascinating problems were mostly behind me. Highly recommended, but be aware.

I had to uninstall it last weekend because I came to realize my playing it was no longer healthy. I mean, that sounds funny, but I'm being totally serious. I've had a few games like that in my life, and I've come to recognize a few signs that what I'm doing is not good for me anymore.

But wow, what a great game.

Re: Understanding Kafka with Factorio

#40

Post by opportune »

chrischen wrote: post #14

The analogy helped me understand, but in factorio belts are required primarily because everything takes physical space. What are real world use cases for kafka? Is it primarily if your data is too much to simply store in a db?

Let's say you have a few tens of millions of people using your webapp and you are collecting analytics/usage data from them. You definitely don't want them all trying to authenticate + write directly to a db; you need an intermediate layer to process all the data coming from tons of different sources.

You could potentially do that with a separate microservice you communicate with via http, but this requires a "liveness" of the microservice that isn't really necessary; you will often lose events if the microservice isn't able to keep up with the incoming load, and you need to process the events just as fast as they come in. The data flow is really just unidirectional so the response is unnecessary, you just need to reliably transmit the data.

Kafka lets you handle unidirectional data flows in a way that is lazier. The data producers just write to a service and the consumers connect to the service. In between, Kafka just behaves like a distributed message queue. Obviously this is a huge benefit over directly writing to a db or any other kind of offline storage since it can greatly reduce the connection overhead. The main benefit over using a microservice is that it relaxes the constraint that all the data is processed/handled exactly as it comes in. It makes non-critical data flow more redundant by adding this queue

(I don't think the linked article does a great job explaining why you would use kafka / what the alternatives are)

Post reply on HN