Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding Kafka with Factorio (2019)

ruurtjan.medium.com

Post reply on HN
Print view View original

61–70 of 112 posts

Re: Understanding Kafka with Factorio (2019)

#61

Post by throitallaway »

gjulianm wrote: post #46
haswell wrote: post #44

I recently started playing Factorio, and I kept thinking that this is what "low code" integration/automation tools should look like. Developer tooling with extremely clear visuals, obvious dataflow, endless combinations into which the rigidly defined components can be assembled to do exactly what they do. As opposed to so many takes on "flow based" programming, which present some imperfect nodal representation of the…

> And by the way, be sure you're ready to sink some time in if you're curious about this game...it's just too good, and I've had to consciously reduce the time I'm spending, because I could just keep optimizing...building...expanding...optimizing...it's built in the shape of the reward center of my brain. I feel the same. It scratches so many itches. I made the mistake of installing Space Exploration mod after the fi…

SE is such a slog but I must keep going!

Re: Understanding Kafka with Factorio (2019)

#62

Post by ineedasername »

haswell wrote: post #44

I recently started playing Factorio, and I kept thinking that this is what "low code" integration/automation tools should look like. Developer tooling with extremely clear visuals, obvious dataflow, endless combinations into which the rigidly defined components can be assembled to do exactly what they do. As opposed to so many takes on "flow based" programming, which present some imperfect nodal representation of the…

>it's built in the shape of the reward center of my brain

Yes, it's like a distillation of the feeling I get from the most enjoyable parts of my job, risking productivity loss from real world responsibilities until the complexity rises high enough to require project organization and advanced planning of tasks that are the least enjoyable parts of my job, along the lines of:

"Crap, I have a sudden urgent need to deal with enemies creeping out from beyond my radar range that will push back operationalization of my proof-of-concept production pipeline. I'd estimate 3 man hours are required to perform a one-off fix on the enemies & radar expansion, maybe 5 to automate long-term... damnit I need a break, let me VPN into work to decompress from Factorio stress."

Re: Understanding Kafka with Factorio (2019)

#63

Post by fluoridation »

dekhn wrote: post #34
fluoridation wrote: post #26

Earlier quoted context omitted.

Then it sounds like you have a disagreement of terminology with FTA, since the article is using the terms like I am. Vertical scaling means increasing the serial performance of the system, and horizontal scaling means increasing the parallel performance of the system. In this sense, vertical scaling past a certain point does indeed get exponentially more expensive, while horizontal scaling almost always scales linear…

The terms are used loosely and it doesn't make a lot of sense to argue about the definitions. I think it's true to say that vertical scaling normally is done by increasing the RAM and CPU of a single machine with a single address space and switch/bus. While horizontal scaling is normally adding more machines (additional addresses spaces and switch/bus). Historically this is because RAM to CPU performance (throughput…

Note that I was referring to scaling of cost, not of performance. If your application parallelizes ideally, then in the worst case your cost will scale linearly, because you just add more machines and increase your power consumption by new_machine_count/previous_machine_count. It's possible adding more processors in the same address space increases the cost by an amount below new_core_count/previous_core_count, in which case the cost scales better than linearly.

Re: Understanding Kafka with Factorio (2019)

#64

Post by RhodesianHunter »

KRAKRISMOTT wrote: post #37
marginalia_nu wrote: post #3

> Vertical scaling — a bigger, exponentially more expensive server This is in practice not true at all. Vertical scaling is typically a sublinear cost increase (up to a point, but that point is a ridiculous beast of a machine), since you're (typically) upgrading just the CPU and/or just the RAM or just the storage; not all of them at once. There are instances where you can get nearly 10x the machine for 2x the cost.

Also beyond a certain point, it makes sense to go straight to dedicated bare metal. The AWS tax is not worth paying if your workload is mostly fixed, somewhat fault tolerant (i.e. failed hardware on the weekends can be replaced on Monday without major interruption to business operations), and CPU bound. Get a high end machine on Hetzner and put everything behind a VPN or API auth and you will save more than 50% in sp…

I haven't found this to be true generally unless your workloads are truly completely static, which I've never actually experienced.

Given what engineers at this level cost, their costs per hour dealing with all of the nonsense clouds handle for you (networking, storage, elastic scaling, instant replacement of faulty servers, load balancing, yadda yadda) end up being higher than whatever tax you're paying for using the cloud.

Economies of scale are real.

Re: Understanding Kafka with Factorio (2019)

#65

Post by jay_kyburz »

ineedasername wrote: post #62
haswell wrote: post #44

I recently started playing Factorio, and I kept thinking that this is what "low code" integration/automation tools should look like. Developer tooling with extremely clear visuals, obvious dataflow, endless combinations into which the rigidly defined components can be assembled to do exactly what they do. As opposed to so many takes on "flow based" programming, which present some imperfect nodal representation of the…

>it's built in the shape of the reward center of my brain Yes, it's like a distillation of the feeling I get from the most enjoyable parts of my job, risking productivity loss from real world responsibilities until the complexity rises high enough to require project organization and advanced planning of tasks that are the least enjoyable parts of my job, along the lines of: "Crap, I have a sudden urgent need to deal…

Yes, but it also embodies the worst parts of my job, which is that there is always more to do and the work is never ending.

Re: Understanding Kafka with Factorio (2019)

#66

Post by RhodesianHunter »

politician wrote: post #43
mavu wrote: post #41

I clicked this and was immeasurably disapointed that the article talks about Apache Kafka, and not the author Kafka and how to understand his work with Factorio. That would be a much much better article.

[flagged]

I'm not sure how to feel about the fact that we're starting to see ChatGPT responses to questions like this in forums.

Re: Understanding Kafka with Factorio (2019)

#67

Post by morelisp »

defendBanana wrote: post #59
morelisp wrote: post #16

Earlier quoted context omitted.

Kafka is also a system that can make pretty good general use of more CPUs and more storage, but doesn't have much need for RAM. Tying the CPU and RAM together whether by CPU model or cloud vendor offerings is annoying if you're trying to scale only vertically.

Kafka can keep a decent bit of data in RAM using file system pages. Often times you end up wasting CPUs on kafka nodes, not memory i think. https://docs.confluent.io/platform/current/kafka/deployment....

I find that if you are seeking lots of consumers around large topics no amount of RAM is really sufficient, and if you are mostly sticking to the tails like a regular Kafka user, even 64GB is usually way more than enough.

CPU isn't usually a problem until you start using very large compactions, and then suddenly it can be a massive bottleneck. (Actually I would love to abuse more RAM here but log.cleaner.dedupe.buffer.size has a tiny maximum value!)

Kafka Streams (specifically) is also configured to transact by default, even though most applications aren't written to be able to actually benefit from that. If you run lots of different consumer services this results in burning a lot of CPU on transactions in a "flat profile"-y way that's hard to illustrate to application developers since each consumer, individually, is relatively small - there's just thousands of them.

Re: Understanding Kafka with Factorio (2019)

#68

Post by lakomen »

Idk... maybe it's because I'm self taught and have been coding since the age of 11, but I don't find the indirect approach helpful, the opposite.

I believe that's why OO is so popular, people who only know the object way of thinking, who have difficulties with the virtual and abstract like OO and condemn the pragmatic approach.

Re: Understanding Kafka with Factorio (2019)

#69

Post by cortesoft »

ineedasername wrote: post #62
haswell wrote: post #44

I recently started playing Factorio, and I kept thinking that this is what "low code" integration/automation tools should look like. Developer tooling with extremely clear visuals, obvious dataflow, endless combinations into which the rigidly defined components can be assembled to do exactly what they do. As opposed to so many takes on "flow based" programming, which present some imperfect nodal representation of the…

>it's built in the shape of the reward center of my brain Yes, it's like a distillation of the feeling I get from the most enjoyable parts of my job, risking productivity loss from real world responsibilities until the complexity rises high enough to require project organization and advanced planning of tasks that are the least enjoyable parts of my job, along the lines of: "Crap, I have a sudden urgent need to deal…

I wonder who the real world equivalent of biters are… sales?

Re: Understanding Kafka with Factorio (2019)

#70

Post by mettamage »

cortesoft wrote: post #69
ineedasername wrote: post #62

Earlier quoted context omitted.

>it's built in the shape of the reward center of my brain Yes, it's like a distillation of the feeling I get from the most enjoyable parts of my job, risking productivity loss from real world responsibilities until the complexity rises high enough to require project organization and advanced planning of tasks that are the least enjoyable parts of my job, along the lines of: "Crap, I have a sudden urgent need to deal…

I wonder who the real world equivalent of biters are… sales?

Security issues in software
Post reply on HN