Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Notes on Distributed Systems for Young Bloods

somethingsimilar.com

Post reply on HN
Print view View original

51–57 of 57 posts

Re: Notes on Distributed Systems for Young Bloods

#51

Post by danenania »

exogenousdata wrote: post #14
sulam wrote: post #4

One that is not mentioned here but that I like as a general principle is that you cannot have exactly once delivery. At most once or at least once are both possible, but you have to pick your failure poison and architect for it.

Can’t stress this enough. Thank you for mentioning it!! In my career I’ve encountered many engineers unfamiliar with this concept when designing a distributed system.

Yeah it’s very common since it becomes an issue as soon as you get sockets involved in an app. A lot of frontend engineers unknowingly end up in distributed system land like this. Coordinating clients can be just as much a distributed systems challenge as coordinating servers—often it’s harder.

Re: Notes on Distributed Systems for Young Bloods

#52

Post by roryirvine »

pradn wrote: post #46

> Distributed systems are different because they fail often The key here is not just the rate of failure, but the rate of failure in a system of multiple nodes. And - "distributed systems problems" don't only arise with several servers connected by a network. Any set of nodes with relations between them - files on disk linked logically, buffers on different IO devices - these are also going to face similar problems.

Absolutely. In fact, it's a class of problems that can - and do - arise on any software system comprising more than a sole single-threaded process that's been locked in memory.

Some old-timers love to scoff at the inordinate amount of complexity that comes from mitigating these issues, and will complain that it would all be so much simpler if you would just run your software on a single server.

In reality, that was barely true even back in the AS/400 or VAXft days - and even then it didn't apply to the rather more chaotic multi-user, multi-process Unix world.

Re: Notes on Distributed Systems for Young Bloods

#54

Post by immibis »

bee_rider wrote: post #47
turtledragonfly wrote: post #3

Excellent list; I like the pragmatic and down-to-earth explanations. No buzzwords, no "microservices" (: I'd say that a good amount of this advice also applies to one-box systems. There can be lots of kinda/sorta distributed sub-components to consider — could be IPC between programs, or even coordination amongst threads in one process. Even the notion of unified memory on one box is a bit of a lie, but at least the h…

The neighboring universe, so tantalizingly close, where AMD gave us different memory spaces for each chiplet, is something I think about often. Imagine, we could all be writing all our code as beautiful distributed memory MPI programs. No more false sharing, we all get to think hard and explicitly about our communication patterns.

This is already here. It's called NUMA. You can access all memory from any CPU, but accessing memory that's connected to your CPU makes the access faster. NUMA-aware operating systems can limit your process to a CPU cluster and allocate memory from the same cluster, then replicate this on the other clusters, so they all run fast and only transfer data between clusters when they need to.

Re: Notes on Distributed Systems for Young Bloods

#55

Post by foota »

j-pb wrote: post #23

The article should really mention CALM (Consistency as Logical Monotonicity)[1], it's much easier to understand and a more fundamental result than CAP. It is also much more applicable and enables people with little experience to build extremely robust distributed systems. Idempotence, CRDTs, WALs, Raft, they are all special cases of the CALM principle. [1]: https://arxiv.org/pdf/1901.01930

I looked at the bloom repo and it seems somewhat stale, do you know if this is something still being worked on?

Re: Notes on Distributed Systems for Young Bloods

#56

Post by einpoklum »

> # If you can fit your problem in memory, it’s probably trivial.

A popular fallacy among some distributed systems engineers.

It's not at all trivial, it's just in a complementary domain of problems to be tackled. The fallacy easily leads to a situation where you need a 100-cluster machine to do the work that proper optimization would let you do on a single machine.

Re: Notes on Distributed Systems for Young Bloods

#57

Post by strken »

withinboredom wrote: post #34
strken wrote: post #11

Earlier quoted context omitted.

It's often not a problem because it's often easy to make a call idempotent. Consider the case where you attach an ID to every event and the subscriber stores data in postgres. You stick everything in a transaction, persist each event's ID, add a unique index to that column, handle failure, and bang, it's now idempotent.

And if that service called an external system but failed before committing the transaction? I’m not sure you should be using db transactions in distributed systems as you can’t recover from partial failures.

It depends on what you're doing. On some projects we've put the job queue into our database, which works great at low and medium traffic volumes and means we can do exactly the same thing for external systems if they let us send them an ID.

I agree that the first port of call is to make your operation legitimately idempotent without passing IDs around, and the second is to ask if it is really important enough to care about delivery, but if you're not operating at ridiculous scale and you're okay with having a single point of failure then you get to avoid the "distributed systems are hard" rule by not actually having a completely distributed system.

Post reply on HN