Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Distributed Systems Reading List

dancres.github.io

Post reply on HN
Print view View original

41–50 of 77 posts

Re: Distributed Systems Reading List

#41

Post by smiths1999 »

tyu2 wrote: post #37
ahelwer wrote: post #34

I guess I'm pretty opinionated about this, but it was odd the author talked about the necessity of changing the way you think without also including anything about TLA+. IMO the "way you think" about distributed systems - if you want to be effective - will basically end up looking exactly like you think when writing a TLA+ spec, and learning TLA+ is a fast-track method of thinking like a distributed systems engineer.…

Is it though? The hard part about distributed systems is performance in our crappy real world environment with unreliable poorly performing and faulty public internet, unreliable hardware, OSes, etc. Which is directly at odds with needing TLA+, because if you do need it, it means the complexity of the algorithms is so great, that you won't be able to keep them in your head and understand every aspect of their perform…

I am not a "TLA+ promoter" but think it is a very valuable tool for anyone building distributed systems. The value of TLA+ is that it forces you to carefully consider your algorithm, which is certainly important if the algorithm is complex but equally important if the algorithm is simple. Most people will struggle to correctly specify even a simple algorithm in TLA+ because they will miss a lot of things they had assumed without ever thinking about.

Real world systems need to handle all the things you mention. TLA+ helps you consider all these issues with spelling them out individually. There is no point in building a complex system if you haven't taken the time to validate the correctness of the target system in the first place.

Re: Distributed Systems Reading List

#42

Post by mjb »

tyu2 wrote: post #37
ahelwer wrote: post #34

I guess I'm pretty opinionated about this, but it was odd the author talked about the necessity of changing the way you think without also including anything about TLA+. IMO the "way you think" about distributed systems - if you want to be effective - will basically end up looking exactly like you think when writing a TLA+ spec, and learning TLA+ is a fast-track method of thinking like a distributed systems engineer.…

Is it though? The hard part about distributed systems is performance in our crappy real world environment with unreliable poorly performing and faulty public internet, unreliable hardware, OSes, etc. Which is directly at odds with needing TLA+, because if you do need it, it means the complexity of the algorithms is so great, that you won't be able to keep them in your head and understand every aspect of their perform…

> Which is directly at odds with needing TLA+, because if you do need it, it means the complexity of the algorithms is so great, that you won't be able to keep them in your head and understand every aspect of their performance to make something work well.

I don't think this is correct.

What TLA+ allows us to do is be more creative in our design and choice of algorithms, while allowing the computer to help us reason about whether the choices we're making still result in a system that is correct. "Correct" in in this context means two things: "safe" as in it doesn't lose or corrupt data, and "live" as in it eventually makes progress without deadlock or other blockers. That doesn't capture "meets the SLA" or "fast enough for real use" or even "tolerates gray failures". All of those are critical properties indeed - but unless you have fundamental safety and liveness you're never going to get those properties anyway. You might think you have them, but then you'll have a bad time eventually.

So TLA+ (and similar tools) aren't a complete solution to the problem, but they are an exceptionally useful one. Fundamentally, they're useful because distributed and concurrent protocols, even very simple ones like 2PC, are wickedly difficult to reason about clearly. Computers can help us reason, and specification languages can help us communicate clearly about our reasoning.

Re: Distributed Systems Reading List

#43

Post by libraryofbabel »

Good list, but is it still being actively updated? Not having Kleppmann’s seminal Designing Data-Intensive Applications (2017) on it would indicate no.

Alex Petrov’s Database Internals: A Deep Dive Into How Distributed Data Systems Work (2019) is another essential recent reference that should be here. Not as broad as Kleppmann but dives a lot deeper into certain topics.

Re: Distributed Systems Reading List

#44

Post by znpy »

herodoturtle wrote: post #19
projectileboy wrote: post #13

Great list. Only thing I’d add for the other enterprise developers out there is to first default to not building a distributed system at all, but rather build a much smaller monolith. In 25 years I’ve worked for so many orgs that wanted to build The World’s Most Scalable System for what would maybe be a few hundred concurrent users. Not surprisingly, those projects tend to tank.

Hear hear. This is really important wisdom being shared. And with a single monolith you can comfortably handle several thousand concurrent users, not just several hundred. From my humble experience I've found that it is also relatively easier to migrate an established monolith to a semi-distributed system, than it is to build and scale a distributed system from the ground up whilst at the same time trying to figure o…

Ah, i kinda agree. My former employer developed and sold an editorial suite architected in about 1999-2000 around the SOA model (along with corba) and later evolved through the years.

So basically some monoliths playing together.

I've always been admiring that design and how much it could do and handle with so little resources.

Most times for a whole editorial newsroom a single 48c/128gb machine can be more than enough.

They were particularly smart on avoiding relational databases and sticking to an object database (Versant oodbms, another mostly unknown marvel of software engineering).

That software, even with its shortcomings, was really a great piece of software engineering.

Re: Distributed Systems Reading List

#45

Post by tyu2 »

mjb wrote: post #42
tyu2 wrote: post #37

Earlier quoted context omitted.

Is it though? The hard part about distributed systems is performance in our crappy real world environment with unreliable poorly performing and faulty public internet, unreliable hardware, OSes, etc. Which is directly at odds with needing TLA+, because if you do need it, it means the complexity of the algorithms is so great, that you won't be able to keep them in your head and understand every aspect of their perform…

> Which is directly at odds with needing TLA+, because if you do need it, it means the complexity of the algorithms is so great, that you won't be able to keep them in your head and understand every aspect of their performance to make something work well. I don't think this is correct. What TLA+ allows us to do is be more creative in our design and choice of algorithms, while allowing the computer to help us reason a…

That's the thing, performance should dictate the algorithms, not the other way around and TLA+ can't make this process any easier, only harder. I get it's not the case at AWS, where distributed services AWS thinks customers might want is what dictates the choices, but this is an exception, not the rule and unless someone wants to work there they have no reason to be doing it this way, especially not for educational purposes learning distributed systems.

Re: Distributed Systems Reading List

#46

Post by markc »

libraryofbabel wrote: post #43

Good list, but is it still being actively updated? Not having Kleppmann’s seminal Designing Data-Intensive Applications (2017) on it would indicate no. Alex Petrov’s Database Internals: A Deep Dive Into How Distributed Data Systems Work (2019) is another essential recent reference that should be here. Not as broad as Kleppmann but dives a lot deeper into certain topics.

Agreed, I looked for Kleppmann and didn't find it. Red flag.

Re: Distributed Systems Reading List

#47

Post by abledon »

shred45 wrote: post #25

I find "distributed systems" to be a huge source of imposter syndrome. Despite having worked almost exclusively with distributed applications for several years now, it is difficult to consider myself experienced. When I'm asked if I've worked with distributed systems, I don't think they are asking me if I've managed a Hadoop cluster. They are interested in building new applications using some of the primitives discus…

great time to bring up the Dunning-Kruger phenomenon in the interview lol

Re: Distributed Systems Reading List

#48

Post by abledon »

How long would it take to realistically get through all these books? for the average developer... i'm thinking 3-4 years? and this includes taking weekends outside of work to painfully work through every chapter (+ break weekends since working+studying this in evening would probably lead to burnout in 1 year)

Re: Distributed Systems Reading List

#49

Post by lostcolony »

shred45 wrote: post #25

I find "distributed systems" to be a huge source of imposter syndrome. Despite having worked almost exclusively with distributed applications for several years now, it is difficult to consider myself experienced. When I'm asked if I've worked with distributed systems, I don't think they are asking me if I've managed a Hadoop cluster. They are interested in building new applications using some of the primitives discus…

Do you know what the CAP theorem is, and can explain it to me like I'm 5? Can you tell me how a SQL DB fits into it, and where something like DynamoDB fits into it?

Congratulations, you are better than 95% of the people that I've interviewed out there saying they are experienced building distributed systems. Including system/solution architects.

Re: Distributed Systems Reading List

#50

Post by johnnujler »

abledon wrote: post #48

How long would it take to realistically get through all these books? for the average developer... i'm thinking 3-4 years? and this includes taking weekends outside of work to painfully work through every chapter (+ break weekends since working+studying this in evening would probably lead to burnout in 1 year)

I would wager around a year at max. Most materials related to Distributed Systems are information heavy(requiring you to read and remember) as opposed to being math heavy(with the possible exception of graph theory, which usually none of the fundamental texts expect the readers to know). Even many popular papers like raft, paxos, chubby, BitTorrent, gfs etc do not require much mathematical knowledge beyond basic arithmetic. You can read them like a non-fiction and understand most of the things.

Not saying that Distributed Systems is easy, doing research in Distributed Systems is very challenging, but I don’t think reading and understanding these materials should be that difficult for an average software engineer.

Post reply on HN