Distributed Systems Reading List • Hacker News Board

Re: Distributed Systems Reading List

#51

Post by toast0 » Tue, Dec 08, 2020, 4:41 PM UTC

amelius wrote: post #18

projectileboy wrote: post #13

Great list. Only thing I’d add for the other enterprise developers out there is to first default to not building a distributed system at all, but rather build a much smaller monolith. In 25 years I’ve worked for so many orgs that wanted to build The World’s Most Scalable System for what would maybe be a few hundred concurrent users. Not surprisingly, those projects tend to tank.

The problem with scaling is that visitor count often goes exponentially. So while in the beginning everything seems calm and quiet, suddenly you can see a massive spike in interest. And if you didn't take the right measures, you might lose all that interest.

There's times when you might be able to expect a huge spike, but even then, if the product is early, so the overall design is still pretty fluid, I would try to keep flexible.

It's more important to be aware of overload, and provide feedback to users (or potential users) and some form of load shedding. And test that all. For a lot of applications, being able to quickly plop together a second monolith could work to address spikes. Or switching to a bigger machine: Epyc 2-socket systems get you up to 128 cores (256 SMT threads), and I think 8TB of ram. You can do an awful lot with that.

Re: Distributed Systems Reading List

#52

Post by stuxnet79 » Tue, Dec 08, 2020, 5:01 PM UTC

abledon wrote: post #48

How long would it take to realistically get through all these books? for the average developer... i'm thinking 3-4 years? and this includes taking weekends outside of work to painfully work through every chapter (+ break weekends since working+studying this in evening would probably lead to burnout in 1 year)

IMO a better approach would be to read Designing Data Intensive Applications (mentioned a million times on HN) which is more like a high-level map of the field. The references in DDIA are also a goldmine of information.

You don't have to read DDIA front to back. Just picking a topic (for instance "Distributed Transactions") is enough to get you started building an intuition about these issues.

Re: Distributed Systems Reading List

#53

Post by chrisweekly » Tue, Dec 08, 2020, 5:11 PM UTC

abledon wrote: post #47

shred45 wrote: post #25

I find "distributed systems" to be a huge source of imposter syndrome. Despite having worked almost exclusively with distributed applications for several years now, it is difficult to consider myself experienced. When I'm asked if I've worked with distributed systems, I don't think they are asking me if I've managed a Hadoop cluster. They are interested in building new applications using some of the primitives discus…

great time to bring up the Dunning-Kruger phenomenon in the interview lol

hahaha

for the uninitiated,

The Dunning–Kruger effect is a cognitive bias in which people with low ability at a task overestimate their ability.

Re: Distributed Systems Reading List

#54

Post by flavius29663 » Tue, Dec 08, 2020, 5:44 PM UTC

znpy wrote: post #44

herodoturtle wrote: post #19

Earlier quoted context omitted.

Hear hear. This is really important wisdom being shared. And with a single monolith you can comfortably handle several thousand concurrent users, not just several hundred. From my humble experience I've found that it is also relatively easier to migrate an established monolith to a semi-distributed system, than it is to build and scale a distributed system from the ground up whilst at the same time trying to figure o…

Ah, i kinda agree. My former employer developed and sold an editorial suite architected in about 1999-2000 around the SOA model (along with corba) and later evolved through the years. So basically some monoliths playing together. I've always been admiring that design and how much it could do and handle with so little resources. Most times for a whole editorial newsroom a single 48c/128gb machine can be more than enou…

People today don't realize how cheap you can get a really upscaled machine. In Azure, a 48 core, 192GB machine is 2000 USD a month. This is nothing compared to upscaling your app on 10 servers and 20 services, with a lot of overhead, maintenance and dev time.

If you can get away with a monolith, it's the way to go, as long as you leave room to quickly grow horizontally as well, if needed.

Re: Distributed Systems Reading List

#55

Post by nwsm » Tue, Dec 08, 2020, 5:58 PM UTC

kohlerm wrote: post #2

good list but for me the reference is now Martin Kleppmanns new lecture: https://www.youtube.com/playlist?list=PLeKd45zvjcDFUEv_ohr_H...

Thanks for sharing! I love having my copy of DDIA and am glad to have this resource from Martin as well.

Re: Distributed Systems Reading List

#56

Post by cyberfart » Tue, Dec 08, 2020, 5:59 PM UTC

Another really good reading about distributed systems is Kyle Kingsbury's (aphyr) distsys-class notes:

https://github.com/aphyr/distsys-class

Re: Distributed Systems Reading List

#57

Post by 0xbadcafebee » Tue, Dec 08, 2020, 6:18 PM UTC

calcorbin wrote: post #29

shred45 wrote: post #25

I find "distributed systems" to be a huge source of imposter syndrome. Despite having worked almost exclusively with distributed applications for several years now, it is difficult to consider myself experienced. When I'm asked if I've worked with distributed systems, I don't think they are asking me if I've managed a Hadoop cluster. They are interested in building new applications using some of the primitives discus…

I'm realizing I've only worked in distributed systems as well, but I'd never feel comfortable telling potential employers I'm an expert. Being an expert in distributed systems seems almost too broad. At a high level couldn't it be expertise at integration, accessible logging, and configuration?

Distributed systems research has been going on since the 70's and Unix Neckbeards have probably forgotten more about them than we have learned, so actually I think impostor syndrome is a bit warranted with them.

The actual hard stuff is not even these papers, it's the implementations that are way more complex than some algorithm or architectural pattern. Anyone who says "X is better than Y" is fooling themselves because it's only the implementation context that matters.

The only thing you can say for certain is that reducing the amount of components and complexity in the system often results in better outcomes.

Re: Distributed Systems Reading List

#58

Post by linux2647 » Tue, Dec 08, 2020, 6:58 PM UTC

libraryofbabel wrote: post #43

Good list, but is it still being actively updated? Not having Kleppmann’s seminal Designing Data-Intensive Applications (2017) on it would indicate no. Alex Petrov’s Database Internals: A Deep Dive Into How Distributed Data Systems Work (2019) is another essential recent reference that should be here. Not as broad as Kleppmann but dives a lot deeper into certain topics.

EDIT: I was looking at the wrong branch, but still it hasn't been updated since 2018: https://github.com/dancres/Pages/commits/gh-pages

Looks like it was last updated in 2013: https://github.com/dancres/Pages/commits/master (there's a commit from 2018, but it doesn't touch the actual page)

Re: Distributed Systems Reading List

#59

Post by pratik661 » Tue, Dec 08, 2020, 7:42 PM UTC

shred45 wrote: post #25

I find "distributed systems" to be a huge source of imposter syndrome. Despite having worked almost exclusively with distributed applications for several years now, it is difficult to consider myself experienced. When I'm asked if I've worked with distributed systems, I don't think they are asking me if I've managed a Hadoop cluster. They are interested in building new applications using some of the primitives discus…

I wonder if this source of imposter syndrome applies to any field where the job roles and success criteria are not clearly defined and vary from company to company (ie data analyst, product owner, distributed systems developer).

Re: Distributed Systems Reading List

#60

Post by alexjplant » Tue, Dec 08, 2020, 8:06 PM UTC

kortex wrote: post #33

herodoturtle wrote: post #19

Earlier quoted context omitted.

Hear hear. This is really important wisdom being shared. And with a single monolith you can comfortably handle several thousand concurrent users, not just several hundred. From my humble experience I've found that it is also relatively easier to migrate an established monolith to a semi-distributed system, than it is to build and scale a distributed system from the ground up whilst at the same time trying to figure o…

The trick is to find "fault lines" throughout your monolith - places where it's easy to create separation of concerns and narrow interfaces. Maintain and document these like they are external APIs. Only pass simple data through it (serialized or anything easy to serialize). When the time comes, cleave these chunks off and wrap them in RPC/REST/graphql. It's good form even for monoliths since it makes for an easy inte…

+1 to you sir. I was just having a discussion this past weekend with some technical friends and noted that a well-architected monolith often lends itself to easy deconstruction such that you get the best of both worlds when they're necessary in your product lifecycle.