Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Distributed Systems Reading List

dancres.github.io

Post reply on HN
Print view View original

71–77 of 77 posts

Re: Distributed Systems Reading List

#71

Post by 0xbadcafebee »

webmaven wrote: post #65
0xbadcafebee wrote: post #57

Earlier quoted context omitted.

Distributed systems research has been going on since the 70's and Unix Neckbeards have probably forgotten more about them than we have learned, so actually I think impostor syndrome is a bit warranted with them. The actual hard stuff is not even these papers, it's the implementations that are way more complex than some algorithm or architectural pattern. Anyone who says "X is better than Y" is fooling themselves beca…

> The only thing you can say for certain is that reducing the amount of components and complexity in the system often results in better outcomes. No, there are a few other things that you can say for certain: Watch out for positive-only feedback loops, you absolutely need negative feedback as well - or only. Eg. exponential back-off. Sometimes, you just need a decentralized solution, rather than a distributed one, an…

Maybe there are some other things you can say for certain. But as to some of your points:

> Watch out for positive-only feedback loops, you absolutely need negative feedback as well - or only. Eg. exponential back-off.

Agreed it may need negative feedback, but I'm not sure about always.

If your service has a latency SLA, exponential back-off might kill your SLA (depending on wordage and where the back-off is). The fix is to soft reject requests (RST rather than dropping packets) when you can't meet the demand. This change may allow you to meet your SLA if it's written to prioritize low latency over service unavailability.

This is it's own negative feedback loop, but change from sending RSTs to silently dropping and you no longer have the feedback.

> Sometimes, you just need a decentralized solution, rather than a distributed one

Agreed

> Loose coupling is your friend.

Until it isn't? :)

> add an extra layer of indirection, but you probably need to pay more attention to cache invalidation

Fixes for additional layers tend to increase system complexity compared to fixes for fewer layers.

> Throughput probably matters more than latency.

Until it doesn't :)

> Reducing the size/number of writes will probably help more than trying to speed them up.

Depending on 20 different things... You really have to account for all the system's limits (and business use cases) and find the solution that matches the implementation needs.

> there is probably a huge business for multi-tenancy-as-a-service

Sure, it's called EKS :-) Just build more clusters... Don't worry, we'll bill you...

> Don't overthink it

Yes and no; Yes, in that there will always be unknowns. But no, in that often improvements in communication will provide better solutions without extra work. Think smarter, not harder!

Re: Distributed Systems Reading List

#72

Post by pron »

tyu2 wrote: post #45
mjb wrote: post #42

Earlier quoted context omitted.

> Which is directly at odds with needing TLA+, because if you do need it, it means the complexity of the algorithms is so great, that you won't be able to keep them in your head and understand every aspect of their performance to make something work well. I don't think this is correct. What TLA+ allows us to do is be more creative in our design and choice of algorithms, while allowing the computer to help us reason a…

That's the thing, performance should dictate the algorithms, not the other way around and TLA+ can't make this process any easier, only harder. I get it's not the case at AWS, where distributed services AWS thinks customers might want is what dictates the choices, but this is an exception, not the rule and unless someone wants to work there they have no reason to be doing it this way, especially not for educational p…

But TLA+ is used to shorten development and help with algorithm performance.

Re: Distributed Systems Reading List

#73

Post by pron »

tyu2 wrote: post #37
ahelwer wrote: post #34

I guess I'm pretty opinionated about this, but it was odd the author talked about the necessity of changing the way you think without also including anything about TLA+. IMO the "way you think" about distributed systems - if you want to be effective - will basically end up looking exactly like you think when writing a TLA+ spec, and learning TLA+ is a fast-track method of thinking like a distributed systems engineer.…

Is it though? The hard part about distributed systems is performance in our crappy real world environment with unreliable poorly performing and faulty public internet, unreliable hardware, OSes, etc. Which is directly at odds with needing TLA+, because if you do need it, it means the complexity of the algorithms is so great, that you won't be able to keep them in your head and understand every aspect of their perform…

> Which is directly at odds with needing TLA+, because if you do need it, it means the complexity of the algorithms is so great, that you won't be able to keep them in your head and understand every aspect of their performance to make something work well.

Having used TLA+ for years, I would say it's the exact opposite. All bugs happen due to someone believing something is simple enough to work out in their head while it actually isn't. So if your judgment about what you can keep in your head is good, you never have any bugs and you really don't need TLA+. But if you do happen to have bugs occasionally, then your belief about how much you can keep in your head is sometimes wrong. TLA+ is a very quick way to write down what's in your head so you can think about it more rigorously. Surely, if you can truly keep it in your head, it should be easy for you to write it down precisely. And just in case you're wrong, there are tools that can check if you're right, just to be extra sure. In short, TLA+ helps if you ever have bugs. It doesn't help if you never do.

Re: Distributed Systems Reading List

#74

Post by lostcolony »

cbsmith wrote: post #70
lostcolony wrote: post #49

Earlier quoted context omitted.

Do you know what the CAP theorem is, and can explain it to me like I'm 5? Can you tell me how a SQL DB fits into it, and where something like DynamoDB fits into it? Congratulations, you are better than 95% of the people that I've interviewed out there saying they are experienced building distributed systems. Including system/solution architects.

Sad, but true. That said... the hiring pool tends to be biased towards people that others have passed on.

Sure, but the OP was talking about their difficulty during interviews; I'm just saying, they're almost assuredly better than the majority of the rest of the hiring pool. The company has an opening they need filled; they can only fill it with people from the hiring pool, not those who aren't in the hiring pool.

Re: Distributed Systems Reading List

#75

Post by webmaven »

0xbadcafebee wrote: post #71
webmaven wrote: post #65

Earlier quoted context omitted.

> The only thing you can say for certain is that reducing the amount of components and complexity in the system often results in better outcomes. No, there are a few other things that you can say for certain: Watch out for positive-only feedback loops, you absolutely need negative feedback as well - or only. Eg. exponential back-off. Sometimes, you just need a decentralized solution, rather than a distributed one, an…

Maybe there are some other things you can say for certain. But as to some of your points: > Watch out for positive-only feedback loops, you absolutely need negative feedback as well - or only. Eg. exponential back-off. Agreed it may need negative feedback, but I'm not sure about always . If your service has a latency SLA, exponential back-off might kill your SLA (depending on wordage and where the back-off is). The f…

Every maxim has caveats and exceptions. Including this one.

Re: Distributed Systems Reading List

#76

Post by BenoitP »

tyu2 wrote: post #45
mjb wrote: post #42

Earlier quoted context omitted.

> Which is directly at odds with needing TLA+, because if you do need it, it means the complexity of the algorithms is so great, that you won't be able to keep them in your head and understand every aspect of their performance to make something work well. I don't think this is correct. What TLA+ allows us to do is be more creative in our design and choice of algorithms, while allowing the computer to help us reason a…

That's the thing, performance should dictate the algorithms, not the other way around and TLA+ can't make this process any easier, only harder. I get it's not the case at AWS, where distributed services AWS thinks customers might want is what dictates the choices, but this is an exception, not the rule and unless someone wants to work there they have no reason to be doing it this way, especially not for educational p…

> performance should dictate the algorithms, not the other way

You want correctness first, and performance second. But these two are very much intertwined. And knowing exactly where the boundary is will help you co-design them.

Some AWS engineers have said the following[1]:

"TLA+ [...] giving us enough understanding and confidence to make aggressive performance optimizations without sacrificing correctness."

[1] https://blog.acolyer.org/2014/11/24/use-of-formal-methods-at...

Re: Distributed Systems Reading List

#77

Post by ethanfinni »

shred45 wrote: post #25

I find "distributed systems" to be a huge source of imposter syndrome. Despite having worked almost exclusively with distributed applications for several years now, it is difficult to consider myself experienced. When I'm asked if I've worked with distributed systems, I don't think they are asking me if I've managed a Hadoop cluster. They are interested in building new applications using some of the primitives discus…

Explain concurrency like I am five and give a concurrency example problem and solution.
Post reply on HN