Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Notes on Distributed Systems for Young Bloods

somethingsimilar.com

Post reply on HN
Print view View original

41–46 of 46 posts

Re: Notes on Distributed Systems for Young Bloods

#41

Post by FourthProtocol »

pm90 wrote: post #29
FourthProtocol wrote: post #15

The timing of this one is scary. I'm working on a mobile app that does a key exchange with a server before allowing a server-based registration or login. It's nowhere near as complex as your average distributed system. That said, I've run into a scary amount of the things mentioned in this article in my tiny little use case. Just trying to ensure a decent user experience (timing out a comms check after two seconds ra…

Just a heads up: here's a course taught by Robert Morris that goes through a lot of great stuff: http://pdos.csail.mit.edu/6.824/

Going through it now!

Re: Notes on Distributed Systems for Young Bloods

#42

Post by [deleted] »

pm90 wrote: post #29
FourthProtocol wrote: post #15

The timing of this one is scary. I'm working on a mobile app that does a key exchange with a server before allowing a server-based registration or login. It's nowhere near as complex as your average distributed system. That said, I've run into a scary amount of the things mentioned in this article in my tiny little use case. Just trying to ensure a decent user experience (timing out a comms check after two seconds ra…

Just a heads up: here's a course taught by Robert Morris that goes through a lot of great stuff: http://pdos.csail.mit.edu/6.824/

[deleted]

Re: Notes on Distributed Systems for Young Bloods

#43

Post by jreichhold »

squarecog wrote: post #34
benjaminwootton wrote: post #33

Earlier quoted context omitted.

I view this the opposite way around. Message queues and asynchronicity are an alternative to backpressure as described. If I synchronously hit some service then I need to build facilities into the service to say that it's too heavily loaded, and perhaps handle those responses on the client. If I pop some job onto a message queue or asynchronous endpoint, things will just slow down for a while. Which is normally as go…

How do you keep the message queue from unbounded growth?

Arrival rate and dequeue/processing time are the keys here. Unbounded queues are easy.. Just limit the queue size. Of course this isn't what the user of the service wants and leads to a very bad experience

Re: Notes on Distributed Systems for Young Bloods

#44

Post by jreichhold »

Very glad that Jeff wrote this up and he deservers mad credit for documenting something that has been frustrating me for years. People don't realize how hard doing this right is and discount the hard work that goes into making large systems scale in a stable manner.

It isn't just load testing but more that the whole system should be considered suspect. If you don't act defensively at all steps you will be hosed by something you thought will never happen. Just had a good talk about this last weekend. Memory, TCP, and all other rock-solid things can and will have issues in large systems.

Re: Notes on Distributed Systems for Young Bloods

#45

Post by peterwwillis »

saosebastiao wrote: post #38
peterwwillis wrote: post #31

Earlier quoted context omitted.

One that has buffers.

Don't they all have buffers?

Yes. Depending on load and services, servers require a great deal more in buffers and cache than your desktop. Figure out the memory allocation for a single TCP connection on a default Linux 3.0 kernel and multiply that times (at least) 10k. Considering you probably want greatly expanded limits and bulk transfer, you might want to multiply that times 10 or 100. That's just TCP buffers for active sessions.

Re: Notes on Distributed Systems for Young Bloods

#46

Post by saosebastiao »

peterwwillis wrote: post #45
saosebastiao wrote: post #38

Earlier quoted context omitted.

Don't they all have buffers?

Yes. Depending on load and services, servers require a great deal more in buffers and cache than your desktop. Figure out the memory allocation for a single TCP connection on a default Linux 3.0 kernel and multiply that times (at least) 10k. Considering you probably want greatly expanded limits and bulk transfer, you might want to multiply that times 10 or 100. That's just TCP buffers for active sessions.

Thanks for the explanation
Post reply on HN