Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How did you learn about the design/architecture of backend systems ?

news.ycombinator.com

Post reply on HN
Print view View original

11–16 of 16 posts

Re: Ask HN: How did you learn about the design/architecture of backend systems ?

#11

Post by mindcrime »

mindcrime wrote: post #2

I'm still not the world's foremost expert, but what I do know I've learned through a combination of trial and error, reading books (I'll edit this later and put in a couple of specific titles), reading stuff on the 'Net and classes I took in school (I did a degree in "High Performance Computing" which had some useful aspects to it). A good place to start, if you're not already familiar with it, is High Scalability: h…

Forgot to include it originally, but it would probably be good to study Map/Reduce as well:

http://en.wikipedia.org/wiki/MapReduce

http://labs.google.com/papers/mapreduce.html

Caching is huge too... IO is expensive, RAM access is cheap. The more you can pre-load, pre-calculate, and/or pre-sort stuff and store it in memory, the better (in terms of avoiding expensive IO anyway). Caching has it's own issues though: if you cache so aggressively that you exhaust physical ram and cause more swapping, you can actually hurt yourself. Also, you have to deal with the possibility of stale data in the cache, and determining when and how to expire and reload items in the cache. But still, caching is essential, it's just not necessarily easy.

Also, for perspective if nothing else, read the papers and stuff on SEDA (Staged Event Driven Architecture). There's still debate about how effective the SEDA approach is, but reading the discussion(s) will help you appreciate the issues involved. http://www.eecs.harvard.edu/~mdw/proj/seda/

Re: Ask HN: How did you learn about the design/architecture of backend systems ?

#12

Post by j_baker »

pkghost wrote: post #5

My answer comes from the perspective of a recent grad who has spent a year at a mid-sized SF social gaming company, working only recently on the back-end (i.e., not a scaling expert). A few things I've learned w/respect to scaling in my context: - I/O is likely to be your bottleneck, so design your db well and anticipate splitting it across multiple machines (and what that means for your access logic) - don't spawn a…

"- cache like a hoarder"

This isn't necessarily true. If the data is likely to be very volatile, it will probably be better just to ping the database than to bother trying to cache it.

Re: Ask HN: How did you learn about the design/architecture of backend systems ?

#14

Post by pkghost »

jaydub wrote: post #9
davidw wrote: post #8

Earlier quoted context omitted.

Apache does not spawn a new thread for every request.

Every request does not require a thread to be spawned (depends on configuration/load), but every connection would require its own thread. "The worker MPM uses multiple child processes with many threads each. Each thread handles one connection at a time. Worker generally is a good choice for high-traffic servers because it has a smaller memory footprint than the prefork MPM. The prefork MPM uses multiple child process…

thanks for the explanation :)

Re: Ask HN: How did you learn about the design/architecture of backend systems ?

#15

Post by yr »

mindcrime wrote: post #2

I'm still not the world's foremost expert, but what I do know I've learned through a combination of trial and error, reading books (I'll edit this later and put in a couple of specific titles), reading stuff on the 'Net and classes I took in school (I did a degree in "High Performance Computing" which had some useful aspects to it). A good place to start, if you're not already familiar with it, is High Scalability: h…

awesome list, thanks!.

Re: Ask HN: How did you learn about the design/architecture of backend systems ?

#16

Post by bpourriahi »

mindcrime wrote: post #2

I'm still not the world's foremost expert, but what I do know I've learned through a combination of trial and error, reading books (I'll edit this later and put in a couple of specific titles), reading stuff on the 'Net and classes I took in school (I did a degree in "High Performance Computing" which had some useful aspects to it). A good place to start, if you're not already familiar with it, is High Scalability: h…

Cloud computing can change 'some of this' or 'all of this' depending on the nature of the work. For many applications/requirements it could change every core design decision (when you can pay for computing by the literal hour). Well worth spending some time learning about (for the OP).
Post reply on HN