Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Learn how to design large-scale systems

github.com

Post reply on HN
Print view View original

121–130 of 199 posts

Re: Learn how to design large-scale systems

#121

Post by e12e »

outworlder wrote: post #105
sillysaurus3 wrote: post #56

Earlier quoted context omitted.

To see how quickly HN changed in the old days, skim the feature requests thread: https://news.ycombinator.com/item?id=363 Some feature requests were a matter of days, like user profiles: https://news.ycombinator.com/item?id=481 A brief essay on some HN design decisions: https://pastebin.com/bSW5dfRQ (from https://news.ycombinator.com/item?id=8424502 ) I think the arc codebase is worth studying and understanding, prim…

> I think the arc codebase is worth studying and understanding Are you talking about the source code for ARC, or for Hacker News? It would be interesting to see how ARC is being employed on such a high profile site. I expect that some algorithms won't be freely available so as not enable people to game the site, but the rest would be interesting to see. I could find any source for Hacker News though. > And since code…

> I could find any source for Hacker News though.

Current hn code (as in the actual code that delivers this comment) isn't open AFAIK (partly because of the shadow banning, filtering etc code.

But there's a full "news" site in Arc source - old and more maintained/evolved:

https://github.com/arclanguage/anarki/blob/master/lib/news.a...

http://arclanguage.org/install

Re: Learn how to design large-scale systems

#122

Post by sillysaurus3 »

e12e wrote: post #121
outworlder wrote: post #105

Earlier quoted context omitted.

> I think the arc codebase is worth studying and understanding Are you talking about the source code for ARC, or for Hacker News? It would be interesting to see how ARC is being employed on such a high profile site. I expect that some algorithms won't be freely available so as not enable people to game the site, but the rest would be interesting to see. I could find any source for Hacker News though. > And since code…

> I could find any source for Hacker News though. Current hn code (as in the actual code that delivers this comment) isn't open AFAIK (partly because of the shadow banning, filtering etc code. But there's a full "news" site in Arc source - old and more maintained/evolved: https://github.com/arclanguage/anarki/blob/master/lib/news.a... http://arclanguage.org/install

Whoops, that's Anarki, not Arc.

pg's and rtm's original arc3.1 is on the "official" branch: https://github.com/arclanguage/anarki/tree/official

news.arc is the old HN source code. You can run it by following the steps in how-to-run-news.

(Run it with "mzscheme -f as.scm" though, not mzscheme.)

Re: Learn how to design large-scale systems

#123

Post by e12e »

sillysaurus3 wrote: post #9

Note that HN, a top-1000 site in the US, runs on a single box via a single racket process. "The key to performance is elegance, not battalions of special cases."

A better example might be stack overflow that ran on four(?) boxes for a long while (two sql server s, two iis/dot.net application/web servers).

They now have a few more bits: https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar...

Re: Learn how to design large-scale systems

#124

Post by ChristianGeek »

bandwitch wrote: post #29
agentultra wrote: post #7

I'd add a section on using TLA+ as a design tool. Diagrams and rules of thumb are useful but they don't catch errors or help you discover the correct architecture. See the Amazon paper [0] on their use of TLA+ in designing (and trouble-shooting) services. [0] https://lamport.azurewebsites.net/tla/formal-methods-amazon....

I feel TLA+ would be too much to ask in a system interview which is what this site is about. In case anybody is interested, there is a nice talk by Hillel Wayne on youtube ( https://www.youtube.com/watch?v=_9B__0S21y8 ) that provides a high-level overview on what TLA+ is about.

It might be too much to ask as an interviewer, but it wouldn’t be too much to offer as a candidate. Sit there submissively and only give me what I ask and you might impress me with you skills. Be proactive and get me engaged over something I haven’t seen before or that I wouldn’t expect you to know and you have a shot at wowing me with your potential.

Re: Learn how to design large-scale systems

#125

Post by e12e »

Interesting how the write api doesn't appear to invalidate/update the memory cache in the first diagram.

Still recommend people read Fielding's REST thesis - as it demonstrates a lot of possible architectures (eg fat client or what we today call SPAs) - not simply REST. Along with some trade-offs. (REST is mainly motivated by simplicity of a simple hypertext application coupled with easy multi-level caching).

https://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm

For a preview of SPAs before the prevalence of Javascript, see 3.5, in particular 3.5.3 "code on demand":

https://www.ics.uci.edu/~fielding/pubs/dissertation/net_arch...

And keep in mind the text is from 2000. Early Ajax was introduced in IE in 1999, and late 2000 in Mozilla - but it took a while for Ajax to become standardized...

Re: Learn how to design large-scale systems

#126

Post by e12e »

yread wrote: post #116
lalwanivikas wrote: post #84

Earlier quoted context omitted.

you need to provide more details to get any useful advice. but just based on what you have described, any db would do the job. add a caching layer and you have your low latencies. again, what is the traffic and bandwidth load like? peak and average values? what kind of data are you planning to store? small values but huge volumes or the opposite? a lot will change based on your system requirements.

To clarify: let's say I have servers in two locations A and B that are 200ms from each other. When I issue a write to the db in A I don't want to wait (multiples) of the 200ms before it returns. I don't really care whether the write appears to a reader at B in 5s or 50 minutes but of course the writes have to be at least causally consistent. I won't have millions (realistically not even thousands) of users and the da…

Can you write to a single rdms C, and maybe cache reads at A and B?

Re: Learn how to design large-scale systems

#127

Post by cjhanks »

rargulati wrote: post #120
cjhanks wrote: post #103

I see something comparable to these diagrams (it feels like) a half-dozen times a year. The architecture is in general 'fine'. But communication paths of subsystems is probably the easiest part of the problem. And in general, re-organizing the architecture of a system is usually possible - if and only if - the underlying data model is sane. The more important questions are; - What is the convention for addressing ass…

Outside of raw experience, what can you do, read, or learn to build the intuition for formulating and answering the above questions? I can answer the above for systems I've built, but I've spent quite a bit of time with those systems. How do I get better at doing this during the planning phases, or even better, for a system I'm unfamiliar with (ie. are there tools you lean on here)?

A bit of caution, I haven't worked in distributed systems for some time now. And I am sure there many people more competent than I.

But in general the cliche of "Great artists steal" applies here. If AWS/GCE/Azure (or any other major software vendor) is offering a service or a feature, then it is almost certainly solving a problem somebody has. If you don't understand what problem is being solved, then you cannot possibly account for that problem in your design. Today, the manuals for these software features are documented in unprecedented accuracy. Read them, and try to reverse engineer in your head how you would build them.

For example; AWS' IAM roles seems like a problem which could be solved by far more trivial solutions. Just put permissions in a DB and query it when a user wants to do something. Why do we need URN's for users, resources, services, operations, etc? And why do those URN's need to map to URI's? Well, if you look at the problem - it ends up being a big graph which is in the general case immutable over namedspaced assets. So reverse engineer that, how would you build that?

Re: Learn how to design large-scale systems

#128

Post by s_ngularity »

bsenftner wrote: post #82
mabbo wrote: post #41

This design, roughly, is being used very widely and is well-documented everywhere. But does anyone know of any lesser-known yet equally functional designs that work at the same scale? Are there cases this design does not work for?

Yes. One can use a C++ library like Restbed and embed the web server directly into a compiled executable that uses SQLite as an embedded database. The "large-scale, multi-system architecture" in such common use today is completely unnecessary when faced with this setup. I have multiple Restbed integrated applications whose entire disk footprint is 7MB; they can run on a $99 Intel Compute Stick, perform industrial gra…

What about fault-tolerance though? That's definitely a single point of failure scenario.

Re: Learn how to design large-scale systems

#129

Post by swaggyBoatswain »

nemo1618 wrote: post #73
madethemcry wrote: post #13

Oh interesting, I have never seen Anki ( https://apps.ankiweb.net/ ) being used for large blocks of source code. Anki is an open source application (desktop + mobile) for spaced repetition learning (aka flashcards). It's a very popular tool among people who want to learn languages (and basically anything else you want to remember). There are many shared decks ( https://ankiweb.net/shared/decks/ ). Creating and format…

I've been experimenting with Anki recently. I loaded up a deck of popular fonts, with the goal being to memorize them to the point where I could recognize them in the real world. Each card contains the sentence, "The quick brown fox jumped over the lazy dog," and you have to identify what font it's written in. The first day was really tough; I missed cards over and over again. 20 new cards (the default) is probably t…

I personally don't see the point of memorizing font types, but I like to use fontninja / chrome debugger to find out the font https://chrome.google.com/webstore/detail/fontface-ninja/elj...

My personal favorite webfonts are proxima nova (for commercial) and roboto(for free google fonts) for modern web typesetting

Re: Learn how to design large-scale systems

#130

Post by ris »

I'm quite tired of everyone wanting to build "large scale systems" and play at being Netflix. The truth of the matter is the vast vast majority of people will never need to do this with their project and instead will just end up making an expensive to maintain mess with way too many moving parts.

At least as important as designing something that can scale up is designing something that can scale down. You don't know when the organization's going to need to deprioritize this project and be able to keep it running without burning a couple of million in resources every year.

See: microservices. (as in, for the problem, not the solution)

Post reply on HN