Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Learn how to design large-scale systems

github.com

Post reply on HN
Print view View original

51–60 of 199 posts

Re: Learn how to design large-scale systems

#51

Post by madethemcry »

byebyetech wrote: post #26
madethemcry wrote: post #13

Oh interesting, I have never seen Anki ( https://apps.ankiweb.net/ ) being used for large blocks of source code. Anki is an open source application (desktop + mobile) for spaced repetition learning (aka flashcards). It's a very popular tool among people who want to learn languages (and basically anything else you want to remember). There are many shared decks ( https://ankiweb.net/shared/decks/ ). Creating and format…

I don't think Anki supposed to be used that way. Each card should be recallable under 10 seconds. So it should be only few lines of content. More content you put in one Anki card, it will take you more time and eventually you will stop looking at the card. A failure scenario.

Yeah that might be the reason I have never seen this. But whatever works for oneself is fine and worth to share.

There are quite nice deck ideas:

  + Facts about friends
  + Standard Library of your programming language 
    (https://ankiweb.net/shared/info/3937203746)
  + Bird Voices
    (https://ankiweb.net/shared/info/2088996377)
More here: https://www.reddit.com/r/Anki/comments/5ka1ny/what_have_you_...

Re: Learn how to design large-scale systems

#52

Post by spraak »

0xbadcafebee wrote: post #21
tryonqc wrote: post #6

Earlier quoted context omitted.

He/she means this one: https://news.ycombinator.com/item?id=17517155 Today's post is way more in-depth. Good follow-up indeed.

Obligatory pedantic HN grammar comment: on the outside chance that the gp's gender is not binary, the word 'they' is a good stand-in gender neutral pronoun to 'he/she'. You also have at least 14 alternatives to choose from ( https://en.wikipedia.org/wiki/Third-person_pronoun#Summary ) and two more if you're at a Renaissance faire ( https://en.wikipedia.org/wiki/Third-person_pronoun#Historica... ). For the grammar sno…

I wouldn't say it's obligatory, especially since the poster already was aware of not assuming gender by using "he/she" (though I know some people identify as neither of those). I do prefer singular they; it's very natural and yes, it's been around in English for a long time.

Re: Learn how to design large-scale systems

#53

Post by [deleted] »

gnahckire wrote: post #8
geggam wrote: post #5

Earlier quoted context omitted.

from my understanding you get the ability to put the DAL into a "pause" mode where it queues all the api requests allowing you do to updates / upgrades to the database with no downtime. It also gives you a way of controlling what queries are used by the API servers preventing a developer from doing silly things and creating a production outage

It also makes changing your DB a lot easier since APIs using the DAL don't need to be updated since they're DB agnostic -- you "only" need to update the DAL API.

[deleted]

Re: Learn how to design large-scale systems

#54

Post by ahartmetz »

moduspol wrote: post #45
sillysaurus3 wrote: post #38

Earlier quoted context omitted.

They add new features constantly. I'd be surprised if there was a single week with no new features being developed. Most of the work is server side, e.g. voting ring detection. We only notice indirectly, when the quality of the site goes up.

To me the comparison just isn't there. The user experience here could be identical to how it was in the '90s. It's certainly something to marvel at to some extent, but a lot of us could get a pretty high level of elegance in our backend if our user experience had no reason to change for 10+ years.

The user experience is very simple and that's a good thing, too. I much prefer HN's UX over Reddit's, which is painfully slow on mobile and noticeably slower on desktop browsers.

Re: Learn how to design large-scale systems

#55

Post by sbov »

twistedpair wrote: post #24
pc86 wrote: post #11

Earlier quoted context omitted.

How often does one change the DB backing a live production application?

I've done it twice. If you're experiencing significant growth or change in access patterns, you may for example go from Postgres to a KV store. In one of the cases where I had to switch, we swapped from Cassandra to S3 for 100x OpEx savings since C* couldn't scale cost effectively to our needs, so we rolled a database on top of S3 instead that well out performed C* for our use case (e.g. need to export a 3B row CSV i…

> you may for example go from Postgres to a KV store.

If its easy to do this then you are using a tiny fraction of Postgres.

If you want it to be easy to switch your database then you need to code to the lowest common denominator. I would rather use my databases to their fullest potential, rather than purposefully handicap myself because I might have to change it in the future.

Re: Learn how to design large-scale systems

#56

Post by sillysaurus3 »

moduspol wrote: post #45
sillysaurus3 wrote: post #38

Earlier quoted context omitted.

They add new features constantly. I'd be surprised if there was a single week with no new features being developed. Most of the work is server side, e.g. voting ring detection. We only notice indirectly, when the quality of the site goes up.

To me the comparison just isn't there. The user experience here could be identical to how it was in the '90s. It's certainly something to marvel at to some extent, but a lot of us could get a pretty high level of elegance in our backend if our user experience had no reason to change for 10+ years.

To see how quickly HN changed in the old days, skim the feature requests thread: https://news.ycombinator.com/item?id=363

Some feature requests were a matter of days, like user profiles: https://news.ycombinator.com/item?id=481

A brief essay on some HN design decisions: https://pastebin.com/bSW5dfRQ (from https://news.ycombinator.com/item?id=8424502)

I think the arc codebase is worth studying and understanding, primarily so that you can extend its simplicity into your own projects. The reason HN was such a success is because it handles so many cases in the same way: Stories, comments, and polls are all the same thing: items. If you want to add a new thing, you just create a new item and add whatever fields you want.

These rapid prototyping techniques have downsides, but the cure is to keep in mind what you can't do. (For example, you can't rename item keys without breaking existing items, so be sure to choose good names.)

Like emacs, HN's design is borne out of simplicity and generality. It's what you get when you write the next most important feature as quickly as possible, then cut as much code as possible, every day. Both halves are equally important.

It's fine to say that our modern applications are so much more complicated that the old lessons don't apply. And in extreme cases, that may be true. I don't think SpaceX has the luxury of rapid prototyping their software.

But the typical app is CRUD. For those, data processing flexibility is perhaps the most important factor in whether you can write new features quickly. And since code is data, a lisp master can write systems with a shocking number of features in shockingly few lines of code. (See Jak'n'dexter: https://all-things-andy-gavin.com/2011/03/12/making-crash-ba...)

Re: Learn how to design large-scale systems

#57

Post by Twisol »

dan7678 wrote: post #47
byebyetech wrote: post #26

Earlier quoted context omitted.

I don't think Anki supposed to be used that way. Each card should be recallable under 10 seconds. So it should be only few lines of content. More content you put in one Anki card, it will take you more time and eventually you will stop looking at the card. A failure scenario.

Yeah, I used Anki a ton in college, and doing things like this was always futile and frustrating. Flashcards are fantastic for learning short bouts of things, but not large structures like many lines of code. Additionally I'd say even if you succeeded in memorizing it this way, it's not making you a better problem solver, which is what actually matters for that particular subject; you're just (temporarily) better at…

I agree in general, but it seems like there might be a particularly constrained situation where it makes sense. I can usually look at a medium-sized block of code and suss out its intent in a short amount of time, even if I don't know all of the details about its behavior or how it works. A deck of flashcards curated from examples like that might be useful for recognizing the higher-level patterns that drive that intuition.

I wouldn't really know for sure; I took the long way 'round (time and experience) for gaining that skill. But it at least seems plausible. Of course, the linked article isn't this.

Re: Learn how to design large-scale systems

#58

Post by matachuan »

mozumder wrote: post #37

What kind of numbers are they talking about for it to be "large-scale"? One well designed fast app server can serve 1000 requests per second per processor core, and you might have 50 processor cores in a 2U rack, for 50,000 requests per second. For database access, you now have fast NVMe disks that can push 2 million IOPS to serve those 50,000 accesses. 50,000 requests per second is good enough for a million concurre…

Why not have a scale-up system?

Re: Learn how to design large-scale systems

#59

Post by paulddraper »

sillysaurus3 wrote: post #9

Note that HN, a top-1000 site in the US, runs on a single box via a single racket process. "The key to performance is elegance, not battalions of special cases."

> runs on a single box via a single racket process

Incomplete. They are fronted by one of the largest CDNs in the world, on whom they rely for most traffic.

Well, I actually don't know the numbers, but I do know that for popular posts HN admins (a) try to break the conversation up over multiple posts and (b) plead with users to log out so as to allow Cloudflare to handle the load.

The biggest lesson HN teaches for designing large scale systems is "use a large scale system someone else has already designed".

Post reply on HN