Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Learn how to design large-scale systems

github.com

Post reply on HN
Print view View original

91–100 of 199 posts

Re: Learn how to design large-scale systems

#91

Post by digikata »

lossolo wrote: post #70
sillysaurus3 wrote: post #9

Note that HN, a top-1000 site in the US, runs on a single box via a single racket process. "The key to performance is elegance, not battalions of special cases."

I am running top-1000 site in one of EU countries on one 4 core machine with 20-30% load. Around 1000 http/https reqs/s. Most of those requests do couple of postgres reqs (read and write) and couple of redis reqs. Elasticsearch - for searching/recommendations Redis - hot data (certain data is only kept in redis) Postgres - for the rest of data Clickhouse - analytics Most of the system is written in Go. Whole system w…

Is postgres on the same machine or on a separate one?

Re: Learn how to design large-scale systems

#92

Post by pc86 »

twistedpair wrote: post #24
pc86 wrote: post #11

Earlier quoted context omitted.

How often does one change the DB backing a live production application?

I've done it twice. If you're experiencing significant growth or change in access patterns, you may for example go from Postgres to a KV store. In one of the cases where I had to switch, we swapped from Cassandra to S3 for 100x OpEx savings since C* couldn't scale cost effectively to our needs, so we rolled a database on top of S3 instead that well out performed C* for our use case (e.g. need to export a 3B row CSV i…

I'm sure there are rare exceptions but I would imagine if you dug deeply into the business rules around "I need to export a 3,000,000,000 row CSV file" and into what the users are actually trying to accomplish at the end of that workflow, you could find a solution that meets those goals better while also obviating the need to export a 3,000,000,000 row CSV file.

Re: Learn how to design large-scale systems

#93

Post by pc86 »

blattimwind wrote: post #25
Lunatic666 wrote: post #19

Earlier quoted context omitted.

I also don't think it's a good idea. If you don't use the database specific functions out of fear you aren't able to switch anymore, you are probably wasting a lot of potential performance.

I think "changing the DB" likely referred to schema changes, not swapping out the DBMS.

> > > > It also makes changing your DB a lot easier since APIs using the DAL don't need to be updated since they're DB agnostic [emphasis mine]

I took it as changing the DBMS under the hood.

Re: Learn how to design large-scale systems

#94

Post by crystalPalace »

bandwitch wrote: post #31

If you liked this page, you might also like the excellent book "Designing Data-Intensive Applications" that among others surveys many characteristics of large-scale systems and presents some. Note that it's not a book for preparing you on system design questions, but it can definitely help.

I've been reading this and it's great so far. Are there other similar books that describe modern enterprise architecture at scale?

Re: Learn how to design large-scale systems

#95

Post by yumaikas »

sillysaurus3 wrote: post #38
moduspol wrote: post #28

Earlier quoted context omitted.

HN has the luxury of being able to make few high level changes over years, though. It might be tougher to maintain that single box elegance and performance if they were adding new features every month or two (which is much more applicable to the rest of us).

They add new features constantly. I'd be surprised if there was a single week with no new features being developed. Most of the work is server side, e.g. voting ring detection. We only notice indirectly, when the quality of the site goes up.

So there's a few features that I'd like to see at some point: Some way of marking "new comments" when I revisit a thread. Rescanning HN threads just to see what's changed since I last looked at one is kinda frustrating. (I realize you're not @dang, but posting this here just as an idea)

Re: Learn how to design large-scale systems

#96

Post by hinkley »

nlawalker wrote: post #10
sillysaurus3 wrote: post #9

Note that HN, a top-1000 site in the US, runs on a single box via a single racket process. "The key to performance is elegance, not battalions of special cases."

Heh, elegance like "There is a story on the front page getting lots of attention, please log out so we can serve you from cache."

Elegance like "The entire site is a giant nested table in 2018"

Re: Learn how to design large-scale systems

#97

Post by hrshtr »

bandwitch wrote: post #31

If you liked this page, you might also like the excellent book "Designing Data-Intensive Applications" that among others surveys many characteristics of large-scale systems and presents some. Note that it's not a book for preparing you on system design questions, but it can definitely help.

+1 to the book. It has been a great resource for me to understand lot of concepts on Distributed Systems

Re: Learn how to design large-scale systems

#98

Post by lossolo »

digikata wrote: post #91
lossolo wrote: post #70

Earlier quoted context omitted.

I am running top-1000 site in one of EU countries on one 4 core machine with 20-30% load. Around 1000 http/https reqs/s. Most of those requests do couple of postgres reqs (read and write) and couple of redis reqs. Elasticsearch - for searching/recommendations Redis - hot data (certain data is only kept in redis) Postgres - for the rest of data Clickhouse - analytics Most of the system is written in Go. Whole system w…

Is postgres on the same machine or on a separate one?

Same machine, everything is on the same machine (dedicated server aka bare metal, not vps/cloud).

Re: Learn how to design large-scale systems

#99

Post by dmoy »

copperx wrote: post #88
nlawalker wrote: post #17

Earlier quoted context omitted.

I know, just a good natured poke :) Plus you could probably take that comment at face value - making use of web caching is definitely an important tool when building a large scale system.

Why is caching out of the window when logged in?

votes, things to hide, etc all change the page and prevent serving everything wholesale out of cache

I'm not actually sure, just guessing...

Re: Learn how to design large-scale systems

#100

Post by ChristianGeek »

nemo1618 wrote: post #73
madethemcry wrote: post #13

Oh interesting, I have never seen Anki ( https://apps.ankiweb.net/ ) being used for large blocks of source code. Anki is an open source application (desktop + mobile) for spaced repetition learning (aka flashcards). It's a very popular tool among people who want to learn languages (and basically anything else you want to remember). There are many shared decks ( https://ankiweb.net/shared/decks/ ). Creating and format…

I've been experimenting with Anki recently. I loaded up a deck of popular fonts, with the goal being to memorize them to the point where I could recognize them in the real world. Each card contains the sentence, "The quick brown fox jumped over the lazy dog," and you have to identify what font it's written in. The first day was really tough; I missed cards over and over again. 20 new cards (the default) is probably t…

Have you tried WhatTheFont?

https://itunes.apple.com/us/app/whatthefont/id304304134?mt=8

Post reply on HN