Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Learn how to design large-scale systems

github.com

Post reply on HN
Print view View original

131–140 of 199 posts

Re: Learn how to design large-scale systems

#131

Post by ris »

pvarangot wrote: post #118
cjhanks wrote: post #103

I see something comparable to these diagrams (it feels like) a half-dozen times a year. The architecture is in general 'fine'. But communication paths of subsystems is probably the easiest part of the problem. And in general, re-organizing the architecture of a system is usually possible - if and only if - the underlying data model is sane. The more important questions are; - What is the convention for addressing ass…

Also this architectures assume there's no need to do the dreaded "network locking", which for some problems regarding dispatch and avoiding triggering expensive/non idempotent batch jobs on background needs to be done. If you want to rely on SQL to do all the locking for you this usually doesn't scale.

> If you want to rely on SQL to do all the locking for you this usually doesn't scale.

I suspect you have a mis-adjusted notion of "usually". "Usually", as in, for the majority of systems designed and in-use in the world, a well tuned, reliable RDBMS will be able to do this absolutely fine. The scale of systems that the world needs vs the quantity of them is an extremely long tailed curve.

Re: Learn how to design large-scale systems

#132

Post by swaggyBoatswain »

madethemcry wrote: post #13

Oh interesting, I have never seen Anki ( https://apps.ankiweb.net/ ) being used for large blocks of source code. Anki is an open source application (desktop + mobile) for spaced repetition learning (aka flashcards). It's a very popular tool among people who want to learn languages (and basically anything else you want to remember). There are many shared decks ( https://ankiweb.net/shared/decks/ ). Creating and format…

The problem I have with using anki for 2+ years is I get too lazy to make cards. It feels like tedious data entry work, I'm wandering if I'm the only one that feels this way? I have to run anki, open the add item panel, figure out how to format the card / phrase the question, pick a deck, pick a card type, move the mouse over click tag, enter tag, write it down, press enter, minimize the application. I would to do this several times throughout the day since I learn things inconsistently. My productivity tanks when I use anki. Because I also have many other tools for jotting notes, emails, slack, blog, etc. I already tried practically all the anki plugins out there too

This is mostly for computer science / programming. If I used anki for learning a language, I would just mass create all my cards at one time or use a curated deck.

Re: Learn how to design large-scale systems

#133

Post by [deleted] »

ris wrote: post #130

I'm quite tired of everyone wanting to build "large scale systems" and play at being Netflix. The truth of the matter is the vast vast majority of people will never need to do this with their project and instead will just end up making an expensive to maintain mess with way too many moving parts. At least as important as designing something that can scale up is designing something that can scale down . You don't know…

[deleted]

Re: Learn how to design large-scale systems

#134

Post by [deleted] »

ris wrote: post #131
pvarangot wrote: post #118

Earlier quoted context omitted.

Also this architectures assume there's no need to do the dreaded "network locking", which for some problems regarding dispatch and avoiding triggering expensive/non idempotent batch jobs on background needs to be done. If you want to rely on SQL to do all the locking for you this usually doesn't scale.

> If you want to rely on SQL to do all the locking for you this usually doesn't scale. I suspect you have a mis-adjusted notion of "usually". "Usually", as in, for the majority of systems designed and in-use in the world, a well tuned, reliable RDBMS will be able to do this absolutely fine. The scale of systems that the world needs vs the quantity of them is an extremely long tailed curve.

[deleted]

Re: Learn how to design large-scale systems

#135

Post by ivanhoe »

swaggyBoatswain wrote: post #132
madethemcry wrote: post #13

Oh interesting, I have never seen Anki ( https://apps.ankiweb.net/ ) being used for large blocks of source code. Anki is an open source application (desktop + mobile) for spaced repetition learning (aka flashcards). It's a very popular tool among people who want to learn languages (and basically anything else you want to remember). There are many shared decks ( https://ankiweb.net/shared/decks/ ). Creating and format…

The problem I have with using anki for 2+ years is I get too lazy to make cards. It feels like tedious data entry work, I'm wandering if I'm the only one that feels this way? I have to run anki, open the add item panel, figure out how to format the card / phrase the question, pick a deck, pick a card type, move the mouse over click tag, enter tag, write it down, press enter, minimize the application. I would to do th…

In my experience creating cards (or writing down the words into a notebook) is an essential part of the process. I'm not sure about the exact mechanics of it, but for me writing a word implants it in my memory much deeper than just reading it.

Re: Learn how to design large-scale systems

#136

Post by digikata »

lossolo wrote: post #98
digikata wrote: post #91

Earlier quoted context omitted.

Is postgres on the same machine or on a separate one?

Same machine, everything is on the same machine (dedicated server aka bare metal , not vps/cloud).

Thanks for the reply! May I ask how far you go with respect to reliability/availability? Do you have a live replicated server on stand by or just replicate a log stream elsewhere, something else?

Re: Learn how to design large-scale systems

#137

Post by swaggyBoatswain »

ivanhoe wrote: post #135
swaggyBoatswain wrote: post #132

Earlier quoted context omitted.

The problem I have with using anki for 2+ years is I get too lazy to make cards. It feels like tedious data entry work, I'm wandering if I'm the only one that feels this way? I have to run anki, open the add item panel, figure out how to format the card / phrase the question, pick a deck, pick a card type, move the mouse over click tag, enter tag, write it down, press enter, minimize the application. I would to do th…

In my experience creating cards (or writing down the words into a notebook) is an essential part of the process. I'm not sure about the exact mechanics of it, but for me writing a word implants it in my memory much deeper than just reading it.

I totally get that too, I write/draw things down and find it helps remembering things easier. Especially math. But I find the time it takes to transfer that thought onto anki is so painstakingly slow. Anki doesn't even have a dedicated shortcut key for minimizing / maximizing the add card panel either, at least for windows.

don't get me wrong here I love using anki and ankidroid but adding cards is a PITA. I add cards only on desktop anki because neither ankidroid nor ankiwebapp support easy-image formatting. But I do add cards from ankidroid if its a picture of some handwritten / whiteboard drawings I've made.

Re: Learn how to design large-scale systems

#138

Post by amorphid »

ris wrote: post #130

I'm quite tired of everyone wanting to build "large scale systems" and play at being Netflix. The truth of the matter is the vast vast majority of people will never need to do this with their project and instead will just end up making an expensive to maintain mess with way too many moving parts. At least as important as designing something that can scale up is designing something that can scale down . You don't know…

I never thought about scaling down as a skill until just now. I kind of assumed "scaling up" implied up && down, our maybe "scaling out" implied out && in. Interesting thought.

Re: Learn how to design large-scale systems

#140

Post by Nican »

yread wrote: post #116
lalwanivikas wrote: post #84

Earlier quoted context omitted.

you need to provide more details to get any useful advice. but just based on what you have described, any db would do the job. add a caching layer and you have your low latencies. again, what is the traffic and bandwidth load like? peak and average values? what kind of data are you planning to store? small values but huge volumes or the opposite? a lot will change based on your system requirements.

To clarify: let's say I have servers in two locations A and B that are 200ms from each other. When I issue a write to the db in A I don't want to wait (multiples) of the 200ms before it returns. I don't really care whether the write appears to a reader at B in 5s or 50 minutes but of course the writes have to be at least causally consistent. I won't have millions (realistically not even thousands) of users and the da…

Look at MySQL asynchronous replication.
Post reply on HN