This design, roughly, is being used very widely and is well-documented everywhere. But does anyone know of any lesser-known yet equally functional designs that work at the same scale?
Are there cases this design does not work for?
Learn how to design large-scale systems
41–50 of 199 posts
Re: Learn how to design large-scale systems
#42Earlier quoted context omitted.
from my understanding you get the ability to put the DAL into a "pause" mode where it queues all the api requests allowing you do to updates / upgrades to the database with no downtime. It also gives you a way of controlling what queries are used by the API servers preventing a developer from doing silly things and creating a production outage
There are easier database-user level solutions to these problems than increasing ops-architecture with yet another layer of indirection. Personally this is why I hate dev-ops culture, no one knows how to use databases properly anymore. > It also gives you a way of controlling what queries are used by the API servers preventing a developer from doing silly things and creating a production outage Also called database r…
I think it's due to the trend against hiring DBAs. Nobody wants to put so much work into their databases anymore. I feel like that was half of the NoSQL craze, let's stop thinking about how to use databases because it's a pain.
After a while of slowly modifying and lumping more crap on a database, and it becomes a slow PITA that everyone is too afraid to touch, the usual result is to lift everything onto a new DB. DALs make this easier, but I agree that this should in no way be the point of a DAL. The point should just be to simplify & improve access to the database.
Re: Learn how to design large-scale systems
#43Earlier quoted context omitted.
HN has the luxury of being able to make few high level changes over years, though. It might be tougher to maintain that single box elegance and performance if they were adding new features every month or two (which is much more applicable to the rest of us).
I don't understand this comment. Why should adding features make an app crumble on a single server? I think the point is that good software is able to serve a lot of users on a single server. A great example imho is Blender. Features are added constantly but because the software is modular it doesn't have any impact on the overall performance. Today the problem is that adding features means: adding the latest and gre…
> Why should adding features make an app crumble on a single server?
It's not inherent, but obviously as you have more developers working on more and more things independently, each with different needs, tolerances, and deadlines, it becomes increasingly unreasonable to presume it can all be managed well on a single box.
If they went and added chat, or Twitter-like features, or subreddits, or similar, it might be a lot tougher to keep it all on a single box. It's a lot easier when we're all looking at the same top 30 stories, and pretty limited in how we interact with them and each other.
Re: Learn how to design large-scale systems
#44Note that HN, a top-1000 site in the US, runs on a single box via a single racket process. "The key to performance is elegance, not battalions of special cases."
A single box?! Do we know its specs?
As of 2015: https://news.ycombinator.com/item?id=9222006
FreeBSD 10.1
Nginx 1.7.10
Racket 6.1.1 with some HN and FreeBSD specific patches.
2x 3GHz Intel Xeon-IvyBridge (E5-2690-V2-DecaCore)
8x16GB Kingston 16GB DDR3 2Rx4
SuperMicro X9DRI-LN4F+_R1.2A
Adaptec 71605
9x 1000GB Western Digital WD RE4
2x 200GB Smart XceedIOPS SSDRe: Learn how to design large-scale systems
#45Earlier quoted context omitted.
HN has the luxury of being able to make few high level changes over years, though. It might be tougher to maintain that single box elegance and performance if they were adding new features every month or two (which is much more applicable to the rest of us).
They add new features constantly. I'd be surprised if there was a single week with no new features being developed. Most of the work is server side, e.g. voting ring detection. We only notice indirectly, when the quality of the site goes up.
To me the comparison just isn't there. The user experience here could be identical to how it was in the '90s. It's certainly something to marvel at to some extent, but a lot of us could get a pretty high level of elegance in our backend if our user experience had no reason to change for 10+ years.
Re: Learn how to design large-scale systems
#46Earlier quoted context omitted.
There are easier database-user level solutions to these problems than increasing ops-architecture with yet another layer of indirection. Personally this is why I hate dev-ops culture, no one knows how to use databases properly anymore. > It also gives you a way of controlling what queries are used by the API servers preventing a developer from doing silly things and creating a production outage Also called database r…
I think it's due to the trend against hiring DBAs. Nobody wants to put so much work into their databases anymore. I feel like that was half of the NoSQL craze, let's stop thinking about how to use databases because it's a pain. After a while of slowly modifying and lumping more crap on a database, and it becomes a slow PITA that everyone is too afraid to touch, the usual result is to lift everything onto a new DB. DA…
Our DBAs loved the DAL. There is a trend today to let developers run all over production doing what they want and it has direct impact on systems.
The complexity of the modern stack is ridiculous. You run java containers inside docker containers inside virtual machines and call it optimized.
Re: Learn how to design large-scale systems
#47Oh interesting, I have never seen Anki ( https://apps.ankiweb.net/ ) being used for large blocks of source code. Anki is an open source application (desktop + mobile) for spaced repetition learning (aka flashcards). It's a very popular tool among people who want to learn languages (and basically anything else you want to remember). There are many shared decks ( https://ankiweb.net/shared/decks/ ). Creating and format…
I don't think Anki supposed to be used that way. Each card should be recallable under 10 seconds. So it should be only few lines of content. More content you put in one Anki card, it will take you more time and eventually you will stop looking at the card. A failure scenario.
Yeah, I used Anki a ton in college, and doing things like this was always futile and frustrating. Flashcards are fantastic for learning short bouts of things, but not large structures like many lines of code.
Additionally I'd say even if you succeeded in memorizing it this way, it's not making you a better problem solver, which is what actually matters for that particular subject; you're just (temporarily) better at regurgitating some lines of code.
Re: Learn how to design large-scale systems
#48[deleted]
Re: Learn how to design large-scale systems
#49Note that HN, a top-1000 site in the US, runs on a single box via a single racket process. "The key to performance is elegance, not battalions of special cases."
HN is relatively easy to optimise though - there are only a few stories with high traffic, so if you have good caching the load on the back end can be very low. It's more difficult to do that with something like github where the users are spread across millions of repos.
Re: Learn how to design large-scale systems
#50Earlier quoted context omitted.
from my understanding you get the ability to put the DAL into a "pause" mode where it queues all the api requests allowing you do to updates / upgrades to the database with no downtime. It also gives you a way of controlling what queries are used by the API servers preventing a developer from doing silly things and creating a production outage
There are easier database-user level solutions to these problems than increasing ops-architecture with yet another layer of indirection. Personally this is why I hate dev-ops culture, no one knows how to use databases properly anymore. > It also gives you a way of controlling what queries are used by the API servers preventing a developer from doing silly things and creating a production outage Also called database r…
This is just a bad idea. Better idea: unless you are rewriting your entire schema from scratch, you should be able to use database views, database triggers, extra/duplicated columns and tables as you make schema swaps.
Even with tools like Liquibase, the more functionality you put in the database (views, stored procedures, triggers, etc.) the harder it is to do deployments and rollbacks and keep the code and the database functionality in sync.