Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Learn how to design large-scale systems

github.com

Post reply on HN
Print view View original

81–90 of 199 posts

Re: Learn how to design large-scale systems

#81

Post by tomnipotent »

scarface74 wrote: post #50
SolarNet wrote: post #32

Earlier quoted context omitted.

There are easier database-user level solutions to these problems than increasing ops-architecture with yet another layer of indirection. Personally this is why I hate dev-ops culture, no one knows how to use databases properly anymore. > It also gives you a way of controlling what queries are used by the API servers preventing a developer from doing silly things and creating a production outage Also called database r…

This is just a bad idea. Better idea: unless you are rewriting your entire schema from scratch, you should be able to use database views, database triggers, extra/duplicated columns and tables as you make schema swaps. Even with tools like Liquibase, the more functionality you put in the database (views, stored procedures, triggers, etc.) the harder it is to do deployments and rollbacks and keep the code and the data…

Like all things, it depends. Certain features/abstractions have more value at different stages of an application.

I LOVE using views early on in a new application's schema as it allows me to evolve the logical model separately from the physical model, and once I've coalesced on something I like it's easy enough to swap the view with a real table and my application code higher in the stack is none the wiser.

Even Facebook at one point relied on MySQL triggers to keep its memcache fleet synced.

Re: Learn how to design large-scale systems

#82

Post by bsenftner »

mabbo wrote: post #41

This design, roughly, is being used very widely and is well-documented everywhere. But does anyone know of any lesser-known yet equally functional designs that work at the same scale? Are there cases this design does not work for?

Yes. One can use a C++ library like Restbed and embed the web server directly into a compiled executable that uses SQLite as an embedded database. The "large-scale, multi-system architecture" in such common use today is completely unnecessary when faced with this setup. I have multiple Restbed integrated applications whose entire disk footprint is 7MB; they can run on a $99 Intel Compute Stick, perform industrial grade facial recognition with multiple HD video streams, and still overwhelm traditional web stacks with events and data when pertinent events the software needs to report start emitting over the wire.

The "only catches" are the developer(s) need experience working in multi-threaded C++, and they need to understand the traditional web stack they are eliminating.

Re: Learn how to design large-scale systems

#83

Post by marcosdumay »

moduspol wrote: post #45
sillysaurus3 wrote: post #38

Earlier quoted context omitted.

They add new features constantly. I'd be surprised if there was a single week with no new features being developed. Most of the work is server side, e.g. voting ring detection. We only notice indirectly, when the quality of the site goes up.

To me the comparison just isn't there. The user experience here could be identical to how it was in the '90s. It's certainly something to marvel at to some extent, but a lot of us could get a pretty high level of elegance in our backend if our user experience had no reason to change for 10+ years.

Most sites have very little need to change once they get a correct target population fit.

That they keep changing (many times against express complaints of their users) is not reason for excusing them a bad experience.

Re: Learn how to design large-scale systems

#84

Post by lalwanivikas »

yread wrote: post #75

I hoped this would help me with this problem I have - I'm coding a web app with a smallish database (<1GB for the next few years, <1% writes). I need low latencies for accessing it. And I would like to have multiple servers over the world sharing the database.

you need to provide more details to get any useful advice. but just based on what you have described, any db would do the job. add a caching layer and you have your low latencies.

again, what is the traffic and bandwidth load like? peak and average values? what kind of data are you planning to store? small values but huge volumes or the opposite? a lot will change based on your system requirements.

Re: Learn how to design large-scale systems

#85

Post by lossolo »

tmaly wrote: post #80
lossolo wrote: post #70

Earlier quoted context omitted.

I am running top-1000 site in one of EU countries on one 4 core machine with 20-30% load. Around 1000 http/https reqs/s. Most of those requests do couple of postgres reqs (read and write) and couple of redis reqs. Elasticsearch - for searching/recommendations Redis - hot data (certain data is only kept in redis) Postgres - for the rest of data Clickhouse - analytics Most of the system is written in Go. Whole system w…

are you using any specific Go web framework or just the standard library?

I am using fasthttp[1]

1. https://github.com/valyala/fasthttp

Re: Learn how to design large-scale systems

#86

Post by marcosdumay »

matachuan wrote: post #58
mozumder wrote: post #37

What kind of numbers are they talking about for it to be "large-scale"? One well designed fast app server can serve 1000 requests per second per processor core, and you might have 50 processor cores in a 2U rack, for 50,000 requests per second. For database access, you now have fast NVMe disks that can push 2 million IOPS to serve those 50,000 accesses. 50,000 requests per second is good enough for a million concurre…

Why not have a scale-up system?

Because it costs money and slows development and ops down. Is there a good reason for getting it when you are not one of the ~200 companies in the world with enough scale to use it?

Re: Learn how to design large-scale systems

#87

Post by scarface74 »

tomnipotent wrote: post #81
scarface74 wrote: post #50

Earlier quoted context omitted.

This is just a bad idea. Better idea: unless you are rewriting your entire schema from scratch, you should be able to use database views, database triggers, extra/duplicated columns and tables as you make schema swaps. Even with tools like Liquibase, the more functionality you put in the database (views, stored procedures, triggers, etc.) the harder it is to do deployments and rollbacks and keep the code and the data…

Like all things, it depends. Certain features/abstractions have more value at different stages of an application. I LOVE using views early on in a new application's schema as it allows me to evolve the logical model separately from the physical model, and once I've coalesced on something I like it's easy enough to swap the view with a real table and my application code higher in the stack is none the wiser. Even Face…

Views aren’t quite as bad as stored procedures. I could see doing a view being decently easy to manage - you can always add s column without breaking backwards compatibility. But you can’t imaging the number of times I’ve seen things like...

SaveCustomer_1 SaveCustomer_2 SaveCustomer_3

And you never know whose using what.

Re: Learn how to design large-scale systems

#88

Post by copperx »

nlawalker wrote: post #17
tambourine_man wrote: post #15

Earlier quoted context omitted.

Admittedly, that’s very rare.

I know, just a good natured poke :) Plus you could probably take that comment at face value - making use of web caching is definitely an important tool when building a large scale system.

Why is caching out of the window when logged in?

Re: Learn how to design large-scale systems

#89

Post by madethemcry »

gradys wrote: post #77
madethemcry wrote: post #13

Oh interesting, I have never seen Anki ( https://apps.ankiweb.net/ ) being used for large blocks of source code. Anki is an open source application (desktop + mobile) for spaced repetition learning (aka flashcards). It's a very popular tool among people who want to learn languages (and basically anything else you want to remember). There are many shared decks ( https://ankiweb.net/shared/decks/ ). Creating and format…

Michael Nielsen just this month published an essay on using spaced repetition (and especially Anki) to augment long-term memory. It seems to be part of his work with YC Research. I highly recommend it if this kind of thing interests you. It gives a solid overview of the theory and practical lessons from his daily use of Anki over the last few years: http://augmentingcognition.com/ltm.html

I stoped after the fourth paragraph just to tell you that this article hit the nail. Really interesting article so far.

Thank you so much!

Re: Learn how to design large-scale systems

#90

Post by zjaffee »

bandwitch wrote: post #31

If you liked this page, you might also like the excellent book "Designing Data-Intensive Applications" that among others surveys many characteristics of large-scale systems and presents some. Note that it's not a book for preparing you on system design questions, but it can definitely help.

Just wanted to give a +1 to Designing data intensive applications, it's really one of the best resources out there in terms of touching really most of the areas necessary for building out big data applications, where you can then know which areas you'd prefer to dive further into.
Post reply on HN