Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Learn how to design large-scale systems

github.com

Post reply on HN
Print view View original

171–180 of 199 posts

Re: Learn how to design large-scale systems

#171

Post by antpls »

cjhanks wrote: post #127
rargulati wrote: post #120

Earlier quoted context omitted.

Outside of raw experience, what can you do, read, or learn to build the intuition for formulating and answering the above questions? I can answer the above for systems I've built, but I've spent quite a bit of time with those systems. How do I get better at doing this during the planning phases, or even better, for a system I'm unfamiliar with (ie. are there tools you lean on here)?

A bit of caution, I haven't worked in distributed systems for some time now. And I am sure there many people more competent than I. But in general the cliche of "Great artists steal" applies here. If AWS/GCE/Azure (or any other major software vendor) is offering a service or a feature, then it is almost certainly solving a problem somebody has. If you don't understand what problem is being solved, then you cannot pos…

Hi,

I agree with you about reverse engineering the giants, it is one way of acquiring knowledge.

However I disagree on :

> If AWS/GCE/Azure (or any other major software vendor) is offering a service or a feature, then it is almost certainly solving a problem somebody has.

AWS/GCE/Azure have industrialized the process of proposing new building blocks. The cost for them to propose and maintain a new service is lower than a few years ago. They are logically able to experiment more with users, and eventually shut down services with no actual needs (or overlapping with another service they propose). Especially true for Google.

I have the intuition it also works as a marketing process : more you spend your time reading their documentation, more you accept their brand, more you are statistically going to buy something from them.

Re: Learn how to design large-scale systems

#173

Post by ris »

FLUX-YOU wrote: post #151
ris wrote: post #130

I'm quite tired of everyone wanting to build "large scale systems" and play at being Netflix. The truth of the matter is the vast vast majority of people will never need to do this with their project and instead will just end up making an expensive to maintain mess with way too many moving parts. At least as important as designing something that can scale up is designing something that can scale down . You don't know…

>I'm quite tired of everyone wanting to build "large scale systems" and play at being Netflix. The truth of the matter is the vast vast majority of people will never need to do this with their project and instead will just end up making an expensive to maintain mess with way too many moving parts. Few companies will take a product that actually needs large scale systems and hire someone that has no prior experience.…

> Few companies will take a product that actually needs large scale systems and hire someone that has no prior experience.

No I think most people end up hiring those who have experience creating big complicated systems but haven't stuck around long enough for their chickens to come home to roost.

Re: Learn how to design large-scale systems

#174

Post by econochoice »

0xFACEFEED wrote: post #146
bovermyer wrote: post #144

You know what hasn't been done? A blog post about how to make a service that fulfills the needs of most people most of the time. All of the online and print material about such things focus on how to achieve massive scale correctly. Don't get me wrong; this is valuable and, generally, sound advice. However, it also ignores the majority of use cases for software. I would love to see a blog post here from someone who h…

This is because the people solving real world problems aren't writing books/tutorials/guides. Real world system design is dirty. Mostly this is due to constraints (time, cost, etc). And no one starts with zero architecture and 10 million users. Guides like this serve no purpose other than to fatten vocabularies and promote the "brand" of people who aren't actually doing the work (speakers, educators, etc).

> Guides like this serve no purpose other than to fatten vocabularies and promote the "brand" of people who aren't actually doing the work (speakers, educators, etc).

Yep. They're often ghostwritten, too.

Re: Learn how to design large-scale systems

#175

Post by id_rsa »

gradys wrote: post #77
madethemcry wrote: post #13

Oh interesting, I have never seen Anki ( https://apps.ankiweb.net/ ) being used for large blocks of source code. Anki is an open source application (desktop + mobile) for spaced repetition learning (aka flashcards). It's a very popular tool among people who want to learn languages (and basically anything else you want to remember). There are many shared decks ( https://ankiweb.net/shared/decks/ ). Creating and format…

Michael Nielsen just this month published an essay on using spaced repetition (and especially Anki) to augment long-term memory. It seems to be part of his work with YC Research. I highly recommend it if this kind of thing interests you. It gives a solid overview of the theory and practical lessons from his daily use of Anki over the last few years: http://augmentingcognition.com/ltm.html

Thanks for sharing. I agree with his advice about keeping the flashcards short. It is easy to get into a flow while answering the cards, but when one of them is too long, it disrupts the flow. Try breaking down larger cards into smaller bits of information.

Re: Learn how to design large-scale systems

#176

Post by staticassertion »

phil21 wrote: post #162
mrep wrote: post #161

Earlier quoted context omitted.

I think you underestimate how many people work at large companies that have to deal with these problems. Not everyone is in the HN startup bubble working for a company with barely any customers.

I think you highly overestimate it. Most large companies are a series of small groups that act as companies that have nearly trivial (to the point of absurdity) engineering concerns. Sure, a few groups in each F500 need epic skills - but I think that's an exceedingly vanishing amount of the (actual) work that is being done. The term Enterprise and what it stands for earned it's laughable reputation for a reason.

I've worked at smaller companies. Even still, it's not hard to hit relatively large amounts of data, depending on the field.

As an example, any kind of analytics could generate terabytes of data a day... per customer.

A side project I am building will have to handle billions of events per day. Per customer. There are 0 customers (this is for fun, not profit), but as soon as it would hit one customer I would need to consider an approach that scales.

How many companies have similar requirements?

But that's actually besides the point.

Microservice architecture, or any architecture that focuses on isolated, asynchronous components, adds complexity. Of course.

But it also reduces work in other areas. If you build async, isolated services, you no longer have to deal with catastrophic service failure. Cascading failures go away at the async bound.

For many of us, I imagine we've spent a lot of time fighting fires at organizations where one service going down was a serious problem, causing other services to fail, and setting your infrastructure ablaze. Hence a bias towards solving that problem upfront.

Re: Learn how to design large-scale systems

#177

Post by bobdole12345 »

ris wrote: post #130

I'm quite tired of everyone wanting to build "large scale systems" and play at being Netflix. The truth of the matter is the vast vast majority of people will never need to do this with their project and instead will just end up making an expensive to maintain mess with way too many moving parts. At least as important as designing something that can scale up is designing something that can scale down . You don't know…

I'm not sure I agree. The most important thing about designing large scale systems is dividing the total work flow into self contained pieces with easily inspected separation points.

In the grand scheme of things this doesn't have to mean microservices across a million hosts, only that you've decomposed the problem into it's elemental parts. Those parts can now be considered separately as their own elements rather than having to contend with the entire architecture in your head when a problem arises.

Re: Learn how to design large-scale systems

#178

Post by mmt »

ris wrote: post #173
FLUX-YOU wrote: post #151

Earlier quoted context omitted.

>I'm quite tired of everyone wanting to build "large scale systems" and play at being Netflix. The truth of the matter is the vast vast majority of people will never need to do this with their project and instead will just end up making an expensive to maintain mess with way too many moving parts. Few companies will take a product that actually needs large scale systems and hire someone that has no prior experience.…

> Few companies will take a product that actually needs large scale systems and hire someone that has no prior experience. No I think most people end up hiring those who have experience creating big complicated systems but haven't stuck around long enough for their chickens to come home to roost.

That's an important point, especially with the oft-repeated statistic of 2-years as the average tenure of an engineer.

Of course, averages (even if true) are like stereotypes.

It would be interesting to see the tenure data on the experts (consultants/implementers) of large-scale systems, other than at the iconic ones (e.g. Google, Netflix).

Re: Learn how to design large-scale systems

#179

Post by nickpsecurity »

avinium wrote: post #156
Jtsummers wrote: post #115

Earlier quoted context omitted.

I didn't get any real interest in my office, but I applied TLA+ to a debugging challenge. I described a well-specified system (abbreviated, for instance data was just "DATA", not the actual myriad number of message types that could be sent). The intention was to understand a bug and possibly identify which of three suspected locations it was occurring in. I had enough of the specification for each part implemented: T…

Any suggested resources/books for getting a better understanding of TLA+?

hwayne, who did learntla.com, is getting close to finishing a book on it. You could start with his online tutorial but get the book soon as it's out. Here's his Twitter in case you want to watch for the book release:

https://mobile.twitter.com/Hillelogram

Re: Learn how to design large-scale systems

#180

Post by nickpsecurity »

Jtsummers wrote: post #115
mxschumacher wrote: post #18

Earlier quoted context omitted.

Could you please talk about your experiences with TLA+? The "tools of thinking" for designing and verifying systems really interest me.

I didn't get any real interest in my office, but I applied TLA+ to a debugging challenge. I described a well-specified system (abbreviated, for instance data was just "DATA", not the actual myriad number of message types that could be sent). The intention was to understand a bug and possibly identify which of three suspected locations it was occurring in. I had enough of the specification for each part implemented: T…

That's awesome! I'm adding this write up to my collection of case studies. You should consider doing a blog post with a mockup problem really similar to illustrate that process with code and specs.
Post reply on HN