Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Learn how to design large-scale systems

github.com

Post reply on HN
Print view View original

191–199 of 199 posts

Re: Learn how to design large-scale systems

#191

Post by nemo1618 »

bjterry wrote: post #186
nemo1618 wrote: post #73

Earlier quoted context omitted.

I've been experimenting with Anki recently. I loaded up a deck of popular fonts, with the goal being to memorize them to the point where I could recognize them in the real world. Each card contains the sentence, "The quick brown fox jumped over the lazy dog," and you have to identify what font it's written in. The first day was really tough; I missed cards over and over again. 20 new cards (the default) is probably t…

I would be careful with what you put in Anki. There is only so much stuff that you can memorize outside of stuff you'd learn from normal life, because the time you have to devote to flashcarding is kind of limited (except if it's something that excites you it creates more time). I think generally when you choose to make the investment to add cards to your Anki deck you should have a really concrete use case and I don…

Thanks for the detailed reply. I agree that this isn't the best use of Anki; I did it more as a way of testing how effective spaced repetition is. And in that respect, the experiment was a success, so now I feel confident using this system to memorize other things I care about. The ML insight was just a nice surprise. :)

Re: Learn how to design large-scale systems

#192

Post by mmt »

ljw1001 wrote: post #188
mmt wrote: post #178

Earlier quoted context omitted.

That's an important point, especially with the oft-repeated statistic of 2-years as the average tenure of an engineer. Of course, averages (even if true) are like stereotypes. It would be interesting to see the tenure data on the experts (consultants/implementers) of large-scale systems, other than at the iconic ones (e.g. Google, Netflix).

I think it likely that people with large-scale experience who aren't at Google would have lower tenures than average, simply because they're becoming more valuable and most companies don't pay people their replacement wage if they've been there very long.

The effect that you mention is already cited for the trend of lowering average tenure of technical professionals, in general, so, absent specific evidence that this subset's market value differential (market value less existing employers' willingness to keep up) is increasing faster than average, there's no reason to believe that's the reason for a shorter than average tenure.

We don't even know if the tenure is shorter than average.

Regardless, neither the primary motivation for a short tenure, nor even any average would be particularly meaningful with regard to what I believe to be ris's implied accusation:

Absent at least one tenure long enough to see through the consequences of the creation of the large-scale system, such a creator cannot be truly considered experienced with large-scale systems, no matter how many such creations are on the resume (even though the market values/hires the latter).

Re: Learn how to design large-scale systems

#193

Post by mmt »

ljw1001 wrote: post #187
ris wrote: post #130

I'm quite tired of everyone wanting to build "large scale systems" and play at being Netflix. The truth of the matter is the vast vast majority of people will never need to do this with their project and instead will just end up making an expensive to maintain mess with way too many moving parts. At least as important as designing something that can scale up is designing something that can scale down . You don't know…

It's odd because there seem to be two conflicting trends. On the one hand, you have people embracing (say) javascript as a server platform because it's easy to get something done, and simultaneously have people designing for outlandish scale. In general, the 'get it done' mentality is the one that makes economic sense, because once you've added together the pile of software that doesn't need to scale to the other pil…

> then a team of engineers builds a crazily complex pipeline to make the python perform in some reasonable time frame. (Hi, Spark!).

Having seen at least a couple of similar setups, I remain skeptical that this isn't, at its core, just a problem of ignorance of how "big" one can make/get a single server, before even paying a premium.

However, even for the "largest" commodity servers, last I looked, the premium at the highest end (over linear price:performance) was only something like 4x.

There was some relevant discussion of single server versus distributed in subthreads of https://news.ycombinator.com/item?id=17492234 a few days ago.

> In my workplace, one example allocates bits of a job to roughly 100 machines, moving data to each, in a cloud environment where the data movement overhead is constantly fighting the benefits of distribution.

I'm confident that cloud environments contribute to hardware ignorance, since cloud providers offer a very limited choice of options, and I have yet to see anything high end.

This is especially a frustration for me with networking options, where high bandwidth (beyond 10Gb/s on AWS, until recently, and still only 40GB/s max, AFAIK) is nonexistent and, otherwise, expensive, and low latency options like Infiniband don't seem to exist, either, even at the now low/obsolete bandwidths of 16 or 32Gb/s.

Re: Learn how to design large-scale systems

#194

Post by rhizome »

aaronblohowiak wrote: post #181
rhizome wrote: post #149

Earlier quoted context omitted.

The figure I saw around maybe 2015 was that AWS (maybe Heroku) is 10x more expensive and bare metal is 10x more efficient/faster, lending a 100x difference in price-performance. Both factors are probably larger nowadays.

Those figures are not in line with my understanding.

I'll try to find the article, I think it might have been Bitnami or Joyent who ran it.

Some older discussion on the topic here: https://news.ycombinator.com/item?id=3165095

Re: Learn how to design large-scale systems

#195

Post by bovermyer »

0xFACEFEED wrote: post #146
bovermyer wrote: post #144

You know what hasn't been done? A blog post about how to make a service that fulfills the needs of most people most of the time. All of the online and print material about such things focus on how to achieve massive scale correctly. Don't get me wrong; this is valuable and, generally, sound advice. However, it also ignores the majority of use cases for software. I would love to see a blog post here from someone who h…

This is because the people solving real world problems aren't writing books/tutorials/guides. Real world system design is dirty. Mostly this is due to constraints (time, cost, etc). And no one starts with zero architecture and 10 million users. Guides like this serve no purpose other than to fatten vocabularies and promote the "brand" of people who aren't actually doing the work (speakers, educators, etc).

I didn't say I was looking for a guide. I'm looking for a story.

Surfacing things like this elevates the entire practice, since it illuminates what that "dirty" work looks like.

Re: Learn how to design large-scale systems

#196

Post by bsenftner »

s_ngularity wrote: post #128
bsenftner wrote: post #82

Earlier quoted context omitted.

Yes. One can use a C++ library like Restbed and embed the web server directly into a compiled executable that uses SQLite as an embedded database. The "large-scale, multi-system architecture" in such common use today is completely unnecessary when faced with this setup. I have multiple Restbed integrated applications whose entire disk footprint is 7MB; they can run on a $99 Intel Compute Stick, perform industrial gra…

What about fault-tolerance though? That's definitely a single point of failure scenario.

Run as many instances as your fault tollerance requirements needs. The expense of adding another physical box is trivial when that physically box is $99 to $250 total to own. They "pay for themselves" in their first month of use, versus any cloud configuration running any 'amp or node or Mean or simply "traditional" web stack.

Re: Learn how to design large-scale systems

#197

Post by ljw1001 »

mmt wrote: post #192
ljw1001 wrote: post #188

Earlier quoted context omitted.

I think it likely that people with large-scale experience who aren't at Google would have lower tenures than average, simply because they're becoming more valuable and most companies don't pay people their replacement wage if they've been there very long.

The effect that you mention is already cited for the trend of lowering average tenure of technical professionals, in general, so, absent specific evidence that this subset's market value differential (market value less existing employers' willingness to keep up) is increasing faster than average, there's no reason to believe that's the reason for a shorter than average tenure. We don't even know if the tenure is shor…

Sorry dude, I didn't know this was a peer reviewed academic journal. I thought it was HN.

Re: Learn how to design large-scale systems

#198

Post by jshmrsn »

ljw1001 wrote: post #189
staticassertion wrote: post #176

Earlier quoted context omitted.

I've worked at smaller companies. Even still, it's not hard to hit relatively large amounts of data, depending on the field. As an example, any kind of analytics could generate terabytes of data a day... per customer. A side project I am building will have to handle billions of events per day. Per customer. There are 0 customers (this is for fun, not profit), but as soon as it would hit one customer I would need to c…

> As an example, any kind of analytics could generate terabytes of data a day... per customer. Wait, what? I've never worked anywhere where one customer generated terabytes of data per day, and I've worked on very large commercial enterprise software. The only thing I have experience with that produces anything close to that kind of data per customer is in genetic sequencing, and you only do a customer once. (Even th…

I was also very confused by OP as well. I’m thinking maybe by customer he means a business using his analytics service. That would explain why it needs to scale for his first customer, and why one customer could have terabytes of data.

Re: Learn how to design large-scale systems

#199

Post by tryonqc »

spraak wrote: post #52
0xbadcafebee wrote: post #21

Earlier quoted context omitted.

Obligatory pedantic HN grammar comment: on the outside chance that the gp's gender is not binary, the word 'they' is a good stand-in gender neutral pronoun to 'he/she'. You also have at least 14 alternatives to choose from ( https://en.wikipedia.org/wiki/Third-person_pronoun#Summary ) and two more if you're at a Renaissance faire ( https://en.wikipedia.org/wiki/Third-person_pronoun#Historica... ). For the grammar sno…

I wouldn't say it's obligatory, especially since the poster already was aware of not assuming gender by using "he/she" (though I know some people identify as neither of those). I do prefer singular they; it's very natural and yes, it's been around in English for a long time.

I thought I did a good thing :(

The use of their / they refering a single person doesn't come naturally to me as english is my 2nd language and we're taught its plural. (it can indeed be used as "third person plural singular" according to oxford dict.)

Since its the "least bad" (to my ears) of the gender-neutral pronouns on the wiki page I'll try to use the "they/their" instead.

Post reply on HN