Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why is everything so scalable?

stavros.io

Post reply on HN
Print view View original

101–110 of 383 posts

Re: Why is everything so scalable?

#101

Post by nicoburns »

potatolicious wrote: post #88
nicoburns wrote: post #18

Earlier quoted context omitted.

I guess the work is deterministic, but it often (unintentionally) makes the systems being developed non-deterministic!

Ah yes. I once worked at a startup that insisted on Mongo despite not having anywhere near the data volume for it to make any sense at all. Like, we're talking 5 orders of magnitude off of what one would reasonably expect to need a Mongo deployment. I was but a baby engineer then, and the leads would not countenance anything as pedestrian as MySQL/Postgres. Anyway, fast forward a bit and we were tasked with building…

I've seen similar with Firebase. Luckily I took over as tech lead at this company, so I was able to migrate us to Postgres. Amusingly, as well as being more reliable, the Postgres version (on a single small database instance) was also much faster than the previous Firebase-based version (due to it enabling JOINs in the database rather than in application code).

Re: Why is everything so scalable?

#102

Post by ahartmetz »

huflungdung wrote: post #94
sgarland wrote: post #47

Earlier quoted context omitted.

THANK YOU. People look at me like I’m insane when I tell them that their overly-complicated pipeline could be easily handled by a couple of beefy servers. Or at best, they’ll argue that “this way, they don’t have to manage infrastructure.” Except you do - you absolutely do. It’s just been partially abstracted away, and some parts like OS maintenance are handled (not that that was ever the difficult part of managing s…

What I say is that we massively underestimate just how fast computers are these days

Indeed - they are incredibly fast, it's just buried under layers upon layers of stuff

Re: Why is everything so scalable?

#103

Post by sarchertech »

ahoka wrote: post #90
robertlagrant wrote: post #37

Earlier quoted context omitted.

I think this sounds more like Domain Driven Design than Clean Code.

It kinda started with Clean Code. I remember some old colleagues walking around with the book in their hand and deleting ten year old comments in every commit they made: "You see, we don't need that anymore, because the code describes itself". It made a generation (generations?) of software developers think that all the architectural patterns were found now, we can finally do real engineering and just have to find th…

It was (is) bad. The worst part is they the majority of people pushing it haven’t even read Clean Code. They’ve read a blog post by a guy who read a blog post by a guy who skimmed the book.

Re: Why is everything so scalable?

#104

Post by charlimangy »

Working in modern architectures that can scale is pretty important for developers that want to have attractive resumes. Given that your startup has a 9 out of 10 chance of failing you're going to need another job. If you want people to stay you have to give them the security of keeping up with at least some of the latest fashions.

Re: Why is everything so scalable?

#105

Post by gamerDude »

When I'm working with new developers I always have to convince them to simplify their setup. Why are we on autoscaled, pay by the query infra when we are serving a few people. Then they complain how expensive it is. I had someone tell me that their costs were $1500/mon when they were in demo stages. I asked them why they aren't hosting on a single small server for $20. And they responded that it didn't matter because they were using free credits.

Except that those free credits will go away and you'll find yourself not wanting to do all the work to move it over when it would've been easier to do so when you just had that first monolith server up.

I think free credits and hyped up technology is to blame. So, basically a gamed onboarding process that gets people to over-engineer and spend more.

Re: Why is everything so scalable?

#106

Post by ahartmetz »

yobbo wrote: post #10

Many startup business models have no chance of becoming profitable unless they reach a certain scale, but they might have less than 1% probability of reaching that scale. Making it scalable is easy work since it is deterministic, but growing customers is not. Another perspective is that the defacto purpose of startups (and projects at random companies) may actually be work experience and rehearsal for the day the fou…

>“dress for the job you want, not the job you have”

I don't think I should dress down any further :>

Re: Why is everything so scalable?

#107

Post by hrimfaxi »

sgarland wrote: post #47
BirAdam wrote: post #29

Just to be honest for a bit here... we also should be asking what kind of scale? Quite a while ago, before containers were a thing at all, I did systems for some very large porn companies. They were doing streaming video at scale before most, and the only other people working on video at that scale were Youtube. The general setup for the largest players in that space was haproxy in front of nginx in front of several…

THANK YOU. People look at me like I’m insane when I tell them that their overly-complicated pipeline could be easily handled by a couple of beefy servers. Or at best, they’ll argue that “this way, they don’t have to manage infrastructure.” Except you do - you absolutely do. It’s just been partially abstracted away, and some parts like OS maintenance are handled (not that that was ever the difficult part of managing s…

Depending on your regulatory environment, it can be cost-effective to not have to maintain your own data center with 24/7 security response, environmental monitoring, fire suppression systems, etc. (of course, the majority of businesses are probably not interested in things like SOC 2)

Re: Why is everything so scalable?

#108

Post by dig1 »

BirAdam wrote: post #29

Just to be honest for a bit here... we also should be asking what kind of scale? Quite a while ago, before containers were a thing at all, I did systems for some very large porn companies. They were doing streaming video at scale before most, and the only other people working on video at that scale were Youtube. The general setup for the largest players in that space was haproxy in front of nginx in front of several…

> The general setup for the largest players in that space was haproxy in front of nginx in front of several PHP servers in front of a MySQL database that had one primary r/w with one read only replica.

You'd be surprised that the most stable setups today are run this way. The problem is that this way it's hard to attract investors; they'll assume you are running on old or outdated tech. Everything should be serverless, agentic and, at least on paper, hyperscalable, because that sells further.

> Today at AWS, it is easily possible for people to spend a multiple of the cost of that hardware setup every month for far less compute power and storage.

That is actually the goal of hyperscalers: they are charging you premium for way inferior results. Also, the article stated a very cold truth: "every engineer wants a fashionable CV that will help her get the next job" and you won't definitely get a job if you said: "I moved everything from AWS and put it behind haproxy on one bare-metal box for $100/mo infra bill".

Re: Why is everything so scalable?

#109

Post by stavros »

CaptainOfCoit wrote: post #3

I've seen startups killed because of one or two "influential" programmers deciding they need to start architecturing the project for 1000TPS and 10K daily users, as "that's the proper way to build scalable software", while the project itself hasn't even found product-market fit yet and barely has users. Inevitably, the project needs to make a drastic change which now is so painful to do because it no longer fits the…

> 1000TPS and 10K daily users

I absolutely agree with your point, but I want to point out, like other commenters here, that the numbers should be much larger. We think that, because 10k daily users is a big deal for a product, they're also a big deal for a small server, but they really aren't.

It's fantastic that our servers nowadays can easily handle multiple tens of thousands of daily users on $100/mo.

Re: Why is everything so scalable?

#110

Post by hedora »

ben_w wrote: post #82
phkahler wrote: post #75

Earlier quoted context omitted.

I thought you were going to say to reduced O(n^2) to O(n*log(n)), but you just deleted the operation. Normally I'd say that's great, but just how much duplicate data is being left around now? Is that OK?

Each element was about, oh I can't remember exactly, perhaps 50 bytes? It wasn't a constant value, there could in theory be a string in there, but those needed to be added manually and when you have 20,000 of them, nobody would. Also, it was overwhelmingly likely that none of the elements were duplicates in the first place, and the few exceptions were probably exactly one duplicate.

I'm kind of surprised no one just searched for "deduplication algorithm". If it was absolutely necessary to get this 1MB dataset to be smaller (when was this? Did it need to fit in L2 on a pentium 3 something?), then it could probably have been deduped + loaded in 300-400ms.

Most engineers that I've worked with that die on a premature optimization molehill like you describe also make that molehill as complicated as possible. Replacing the inside of the nested loop with a hashtable probe certainly fits the stereotype.

Post reply on HN