Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Does it scale? Who cares (2011)

jacquesmattheij.com

Post reply on HN
Print view View original

251–260 of 285 posts

Re: Does it scale? Who cares (2011)

#251

Post by user5994461 »

luord wrote: post #218
user5994461 wrote: post #200

Earlier quoted context omitted.

> You can probably push your solution by an order of one magnitude if you need it quickly. You can always push a python/ruby app an order of magnitude by putting an order of magnitude more AWS instances. It will almost always bankrupt you in the medium term. The only place I've seen it sustainable is a place that was generating a $100 per user, and there weren't many active users either (thousands, not millions).

> You can always push a python/ruby app an order of magnitude by putting an order of magnitude more AWS instances. Or a Java app or a Go app. Really, if one's working in a domain where the language would become the bottleneck, one deliberately screwed up by going against the grain because that language is little used in that domain. For almost everything else or with very specific exceptions, something else is the bo…

Java or Go are easily 10 times faster than Python/Rail on the same hardware. (Potentially much more, especially when you have non trivial logic to process in the code).

That's a recurrent issue with API code (that usually has to be somewhat fast). Not so much for frontend generation.

P.S. Sorry but hundreds of customer is nothing. That's served by a pair of boxes (DB + webserver) irrelevant of the language.

Re: Does it scale? Who cares (2011)

#252

Post by manveru »

stephenr wrote: post #120
yummyfajitas wrote: post #115

Earlier quoted context omitted.

I can name one: Darcs. It was my favorite certain control system, apart from the lack of scalability.

Darcs is a distributed version control system. It is inherently not a hosted solution that "needs to scale" the way the article is talking about.

I'd definitely say that Darcs failed to catch on because it didn't scale with the size of the repo, the larger your history was the more likely you were to run into performance issues, and there was little you could do about it because the underlying theory is flawed.

The scaling problem with Darcs wasn't about hosting, but in the implementation of their theory of patches and how it handled conflicts. See http://darcs.net/FAQ/ConflictsDarcs1#problems-with-conflicts for an overview of the issues in Darcs 1 and 2.

This seems to be also one of the driving forces behind the development of Pijul, which is also patch-based instead of snapshot-based, which makes it easier to understand and use, but all implementations so far had major performance issues once repositories grow. For more on that, see https://pijul.org/faq.html

I was one of the early users of Darcs 1, back before Git existed. I wanted to use version control, but the alternatives were pretty hard to understand and use. While Darcs was really nice to use (, and fast on small repos, after a few years I had to convert everything to git because exponential times on most operations was just not sustainable, and fixing that required constant vigilance and altering history, not very friendly for new contributors.

Even GHC moved from Darcs to git in 2011 because of this: https://mail.haskell.org/pipermail/glasgow-haskell-users/201...

Re: Does it scale? Who cares (2011)

#253

Post by frik »

nickpsecurity wrote: post #246
frik wrote: post #166

Earlier quoted context omitted.

1) Friendster had scaling issues. They insisted on keeping the x-degree-of-friendship-calculation which is quite resource expensive and doesn't scale. Instead making the software scalable, they brought in expensive exec who decided to through more and more very high end servers and enterprise storage on it. It eat their startup money, and they were constantly firefighting with very high page load time like 12+ sec, i…

"MySpace took over Friendster because of this." Friendster was mostly focused on Asia plus lost a ton of users around the same time Facebook gained a ton of users. I don't think it was scaling architecture that did them in. It was the market choosing the competition for the user experience plus what their friends were on. That's for most of the world. I have no idea what contributed to their failure in Asia since I d…

Asia? You came a few years to late... Friendster was US focused, at least until it lost. Then the probably pivoted, but I wrote about the first phase of Friendster (the one everyone in US remembers who was active online in early 2000s).

Re: Does it scale? Who cares (2011)

#254

Post by user5994461 »

franciscop wrote: post #211
user5994461 wrote: post #190

Earlier quoted context omitted.

An alternative view on history: Facebook succeeded because the competitors failed to scale. Do you remember MySpace? The big social network with hundreds of million of users that came before facebook. They had massive scaling and performance issues. At the peak when everyone was moving to social media (almost a decade ago) the site could take an entire minute to load (if loading at all). They lost a lot of users, who…

I was a big MySpace user and late FB user, and I can tell you that it had nothing to do with [tech] scale. Back then I didn't know about programming (my first lines of code were some CSS in MySpace!) and the real reason is that users were becoming more hostile in a site that was a niche on its own. You had to start filtering out and removing SPAM all the time from your public wall. The fact that people could personal…

MySpace and Friendset have been largely commented in this discussion, in other comments.

I agree, they were different from facebook enough, that it's not all about scaling. Yet, don't underestimate the impact of having your site unreachable, when the competitor is coming strong. Not a good position to be in.

Re: Does it scale? Who cares (2011)

#255

Post by timewarrior »

xiaoma wrote: post #250
technosamay20 wrote: post #15

Couldn't agree with this article more. I built the biggest social network to come out of India from 2006-2009. It was like Twitter but over text messaging. At it's peak it had 50M+ users and sent 1B+ text messages in a day. When I started, the app was on a single machine. I didn't know a lot about databases and scaling. Didn't even know what database indexes are and what are their benefits. Just built the basic produ…

> "I know of absolutely no service which failed because it couldn't scale." Zooomr did. They came out of nowhere in 2006 and were a real threat to Flickr. They had AJAX-powered editss, geotagging, various other unique features and they were starting to pull in some highly followed followed photographers. But, the site kept crashing as traffic grew and some scaling problems even lead to data loss. I wanted to see them…

Main question here is that would they have done better if they had started with scale in mind. When you start you don't know how you product will look like when it has millions of users. You will likely do major product changes quite a few times.

If someone cannot scale your product which has adoption, they likely don't have insights, problem definition, capabilities, skills and resources to do that when they launch.

If I could edit my original post, I would put a rider related to capable team for service not dying.

Re: Does it scale? Who cares (2011)

#256

Post by user5994461 »

nickpsecurity wrote: post #243
user5994461 wrote: post #212

Earlier quoted context omitted.

You can't try 5 products or features a week on real users. They don't keep up.

Facebook deploys thousands a week: https://swreflections.blogspot.com/2013/09/this-is-how-faceb...

Facebook have thousands of products and billions of users. You do not ;)

Re: Does it scale? Who cares (2011)

#257

Post by Nican »

skinp wrote: post #30
devduderino wrote: post #11

I care because it usually goes like this: Product manager > "Niche sass app {x} will never need to support more than 10-20 users" Two weeks after launch > "We have 10k users and counting, why didn't you architect this for scale?" Always assume you underestimated the scope of the project.

Interesting. It might be a by-product of having worked in larger companies, but my experience has pretty much always been: Product manager > "We need this to work for 10k r/s at launch". Actual launch throughput (and all time peak): 100 r/s Now I always mostly just assume (capacity asked / 10) is more than enough!

As an employee of a large company, I can relate to this experience. The manager has grandiose views about the product, but has no data to back his claims.

Re: Does it scale? Who cares (2011)

#258

Post by hamburglar »

didibus wrote: post #235

The reason scale isn't so important today is because most DBs can actually scale vertically to really high numbers. The tipping point is high enough that if you have this problem, you probably can also afford to fix it. What matters though is performance and availability. No matter what scale you work at, you can't be slow, that will drive people away. You also can't be unavailable. This means that you might have to…

> Where scale matters is at big companies. When you already have a lot of customers, you’re first version of any new feature or product must already be at scale. Amazon couldn't have launched a non scalable prime now, or echo. Google can't launch a non scalable chat service, etc.

This is the point I was going to make. When you are designing the very platforms on which all these "does it scale? who cares" startups are going to be built, you do not have the luxury of having that attitude. Definitely don't take that mindset into an interview at google or amazon. :)

I do agree with the OP's sentiment for startup projects in general. I have had the experience of worrying about scale too early and over-engineering a system which never had more than a few dozen users, and the opposite experience of tossing something together that wouldn't scale and then going through the hairy scrambling at every order of magnitude of scale through tens of millions of users. The latter was definitely a better strategy.

Re: Does it scale? Who cares (2011)

#259

Post by timewarrior »

jaimex2 wrote: post #182
technosamay20 wrote: post #15

Couldn't agree with this article more. I built the biggest social network to come out of India from 2006-2009. It was like Twitter but over text messaging. At it's peak it had 50M+ users and sent 1B+ text messages in a day. When I started, the app was on a single machine. I didn't know a lot about databases and scaling. Didn't even know what database indexes are and what are their benefits. Just built the basic produ…

You started with Java... a language designed to handle credit card transactions world wide. Your story would be very different if you started with say Rails.

If you build stateless horizontally scalable service (and you should) - you are almost always going to be blocked on DB. On the app server front you just keep adding more machines behind load balancer.

As I look back, if we had used RoR or Python we might have moved faster with no negative impact on scale. We would have spent more money on extra servers though!

Re: Does it scale? Who cares (2011)

#260

Post by bedros »

kiriakasis wrote: post #158
bedros wrote: post #109

Earlier quoted context omitted.

myspace.com failed because it could not scale I personally switched to facebook for two reasons: got sick of repeated errors every time I browse myspace facebook has better album permissions, (myspace has none)

as another comment said these look like bad design not bad scaling.

Many problems including not filtering user input and comments for CSS, JS, etc.

But main problem, they crumbled under heavy traffic

Post reply on HN