Live data from Hacker News

Does it scale? Who cares (2011)

jacquesmattheij.com

51–60 of 285 posts

Re: Does it scale? Who cares (2011)

#51

Couldn't agree with this article more. I built the biggest social network to come out of India from 2006-2009. It was like Twitter but over text messaging. At it's peak it had 50M+ users and sent 1B+ text messages in a day. When I started, the app was on a single machine. I didn't know a lot about databases and scaling. Didn't even know what database indexes are and what are their benefits. Just built the basic produ…

This is a delightful retelling; thanks for sharing. > I know of absolutely no service which failed because it couldn't scale. Genuine question: what about Friendster?

I remember reading that Friendster founder was complaining that there were facing a lot of technical issues and investors were not letting him invest effort in fixing them.

IMHO, if your investors need to be involved in seeking approval for handling scaling pains and on top of that if they reject it - there are deeper issues within the company (management issues, politics, technical incompetence, leadership issues).

So to clarify my point - any team could still screw up technical execution. However there is no reason that given a good team and support from within the company, a product can't be scaled.

Re: Does it scale? Who cares (2011)

#52

Couldn't agree with this article more. I built the biggest social network to come out of India from 2006-2009. It was like Twitter but over text messaging. At it's peak it had 50M+ users and sent 1B+ text messages in a day. When I started, the app was on a single machine. I didn't know a lot about databases and scaling. Didn't even know what database indexes are and what are their benefits. Just built the basic produ…

> I know of absolutely no service which failed because it couldn't scale.

I would say this is because of the simple fact of visibility and adoption. You've probably never heard of these services probably because they ground to a halt with a mere 1000 users, so they never got mainstream enough to be recognised as a viable service. It is a bit like how no one remembers the dozens of people who failed to achieve sustained powered flight before the Wright brothers. Doesn't mean there weren't any, and scalability, technical or planning issues killed those efforts before anyone knew of it.

Some 'scalability' issues are inherent to your initial design, and not just the choice or configuration of your hardware/software platforms.

For instance, what if you were building a contact database of some sort. At first, you may have things like 'Phone Number' and 'Email Address' as part of the 'Person' database. Then, as your service gets popular, you notice people asking for extra contact like Twitter handles, LinkedIn pages etc. So you start adding those to your Person table as extra columns.

Eventually, you realise that you should have though more about this at the outset and have contact details stored in another data table altogether, linked to a 'Contact Type' table and related back to the Person table. This would have been mitigated at the start via better database design and catering for eventualities that you might never have foreseen. Migrating the original database to normalise it is a massive effort in its own right, and probably will take more time, cause more outage time, and cause more bugs in existing code than designing for that eventuality in the first place.

Even if 99% of your users only ever enter Phone and Email contact details, the second option, designed for scalability, will still handle that without a sweat, and 'scaling' to meet additional demands later is merely a matter of adding new contact types in the 'Contact Type' data table so that they become an extra option for all your users.

I am willing to bet that 9 out of 10 'weekend projects' have had to be thrown out completely and redeveloped from scratch when the number of users became significant. Of those rebuilds, I would be interested to see some research into how many users abandoned the said platform when (a) the original one started to grind to a halt or constantly fell over with errors and (b) the new platform came out with new features or a different UX that broke the 'look and feel' of the original.

Re: Does it scale? Who cares (2011)

#53

Taking a completely blasé approach to efficiency is potentially as dangerous as becoming hyper-focused on it. Not all businesses become roaring successes, and those who achieve moderate success often don't get the resources to fix deep-seated performance or architectural issues (either via engineering and/or throwing hardware at it.) Eventually these technical woes can completely halt momentum and I've seen it even d…

Agreed that completely blasé is pretty bad. However in this case being blasé would mean, not working hard to scale when you users are getting a bad experience. A basic stack these days node.js+mongo, go+*sql can easily handle more than 100k users even with one of the worst implementations. Most products don't reach that point!

> A basic stack these days node.js+mongo, go+*sql can easily handle more than 100k users...

That is making some very large assumptions about application workload. For an application which is purely a CRUD interface to a database, yes.

Re: Does it scale? Who cares (2011)

#54

Couldn't agree with this article more. I built the biggest social network to come out of India from 2006-2009. It was like Twitter but over text messaging. At it's peak it had 50M+ users and sent 1B+ text messages in a day. When I started, the app was on a single machine. I didn't know a lot about databases and scaling. Didn't even know what database indexes are and what are their benefits. Just built the basic produ…

[deleted]

Re: Does it scale? Who cares (2011)

#55
post #16

I care because it usually goes like this: Product manager > "Niche sass app {x} will never need to support more than 10-20 users" Two weeks after launch > "We have 10k users and counting, why didn't you architect this for scale?" Always assume you underestimated the scope of the project.

Ok what can you write it in that you can't scale to 10K users by just adding web servers? The hard to scale part is storage layer but even crappy options like RDS will happily support 10K users.

> but even crappy options like RDS

Curious as to why you see RDS as a 'crappy' option??

Given the choice of building and maintaining a MySQL box in the corner of your bedroom and hosting it on RDS - I know which way I would go (and I have done both over the years).

If you are talking 'scaling' in terms of the OP's article, then RDS is almost a no brainer, and you can scale your instances (and add replicated instances etc.), hide it behind a VPN, set up firewalls to prevent DDoS attacks all in a matter of minutes, with a few mouse clicks.

Re: Does it scale? Who cares (2011)

#56

Couldn't agree with this article more. I built the biggest social network to come out of India from 2006-2009. It was like Twitter but over text messaging. At it's peak it had 50M+ users and sent 1B+ text messages in a day. When I started, the app was on a single machine. I didn't know a lot about databases and scaling. Didn't even know what database indexes are and what are their benefits. Just built the basic produ…

Obligatory "Mongo DB Is Web Scale" https://www.youtube.com/watch?v=b2F-DItXtZs

Re: Does it scale? Who cares (2011)

#57
post #16

Earlier quoted context omitted.

Ok what can you write it in that you can't scale to 10K users by just adding web servers? The hard to scale part is storage layer but even crappy options like RDS will happily support 10K users.

> but even crappy options like RDS Curious as to why you see RDS as a 'crappy' option?? Given the choice of building and maintaining a MySQL box in the corner of your bedroom and hosting it on RDS - I know which way I would go (and I have done both over the years). If you are talking 'scaling' in terms of the OP's article, then RDS is almost a no brainer, and you can scale your instances (and add replicated instances…

The highest IOPS you can get on RDS is about a single consumer grade SSD.

Re: Does it scale? Who cares (2011)

#58

Couldn't agree with this article more. I built the biggest social network to come out of India from 2006-2009. It was like Twitter but over text messaging. At it's peak it had 50M+ users and sent 1B+ text messages in a day. When I started, the app was on a single machine. I didn't know a lot about databases and scaling. Didn't even know what database indexes are and what are their benefits. Just built the basic produ…

Can you share a link to the site?

SMSGupShup, now known as GupShup.

http://techcircle.vccircle.com/2013/02/06/mobile-group-messa...

Re: Does it scale? Who cares (2011)

#59

Couldn't agree with this article more. I built the biggest social network to come out of India from 2006-2009. It was like Twitter but over text messaging. At it's peak it had 50M+ users and sent 1B+ text messages in a day. When I started, the app was on a single machine. I didn't know a lot about databases and scaling. Didn't even know what database indexes are and what are their benefits. Just built the basic produ…

> I know of absolutely no service which failed because it couldn't scale. I would say this is because of the simple fact of visibility and adoption. You've probably never heard of these services probably because they ground to a halt with a mere 1000 users, so they never got mainstream enough to be recognised as a viable service. It is a bit like how no one remembers the dozens of people who failed to achieve sustain…

I mentioned something similar in a separate comment.

My initial DB schema was pretty bad. We did at 2 schema rewrites and migrations from the launch to 5M users. Each time it took 2 weeks of sleep less nights.

The machines today are really powerful. You can do a lot with 244 GB RAM machines backed by SSD.

Someone who doesn't have the skill set to be able to scale once they get traction - it's likely they will not have the skills to design for scale at start.

My recommendation to everyone would be pick a language and db you are most comfortable with and get started as soon as you can. You will fail on the product side a lot more times before you will fail on the technical side.

And if you are failing on technical side, reach out to me. I will definitely be able to help you find a way out. I am not sure if there is any product guy in the world who an make a similar claim on the product front. However there are at least dozens of technical guys in the world who can make a claim like I did.

So focus on launching the product as soon as possible. Work hard, reach out for help if needed. You will eventually get success.

Re: Does it scale? Who cares (2011)

#60
post #38

Earlier quoted context omitted.

Can you go into a bit more detail? I've seen amplification referenced before. My estimation is that the database will be pretty much read heavy. I will need to split a given database across multiple instances.

https://blog.2ndquadrant.com/on-the-impact-of-full-page-writ... this is fairly detailed analysis of this issue

Thank you. I'm reading over it now and for the next few days (to really grok it). I do wonder how bad it will be for low write situations. At the same time I'm open to switching to BigInt and using Postgresql-xl.
Post reply on HN