Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Does it scale? Who cares (2011)

jacquesmattheij.com

Post reply on HN
Print view View original

81–90 of 285 posts

Re: Does it scale? Who cares (2011)

#81

Post by timewarrior »

john61 wrote: post #75
technosamay20 wrote: post #15

Couldn't agree with this article more. I built the biggest social network to come out of India from 2006-2009. It was like Twitter but over text messaging. At it's peak it had 50M+ users and sent 1B+ text messages in a day. When I started, the app was on a single machine. I didn't know a lot about databases and scaling. Didn't even know what database indexes are and what are their benefits. Just built the basic produ…

> Time is the only currency you have which can't be earned and only spent. I would argue that you can earn lifetime with a healthy life style. I remember to have read a study that measured the average difference between healthy and unhealthy life styles with 14 years. If you spend 1 day a week more for a healthy lifestyle (exercise, self cooking, enough sleep, little stress, enough recreation time) for 40 years you h…

You make a great point. Health should be the topmost priority and I need to get better at it.

However to my original point - you can lose fitness (within reasonable limits) and then gain it back and not notice a difference later on. However time lost is lost forever.

Re: Does it scale? Who cares (2011)

#82

Post by timewarrior »

an_d_rew wrote: post #80
pbreit wrote: post #69

Earlier quoted context omitted.

You're refusing to learn from someone who lived it. I also know of zero services which failed because they couldn't scale their technology. But I know of 100s that failed because they couldn't get enough users or usage. Your example about the DB tables is exactly the trap to avoid while you are iterating rapidly to try and find product/market fit.

Absence of evidence is not evidence of absence. I've been part of many projects that couldn't scale because lack of forethought meant there was a nasty implicit O(n^2) thing going on when it could have easily been designed to be O(n log n). Almost all of these projects were killed with CYA hand waving about "just not fast enough".

Can you please share some examples. I would love to learn from your experience.

Re: Does it scale? Who cares (2011)

#83

Post by cyberferret »

pbreit wrote: post #69
cyberferret wrote: post #52

Earlier quoted context omitted.

> I know of absolutely no service which failed because it couldn't scale. I would say this is because of the simple fact of visibility and adoption. You've probably never heard of these services probably because they ground to a halt with a mere 1000 users, so they never got mainstream enough to be recognised as a viable service. It is a bit like how no one remembers the dozens of people who failed to achieve sustain…

You're refusing to learn from someone who lived it. I also know of zero services which failed because they couldn't scale their technology. But I know of 100s that failed because they couldn't get enough users or usage. Your example about the DB tables is exactly the trap to avoid while you are iterating rapidly to try and find product/market fit.

You are assuming that I haven't "lived it". Over the past 2 or 3 years, I have built around 8 web apps. Some which got virtually no traction at all, and some which have reached a happy medium of users and income. None which have reached mega scale or millions of users (yet).

Some of the (real world) feedback I got from the web apps that failed to get off the ground were due mainly to our customers complaining that:

* pages were taking too long to reload.

* not enough fields on a particular data table to store information in

* missing API

* pages that would refresh in entirety instead of just updating the changed portions

Most customers didn't stick around to wait for us to address or fix the issues - there were plenty of other competitive products that fitted them better that they could start using that same day.

A couple of those sites also got negative feedback on Reddit or HN because I hosted the front end website on a $5 VPS server and when the site suffered the inevitable 'hug of death' from posting to these sites, they immediately went down due to inability to scale under the sudden deluge of visitors, and I received uncomplimentary feedback on that (where people actually elected to post feedback rather than just close the browser tab and completely forget and move on from my app).

Yes, over thinking scaling and design is bad, but putting it aside as a 'totally not important now' factor is also just as bad, if not worse, in my 'lived it' experience.

Re: Does it scale? Who cares (2011)

#84

Post by sbuttgereit »

The overall premise of the blog is exactly correct; though I would say some areas you probably need to consider more than others.

Spending a lot of time figuring out what exact microservice/sharding/etc strategy you need to serve a zillion visits a day and building it before you've even got customer/visitor one is overkill out of the gate. But that shouldn't mean you shouldn't think about how you'll scale over the short term or medium term at all.

When I approach scaling, I'll tend to spend much more time on the data retention strategy than anything else: databases (or other stores), being stateful, means that it's a harder problem to deal with later than earlier as compared to the stateless parts of the system. Even so, I'm typically not developing the data services for the Unicorn I wish the client will become, I'm just putting a lot more thought into optimizing the data services I am building so it won't hit the breaking point as early as it might if I were designing for functionality alone. I do expect there to be a breaking point and a need to change direction at some point in these early stage designs. But in that short to medium term period, the simpler designs are regularly easier to maintain than the fully "scalable" approaches that might be tried otherwise, and rarely do those companies ever need anything more.

Re: Does it scale? Who cares (2011)

#85

Post by timewarrior »

cyberferret wrote: post #83
pbreit wrote: post #69

Earlier quoted context omitted.

You're refusing to learn from someone who lived it. I also know of zero services which failed because they couldn't scale their technology. But I know of 100s that failed because they couldn't get enough users or usage. Your example about the DB tables is exactly the trap to avoid while you are iterating rapidly to try and find product/market fit.

You are assuming that I haven't "lived it". Over the past 2 or 3 years, I have built around 8 web apps. Some which got virtually no traction at all, and some which have reached a happy medium of users and income. None which have reached mega scale or millions of users (yet). Some of the (real world) feedback I got from the web apps that failed to get off the ground were due mainly to our customers complaining that: *…

It is great that you got a few which got traction out of 8 tries. This is an incredible success rate.

Most people see much less success. The original suggestion is geared towards first few tries that people make. Once they have made a few attempts, they learn from it and naturally build more scalable products without spending extra effort. You have already read my story about my first try which had a very poor start. My subsequent efforts have scaled in that order without needing a single rewrite.

However, if they spend too long procrastinating, worrying about and investing in scale - they are wasting precious time which would be better spent finding product market fit.

Re: Does it scale? Who cares (2011)

#86

Post by apersona »

technosamay20 wrote: post #15

Couldn't agree with this article more. I built the biggest social network to come out of India from 2006-2009. It was like Twitter but over text messaging. At it's peak it had 50M+ users and sent 1B+ text messages in a day. When I started, the app was on a single machine. I didn't know a lot about databases and scaling. Didn't even know what database indexes are and what are their benefits. Just built the basic produ…

> I know of absolutely no service which failed because it couldn't scale.

Bullshit. What about the site (voat something?) that tried to be an alternative to reddit when there was some scandal there but couldn't keep users because they kept crashing?

Re: Does it scale? Who cares (2011)

#87

Post by narrator »

technosamay20 wrote: post #15

Couldn't agree with this article more. I built the biggest social network to come out of India from 2006-2009. It was like Twitter but over text messaging. At it's peak it had 50M+ users and sent 1B+ text messages in a day. When I started, the app was on a single machine. I didn't know a lot about databases and scaling. Didn't even know what database indexes are and what are their benefits. Just built the basic produ…

>I know of absolutely no service which failed because it couldn't scale.

These days there would be hardly any way to fail to scale given a large budget. The technology to do so is all so good. However, back in the day it wasn't always so. I'd say Friendster was an example of a company that failed due to scaling issues.

Re: Does it scale? Who cares (2011)

#88

Post by grey-area »

cyberferret wrote: post #52
technosamay20 wrote: post #15

Couldn't agree with this article more. I built the biggest social network to come out of India from 2006-2009. It was like Twitter but over text messaging. At it's peak it had 50M+ users and sent 1B+ text messages in a day. When I started, the app was on a single machine. I didn't know a lot about databases and scaling. Didn't even know what database indexes are and what are their benefits. Just built the basic produ…

> I know of absolutely no service which failed because it couldn't scale. I would say this is because of the simple fact of visibility and adoption. You've probably never heard of these services probably because they ground to a halt with a mere 1000 users, so they never got mainstream enough to be recognised as a viable service. It is a bit like how no one remembers the dozens of people who failed to achieve sustain…

I am willing to bet that 9 out of 10 'weekend projects' have had to be thrown out completely and redeveloped from scratch when the number of users became significant.

9 out of 10 weekend projects never get to a significant number of users.

Re: Does it scale? Who cares (2011)

#89

Post by graphememes »

narrator wrote: post #87
technosamay20 wrote: post #15

Couldn't agree with this article more. I built the biggest social network to come out of India from 2006-2009. It was like Twitter but over text messaging. At it's peak it had 50M+ users and sent 1B+ text messages in a day. When I started, the app was on a single machine. I didn't know a lot about databases and scaling. Didn't even know what database indexes are and what are their benefits. Just built the basic produ…

>I know of absolutely no service which failed because it couldn't scale. These days there would be hardly any way to fail to scale given a large budget. The technology to do so is all so good. However, back in the day it wasn't always so. I'd say Friendster was an example of a company that failed due to scaling issues.

Depends on what you're doing, easy to fail when you do analytics or something like datadog.

Re: Does it scale? Who cares (2011)

#90

Post by graphememes »

virmundi wrote: post #14
devduderino wrote: post #11

I care because it usually goes like this: Product manager > "Niche sass app {x} will never need to support more than 10-20 users" Two weeks after launch > "We have 10k users and counting, why didn't you architect this for scale?" Always assume you underestimated the scope of the project.

I've literally been waffling back and forth between UUID and BigInt for the last week. The reason for this is I need to handle eventually distributed systems. Do I need UUIDs? Maybe not, but after much debt, I've decided that the storage requirements of such an index in memory is worth the ability to move from Postgres to Postgresql-xl.

luid, check it out
Post reply on HN