Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Does it scale? Who cares (2011)

jacquesmattheij.com

Post reply on HN
Print view View original

271–280 of 285 posts

Re: Does it scale? Who cares (2011)

#271

Post by nickpsecurity »

frik wrote: post #253
nickpsecurity wrote: post #246

Earlier quoted context omitted.

"MySpace took over Friendster because of this." Friendster was mostly focused on Asia plus lost a ton of users around the same time Facebook gained a ton of users. I don't think it was scaling architecture that did them in. It was the market choosing the competition for the user experience plus what their friends were on. That's for most of the world. I have no idea what contributed to their failure in Asia since I d…

Asia? You came a few years to late... Friendster was US focused, at least until it lost. Then the probably pivoted, but I wrote about the first phase of Friendster (the one everyone in US remembers who was active online in early 2000s).

Hmm. It peaked in Asia with most users in Asia operating from Asia. You don't think the Asian focus could affect its marketing strategy for Americans? Facebook's college angle worked really well after MySpace's express yourself in cluttered pages worked before that. Im not sure Friendster understood our userbase enough to keep it.

Re: Does it scale? Who cares (2011)

#272

Post by nemothekid »

acdha wrote: post #208
nemothekid wrote: post #204

Earlier quoted context omitted.

>believe they'd need a clustered NoSQL database or it wouldn't scale. While for many developers, NoSQL might be overkill - Stackoverflow is a bad example. If you were any fast growing startup in the cloud, and you wanted to go the SO route it would have meant going CoLo. SO has set of machines with a nearly a terabyte of RAM - GCE doesn't even offer cloud machines with the same specs. And even then their setup is far…

One: run the numbers on the dozens of cloud nodes you mentioned. Is that really cheaper than renting some servers in a colo? Second: how much time would they have spent rolling all of the data integrity, reporting, etc. features they'd have needed to add. I'm inclined to take them at their word when they say this was safer and cheaper given their resources.

>I'm inclined to take them at their word when they say this was safer and cheaper given their resources.

I never meant to claim that cloud was safer and cheaper. What I meant was, for the majority of operations, staying in the cloud with some distributed setup is likely more feasible that moving to CoLo (see GitLab).

Re: Does it scale? Who cares (2011)

#273

Post by RugnirViking »

grey-area wrote: post #88
cyberferret wrote: post #52

Earlier quoted context omitted.

> I know of absolutely no service which failed because it couldn't scale. I would say this is because of the simple fact of visibility and adoption. You've probably never heard of these services probably because they ground to a halt with a mere 1000 users, so they never got mainstream enough to be recognised as a viable service. It is a bit like how no one remembers the dozens of people who failed to achieve sustain…

I am willing to bet that 9 out of 10 'weekend projects' have had to be thrown out completely and redeveloped from scratch when the number of users became significant. 9 out of 10 weekend projects never get to a significant number of users.

indeed, I would go so far as to say 9/10 weekend projects never get finished, let alone released, let alone getting enough users

Re: Does it scale? Who cares (2011)

#274

Post by luord »

shadowmint wrote: post #268
luord wrote: post #267

Earlier quoted context omitted.

> If you're using python, it can be the bottle neck > data processing workload you need to do in parallel It's funny how often those two go together, you're of course correct (for now). I never wrote that the language couldn't be the bottleneck, though. If you are doing something that requires heavy parallelization then, by all means, don't use python or replace it if you're already using it... But that kind of workl…

> Only if one is working with poor developers It's not about the developers, it's about the workload. Objectively, you can't write high performance multi-threaded python. No one can; it's not possible; it's just slow. If you're rewriting your code in C++ so its not slow and pretending its python, you should just rewrite your code in C++. That's not writing fast python, it's writing C++. /shrug So python can easily be…

> it's about the workload.

... Why did you repeat yourself? I already told you I agreed with you on that.

Is it something specific you want me to tell you? It's going to be easier if you tell me what you want to read, otherwise we'll keep going in circles.

> Quite easily [...] poor purpose

That's an immediate contradiction. Yes, if one picks the wrong tool, that tool often becomes the bottleneck; but for every other scenario in which the tool is alright, it takes poor developers for the tool to become a bottleneck.

In my first comment I wrote that the language isn't the bottleneck, except for specific circumstances... And you took one of those specific circumstances and keep running with it. It's not a counterargument to my original point, if that's what you're trying to do.

Re: Does it scale? Who cares (2011)

#275

Post by ccmonnett »

technosamay20 wrote: post #37
doctorcroc wrote: post #22

Earlier quoted context omitted.

Awesome story and takeaways. What happened to your product and what did you learn you could not keep up with or grow into?

I built this product as a side project within another startup I was working in. We had lot of money, had been around for 3 years and did not have a product. Because of this I had multiple levels of bosses above me in the company. They all got really interested in starting to manage me and the product once we hit 5M users. They had a different vision, attachment and passion about the product. Around 2008, we were spen…

Thanks for sharing your experience. People are getting caught up in Ruby v Java v whatever and not seeing the forest for the trees - you have a valuable perspective and an effective way of communicating it. Appreciate it!

May I ask you about your first 'biggest learning': how do you eschew cofounder responsibilities in an early company? Someone who genuinely cares and/or is really interested in the core business (pretty common among first hires of startups) may have a hard time turning work away or willfully not participating in major discussions/planning when they believe their contributions could be valuable. In other words, situations when you know you can contribute but the company will get the 'better end of the deal' as you take on more responsibility without taking on more compensation.

It sounds difficult to ride that out - were you able to do that successfully or are you saying "This happened to me, don't let it happen to you?"

Re: Does it scale? Who cares (2011)

#276

Post by simonh »

sidlls wrote: post #229
simonh wrote: post #185

Earlier quoted context omitted.

Having worked on corporate Python platform with tens of thousands of developers and hundreds of applications running hubdress of thousands of instances on a many million line common core code base, I beg to differ. We were sometimes able to go from concept to deployed application in a single day.

> We were sometimes able to go from concept to deployed application in a single day. This just makes me shudder a little bit. I've seen such things before, even recently at my current job. These kinds of applications tend to be a nightmarish mix of unmaintainably overengineered and underdesigned terrible code.

It's not ideal certainly, most such cases would be either a variation of an existing application or something that isn't business facing such as a compliance or operational metrics report.

The key thing is that the testing, release and deployment tooling were not a significant bottleneck. You could develop the code, perform testing in a production like environment and sign it off with a minimum of down time in between waiting for compilation, package building, etc. So a lot of that was down to Python being interpreted.

Re: Does it scale? Who cares (2011)

#277

Post by nasalgoat »

technosamay20 wrote: post #15

Couldn't agree with this article more. I built the biggest social network to come out of India from 2006-2009. It was like Twitter but over text messaging. At it's peak it had 50M+ users and sent 1B+ text messages in a day. When I started, the app was on a single machine. I didn't know a lot about databases and scaling. Didn't even know what database indexes are and what are their benefits. Just built the basic produ…

> I know of absolutely no service which failed because it couldn't scale.

I know HN skews young when no one remembers Friendster.

Re: Does it scale? Who cares (2011)

#278

Post by developer2 »

ne01 wrote: post #248
developer2 wrote: post #194

Earlier quoted context omitted.

>> You can do a lot with 244 GB RAM machines Is this a typo of "244 GB" instead of "24 GB"? Nearly any company that has a single machine provisioned with 244 GB of RAM is doing something severely wrong, likely putting the company's ability to grow at risk. Such a machine screams of trying to vertically scale a poorly performing legacy product instead of figuring out to horizontally scale out with 16-64 GB servers. Th…

I know Basecamp uses a server with 2TB of RAM for their single MySQL server. FYI CTO of Basecamp is the creator of Ruby on Rails -- smart people. Checkout​ stack overflow stack too. Their system is faster too.

That is actually insane. The day their MySQL cluster experiences a severe downtime incident, it is likely going to take an enormous amount of time to recover.

That is my intuition. If you know of a video or blog post (conference, talk, article, etc.) where someone explains the benefits of a 2 TB MySQL server and how it is not a crazy bad idea, I would love to see it. Because my 15 years of professional experience screams "NO WAY IN HELL" at that one.

Re: Does it scale? Who cares (2011)

#279

Post by fulafel »

developer2 wrote: post #194
technosamay20 wrote: post #59

Earlier quoted context omitted.

I mentioned something similar in a separate comment. My initial DB schema was pretty bad. We did at 2 schema rewrites and migrations from the launch to 5M users. Each time it took 2 weeks of sleep less nights. The machines today are really powerful. You can do a lot with 244 GB RAM machines backed by SSD. Someone who doesn't have the skill set to be able to scale once they get traction - it's likely they will not hav…

>> You can do a lot with 244 GB RAM machines Is this a typo of "244 GB" instead of "24 GB"? Nearly any company that has a single machine provisioned with 244 GB of RAM is doing something severely wrong, likely putting the company's ability to grow at risk. Such a machine screams of trying to vertically scale a poorly performing legacy product instead of figuring out to horizontally scale out with 16-64 GB servers. Th…

Keeping all data cached in RAM is something that eats a bunch of memory, but is not usually wrong.

Re: Does it scale? Who cares (2011)

#280

Post by fulafel »

bpicolo wrote: post #206
developer2 wrote: post #194

Earlier quoted context omitted.

>> You can do a lot with 244 GB RAM machines Is this a typo of "244 GB" instead of "24 GB"? Nearly any company that has a single machine provisioned with 244 GB of RAM is doing something severely wrong, likely putting the company's ability to grow at risk. Such a machine screams of trying to vertically scale a poorly performing legacy product instead of figuring out to horizontally scale out with 16-64 GB servers. Th…

Pretty sure it's not a typo. Amazon i3.16xlarge is 488 GB of Ram, 64 vCPUs, and a 20 Gigabit connection. Postgres/Mysql can absolutely scream perf-wise on that.

AWS also has 2 TB instances: https://aws.amazon.com/ec2/instance-types/x1/

Outside AWS, you can put 3-6 TB in a normal-ish server from Dell or HP, or 64 TB in a big iron box from Fujitsu or IBM.

Post reply on HN