Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

StackOverflow Update: 560M Pageviews a Month, 25 Servers

highscalability.com

Post reply on HN
Print view View original

171–180 of 278 posts

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#171

Post by gambler »

kbenson wrote: post #33

> The cost of inefficient code can be higher than you think. Efficient code stretches hardware further, reduces power usage, makes code easier for programmers to understand. I'm curious what the reasoning is for "Efficient code ... makes code easier for programmers to understand". To my mind, efficient code (in this case, I assume coding to the hardware, as they mention elsewhere), has many benefits, but making it ea…

In complex systems low-level efficiency is far less beneficial than high-level efficiency. To achieve high-level efficiency you need to have a clear understanding of how the system works as a whole. That is impossible is the code is unintelligible bag of tricks.

I've seen this a lot when working in PHP. I wrote some websites from ground-up (using something similar to my own framework). I optimized int multiple times to great results. The biggest benefits didn't come from making a particular function faster, they came from realizations that large chunks of complexity in templating, routing and permissions checking subsystems simply weren't necessary. It doesn't matter how clever those chunks were written, because I got rid of them completely.

I think this is what the author speaks about.

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#172

Post by rjbwork »

jacques_chester wrote: post #170
numbsafari wrote: post #119

Earlier quoted context omitted.

My guess is that they feel that the layers of indirection and abstraction often needed to make TDD work result in an object creation pattern that results in heavy GC load during normal operation. The references to "using static methods" is probably related to this. ps. That's my guess, but I'd encourage you to post your question to the meta site for SO.

Insofar as this is your guess, it's down to the language. In Ruby or Javascript it's very easy to TDD. In Java, much less so. (I'm not sure where C# lies on the spectrum).

Somewhere in the middle, but leans towards Java.

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#173

Post by jacques_chester »

rjbwork wrote: post #172
jacques_chester wrote: post #170

Earlier quoted context omitted.

Insofar as this is your guess, it's down to the language. In Ruby or Javascript it's very easy to TDD. In Java, much less so. (I'm not sure where C# lies on the spectrum).

Somewhere in the middle, but leans towards Java.

Sounds about right.

At the risk of exposing my slow transmutation into a hipster programmer, I and a colleague found that mocking in Go was much easier than we anticipated, thanks to the way interfaces work.

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#174

Post by Nick-Craver »

ck2 wrote: post #134

This is the fascinating part to me, their SSD have not failed: Failures have not been a problem, even with hundreds of intel 2.5" SSDs in production, a single one hasn’t failed yet. One or more spare parts are kept for each model, but multiple drive failure hasn't been a concern.

Yep, still true. We lost one Intel 910 drive (PCIe SSD), and that was very abnormal - died so soon it was almost DOA. We hooked up directly with Intel for them to The replacement is still going strong as is another 910 we have.

All of those 2.5" Intels though, still trucking along! We're looking at some P3700s PCIe NVMe drives now, blog post coming about that.

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#175

Post by Nick-Craver »

Alupis wrote: post #87
ufo wrote: post #27

Earlier quoted context omitted.

Anyone who knows Joel Spolsky would expect a MS stack :)

It's difficult to take pragmatic advice from someone who took the least pragmatic (and most expensive) route for a startup web company.

I think you're forgetting BizSpark here, which we were in: http://www.microsoft.com/bizspark/

IMO, it has one of the best tooling sets of any platform out there (Visual Studio) which saves lots of programmer time and increases productivity. With BizSpark the initial cost argument is out...so how's that not pragmatic?

I'm biased of course, but I think it worked out pretty well.

Nick Craver - Stack Exchange Sysadmin & Developer

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#176

Post by bagels »

voltagex_ wrote: post #158
Nick-Craver wrote: post #155

Earlier quoted context omitted.

Peak is more like 2600-3000 requests/sec on most weekdays. Remember that programming, being a profession, means our weekdays are significantly busier than weekends (as you can see here: https://www.quantcast.com/p-c1rF4kxgLUzNc ). It's almost all over 9 servers, because 10 and 11 are only for meta.stackexchange.com, meta.stackoverflow.com, and the development tier. Those servers also run around 10-20% CPU which means…

>I actually just wrote a lengthy email about what we're planning for storage on one of our SQL clusters...perhaps I should echo it verbatim as a blog post? I'm not sure how many people care about that sort of stuff outside our teams. I'm sure some DBAs and devs here would find it interesting.

This is definitely the case. Every writeup from those in the trenches I read and share with coworkers.

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#178

Post by jaequery »

valarauca1 wrote: post #80
bradyd wrote: post #14

Earlier quoted context omitted.

I don't believe that much RAM is uncommon for large scale database servers. 384GB RAM is only about $5000 from Dell. They also have a new server model coming out that supports up to 6TB of RAM [0]. [0] http://www.dell.com/us/business/p/poweredge-r920/pd

Most people over estimate hardware cost. Buying a 300-500GB ram server maybe very expensive. But buying a distributed data base expert likely costs at least an order of magnitude higher per year.

Same server in AWS, it'll cost $50k+/month for the same thing. SO just bought it one time fee for probably around $30k or so, I'll say they are doing something right!

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#179

Post by Nick-Craver »

Alupis wrote: post #85
UVB-76 wrote: post #18

Anyone else not quite expecting StackExchange to be using a Microsoft stack?

One must wonder, how much better this setup would of scaled had it been a more appropriate webserver stack such as Linux/BSD + Apache/Nginx or similar. Perhaps less boxes would be needed, or with the same number of boxes, more concurrent users. > Microsoft infrastructure works and is cheap enough not as cheap as just paying your team to maintain the boxes. I wonder how many times in the past few years SE has needed t…

> One must wonder, how much better this setup would of scaled had it been a more appropriate webserver stack such as Linux/BSD + Apache/Nginx or similar

I guess you're gonna have to define "more appropriate" for me, I have a feeling we'll have a fundamental disagreement there. We could (and have) run the entirety of our peak load with 1 SQL server and 2 web servers (and not pegging them). Don't forget we run with a crazy amount of headroom at all times, by design. I'm not sure how much better you picture that scaling on a linux environment.

> not as cheap as just paying your team to maintain the boxes

That makes a lot of assumptions about the team, their expertise, and what support issues would arise.

> I wonder how many times in the past few years SE has needed to call Microsoft for support

In the past 4 years I've been here? Twice. Both to report bugs in CTP versions of SQL server. At no cost, and we improved SQL server as a result of being testers. We have a very good relationship with Microsoft and talk to the developers that make the tools and platform we use in order to make life better for both of us. We do the same thing for redis, Elasticsearch, etc. It's the same reason we open source almost all of the tools we make.

> If something like CentOS then there would be no support cost unless you need to bring in an outside contractor (for a particularly nasty issue)

We use CentOS for all our linux systems and are deploying new servers on CentOS 7 now. We'll me migrating the others in the coming months. That doesn't mean it's free. Developer or sysadmin time to control Puppet deployments and such still eat some factor of time.

Nick Craver - Stack Exchange Sysadmin & Developer

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#180

Post by csharrison »

beachstartup wrote: post #70
programminggeek wrote: post #16

One smart thing they are doing is putting different sites on different databases. It effectively as a kind of partitioning to allow for horizontal scaling if needed. If nothing else, it keeps less data in each table, so queries should be faster due to smaller datasets. Sounds like they know what they are doing.

beware that you can over-optimized this. for example, do not put every single customer of an saas solution into a separate database.

I'm curious, why not? If your saas was small enough and had few enough customers with large data needs, separating them each out into a separate database seems like a viable solution.
Post reply on HN