Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

StackOverflow Update: 560M Pageviews a Month, 25 Servers

highscalability.com

Post reply on HN
Print view View original

211–220 of 278 posts

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#211

Post by noelwelsh »

skndr wrote: post #209
noelwelsh wrote: post #207

StackOverflow shows just how powerful using a fast language can be. Compare the TechEmpower benchmarks for, say, Java vs Rails on a very simple JSON serialization benchmark (we can assume .Net would achieve comparable performance to the JVM): http://www.techempower.com/benchmarks/ The Java servers get around 900K requests/s on beefy hardware while Rails squeezes out 6K. That's a 150x difference! Any real application…

Conversely, you might be able to iterate faster with a 'web' language and create a product people really use in a shorter time frame and worry about scaling later!

5 years ago it might have been true that RoR was that much more productive it was worth using, but that doesn't match my experience today using RoR and Scala.

Update: To try to avoid turning this into a language war, there are other good reasons to use something like RoR. Ease of hiring is one. Using known tools is another, if you're not building anything you expect to get load.

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#212

Post by AlisdairO »

AlisdairO wrote: post #200
samstave wrote: post #138

Earlier quoted context omitted.

> For non-trivial applications scale-out has substantial complexity costs attached to it Forgie me if I am misunderstanding you - but non-trivial applications can actually require scale out. From my perspective, StackExchange is not techinically that complex. They have built a very efficient, cost-effective and performant stack for their singular application and that works very well for them, but the complexity of th…

By nontrivial I mean 'application which, in your document database, requires multi-document updates to perform some individual logical operations'. This is a rather low bar :-). The fact is, the vast majority of projects that programmers are working on are less computationally complex than stack overflow. That's not to say that forum software is all that complex, more that most problems are pretty simple. Of course t…

Er, I don't seem to be able to edit, but I guess I should specify that I'm really talking about scale-out of writes here. Read scalability is an obviously easier problem..

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#213

Post by MrBuddyCasino »

Griever wrote: post #89
MichaelGG wrote: post #51

Earlier quoted context omitted.

I find the opposition to static classes odd. Unless you need the data encapsulation of an object, why force people to create an object to call functions? I find too much C# code loves creating objects for no real purpose other than that's how it works. For SO, if we guess they max out around 2000 req/sec, then if a bunch of objects are being allocated for each request, there could be added GC pressure. From my own ex…

I think the real issue (although it doesn't seem to be an issue for SO) is testability. AFAIK, static classes cannot be mocked, and you are pretty much boxing yourself into only doing integration tests for everything. Not necessarily a bad way to go, just lots more setup per test.

Yep, that used to be one of the main reasons to have DI in Java. But now that everything (even static methods) can be mocked, that reason is no longer valid. Is this not possible in .NET?

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#214

Post by farresito »

ksec wrote: post #210

It is fascinating to see how well its server it holding out. With Plenty of headroom to spare!. With Higher Capacity DDR4, and Haswell or even Broadwell Xeon, and PCIe SSD getting cheaper and faster. It is not hard to imagine they could handle 3 - 4 times the load when they upgrade their server two years down the line. But i would love to see Joel's take on it though. Since he is a Ruby guy now, and I dont think you…

It would definitely be several orders of magnitude slower. That's the price you pay when you are using a language like ruby: it's very productive and fast to iterate in, but way less performant. At the end of the day, what you use depends on your needs.

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#215

Post by Just-A-Guest »

Nick-Craver wrote: post #38
junto wrote: post #12

Earlier quoted context omitted.

Their SQL Server is Microsoft SQL Server running on (shock horror) Windows. Says in the article: Windows 2012 is used in New York but are upgrading to 2012 R2 (Oregon is already on it).

To be clear, most servers are moving to 2012 R2 (most already have) but the machines our SQL clusters run on, specifically, will not. This is because you have to effectively (or literally) rebuild the Windows cluster from scratch and we just don't get that level of benefit from the 2012 to 2012 R2 upgrade. There are quite a few improvements we care about: native NVMe, better dynamic quorum, better DSC support, better…

Do I get this right? You can not upgrade a server cluster to the next version of the operating system without complete new installation??? Could you please write something about these kind of things, this is very interesting!

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#216

Post by Nick-Craver »

Just-A-Guest wrote: post #215
Nick-Craver wrote: post #38

Earlier quoted context omitted.

To be clear, most servers are moving to 2012 R2 (most already have) but the machines our SQL clusters run on, specifically, will not. This is because you have to effectively (or literally) rebuild the Windows cluster from scratch and we just don't get that level of benefit from the 2012 to 2012 R2 upgrade. There are quite a few improvements we care about: native NVMe, better dynamic quorum, better DSC support, better…

Do I get this right? You can not upgrade a server cluster to the next version of the operating system without complete new installation??? Could you please write something about these kind of things, this is very interesting!

I'm not sure a full writeup would have much more detail, but yes. With Windows clustering (on top of which SQL 2012/2014 Availability Groups are based), you can only have a homogenous OS version across the board. Since they all must have the same version, you can't have something like a single 2012 R2 instance in an otherwise 2012 cluster...and since you can't do that, you can't upgrade one at a time. Yay! New cluster time instead!

Trust me, we're bitching about this as are most people and I think changes must be coming there. It doesn't matter how many fancy features you add to the OS if we can't upgrade to it, so they'll have to stop and address that problem.

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#217

Post by Nick-Craver »

jaequery wrote: post #178
valarauca1 wrote: post #80

Earlier quoted context omitted.

Most people over estimate hardware cost. Buying a 300-500GB ram server maybe very expensive. But buying a distributed data base expert likely costs at least an order of magnitude higher per year.

Same server in AWS, it'll cost $50k+/month for the same thing. SO just bought it one time fee for probably around $30k or so, I'll say they are doing something right!

Worth noting: AWS also doesn't have Enterprise SQL available, so plan on buying that yourself. That's a not-so-insignificant piece of the overall cost to many people.

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#218

Post by ZoFreX »

noelwelsh wrote: post #207

StackOverflow shows just how powerful using a fast language can be. Compare the TechEmpower benchmarks for, say, Java vs Rails on a very simple JSON serialization benchmark (we can assume .Net would achieve comparable performance to the JVM): http://www.techempower.com/benchmarks/ The Java servers get around 900K requests/s on beefy hardware while Rails squeezes out 6K. That's a 150x difference! Any real application…

> That's a 150x difference! Any real application will be slower than this

How many applications are rate limited by the speed of the front-end language? Not that many; the speed of the backing store will usually be the bottleneck.

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#219

Post by Nick-Craver »

zaroth wrote: post #187
Nick-Craver wrote: post #155

Earlier quoted context omitted.

Peak is more like 2600-3000 requests/sec on most weekdays. Remember that programming, being a profession, means our weekdays are significantly busier than weekends (as you can see here: https://www.quantcast.com/p-c1rF4kxgLUzNc ). It's almost all over 9 servers, because 10 and 11 are only for meta.stackexchange.com, meta.stackoverflow.com, and the development tier. Those servers also run around 10-20% CPU which means…

I'm most impressed that all your traffic is passing through a single active HAproxy which is also terminating SSL? TLS Session IDs and tickets of course are absolutely essential, but I'd be curious how many peak TPS (number of full handshakes / sec) you are seeing on HAproxy. The alternative, fanning out your SSL termination to your IIS endpoints, unfortunately means running HAproxy at L2, so you lose all your nice l…

I'm not sure how many full handshakes/sec we're running - the logging isn't as readily available there because we're explicitly not logging the listeners involved on separate proc binds. We are logging the traffic on the port 80 front end they feed. We have tuned the performance there via best practices and taking into account very recent HAProxy changes for various SSL windows added by Willy, Emeric and help from Ilya Grigorik. They're all awesome and tremendously helpful. We are on a version slightly newer than 1.5.2 (we build straight from the repo).

Even with all that SSL we're only running around 15% CPU at peak so it's not having any trouble. Most of that CPU usage does come from the SSL termination though - it ran around 3-5% CPU before. We're also working on much larger infrastructure changes that mean SSL termination much more local to our users, which means the HAProxy load will drop to just about nothing again. I'm working on a followup to my previous SSL post: http://nickcraver.com/blog/2013/04/23/stackoverflow-com-the-... that will explain why it's not already on for all hits...I think hacker news will have fun with hat one.

All that being said, there's no reason we can't forward that syslog traffic from the listeners to our logging infrastructure to get at least a counter out of it. If you're curious how we log, I'll explain a bit.

We feed it into a logging display called Realog built by Kyle Brandt and Matt Jibson in Go. Here are a few snapshots of the dashboard: http://i.stack.imgur.com/OIqhm.png http://i.stack.imgur.com/iQOP9.png http://i.stack.imgur.com/bfpUb.png http://i.stack.imgur.com/JZNy6.png

This lets us forward on custom headers to logging that the web servers are sending back. It tells us how much time we spent in SQL, redis, etc. We can graph it (bottom of the first link) and identify the root cause of any issues faster. It also handles parsing of that traffic from the syslog entry already, so we use it as a single logging point from HAProxy and it handles the JSON generation to forward that traffic data into logstash (a 300 TB cluster we're working on setting up right now).

As soon as we get the logstash piece working well and dogfooded thoroughly, I'll poke Kyle to open source Realog so hopefully some others can get use from it.

Nick Craver - Stack Exchange Sysadmin & Developer

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#220

Post by Nick-Craver »

doczoidberg wrote: post #206

I would love to see comparing costs of their solution against cloud solutions as Azure. They say that hardware is cheaper than programmers. Wouldn't that speak for the cloud solution (software based scaling instead of admins)?

Hardware is cheaper than developers and efficient code. But you're only as fast as your slowest bottleneck and all the current cloud solutions have fundamental performance or capacity limits we run into.

Could you do it well if building for the cloud from day one? Yeah sure, I think so. Could you consistency render all your pages performing several up to date queries and cache fetches across that cloud network you don't control and getting sub 50ms render times? Well that's another matter, and unless you're talking about substantially higher cost (at least 3-4x), the answer is no - it's still more economical for us to host our own servers.

And yes I realize that's a fun claim to make without detail. As soon as I get a free day (I think there's one scheduled in 2017 somewhere), I'll get with George to do another estimate of what AWS and Azure would cost and what bottlenecks they each have.

Nick Craver - Stack Exchange Sysadmin & Developer

Post reply on HN