Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

StackOverflow Update: 560M Pageviews a Month, 25 Servers

highscalability.com

Post reply on HN
Print view View original

181–190 of 278 posts

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#181

Post by Nick-Craver »

yblu wrote: post #177

> Redis has 2 slaves, SQL has 2 replicas, tag engine has 3 nodes, elastic has 3 nodes - any other service has high availability as well (and exists in both data centers) Does this mean the 2 Redis slaves, 2 SQL replicates etc. are in a secondary data center?

We have 2 SQL Clusters. In the primary data center (New York) there is usually 1 master and 1 read-only replica in each cluster. There's also 1 read-only replica (async) in the DR data center (Oregon). If we're running in Oregon then the primary is there and both of the New York replicas are read-only and async.

For redis both data centers have 2 servers, the chain would look like this typically:

  ny-redis01
   - ny-redis02
     - or-redis01
       - or-redis02
Not everything is slaved between data centers (very temporary cache data we don't need to eat bandwidth by syncing, etc.) but the big items are so we don't come up without any shared cache in case of a hard down in the active data center. We can start without a cache if we need to, but it isn't very graceful.

Here's a screenshot of our redis dashboard just a moment ago (counts are low because we upgraded to 2.8.12 last week, which needed a restart): http://i.stack.imgur.com/IgaBU.png

Nick Craver - Stack Exchange Sysadmin & Developer

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#182

Post by guiomie »

ygra wrote: post #107
tjdetwiler wrote: post #90

Earlier quoted context omitted.

A singleton object doesn't need to be re-allocated.

Static classes are an easy way of implementing the Singleton pattern in C# ...

A singleton can implement interfaces, static classes cant.

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#183

Post by jc4p »

jacques_chester wrote: post #169
jc4p wrote: post #34

Earlier quoted context omitted.

The things that obviously should have tests have tests. That means most of the things that touch money on our Careers product, and easily unit-testable features on the Core end (things with known inputs, e.g. flagging, our new top bar, etc), for most other things we just do a functionality test by hand and push it to our incubating site (formerly meta.stackoverflow, now meta.stackexchange). You can look at reported b…

You should come down our way and do a tech talk about it. If only for the wailing, gnashing of teeth and rending of garments.

Your people would literally murder me. Are you back from west coast / toronto?

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#184

Post by Shog9 »

phkahler wrote: post #91

I haven't been able to use SO since they shut off myopenid last year. I can't log in, so can't change my credentials ;-( Any ideas HN?

You can email us (as George said) or you can just punch your email address into the account recovery tool and get an automated email telling you what to click within the space of a few minutes: http://stackoverflow.com/users/account-recovery

We... probably need to make that link a bit more obvious.

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#185

Post by kondro »

jaequery wrote: post #178
valarauca1 wrote: post #80

Earlier quoted context omitted.

Most people over estimate hardware cost. Buying a 300-500GB ram server maybe very expensive. But buying a distributed data base expert likely costs at least an order of magnitude higher per year.

Same server in AWS, it'll cost $50k+/month for the same thing. SO just bought it one time fee for probably around $30k or so, I'll say they are doing something right!

The don't have servers with that much RAM, but a 244GB RAM server with 8x800GB SSDs and 32 cores costs <$5k a month ad-hoc and about $1300/month with a 3 year commitment.

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#186

Post by kondro »

josephscott wrote: post #62

"with just 25 servers" This really needs to stop. Go to stackexchange.com and you'll find that more than half of the HTTP requests are to cdn.sstatic.net. Looking up the IP addresses for cdn.sstatic.net returned five entries for me, all owned by CloudFlare. None of the CloudFlare servers that they are using seem to be in that 25 count. Sure, these are all for static assets, that isn't the point. There are way more th…

Sort of true, but static assets are also trivial to serve. What they get by using CloudFlare is getting their assets as close to end users as possible to reduce latency. This isn't a complex problem to solve (for a single site), just a really expensive one that very few single app companies could ever justify solving themselves. I believe Google and Amazon are the only ones with their own significant CDN infrastructure, everyone else outsources it.

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#187

Post by zaroth »

Nick-Craver wrote: post #155
MichaelGG wrote: post #55

So this is what, a 2000 request/sec peak? Over 11 servers, that's like 200 requests/sec peak per frontend? The problem with scale-up is if you actually have to get a few times larger, it becomes super expensive. But fortunately hardware is increasing so much that you can probably just get away with it now. There's probably a crossover point we're rapidly approaching where even global-scale sites can just do all their…

Peak is more like 2600-3000 requests/sec on most weekdays. Remember that programming, being a profession, means our weekdays are significantly busier than weekends (as you can see here: https://www.quantcast.com/p-c1rF4kxgLUzNc ). It's almost all over 9 servers, because 10 and 11 are only for meta.stackexchange.com, meta.stackoverflow.com, and the development tier. Those servers also run around 10-20% CPU which means…

I'm most impressed that all your traffic is passing through a single active HAproxy which is also terminating SSL?

TLS Session IDs and tickets of course are absolutely essential, but I'd be curious how many peak TPS (number of full handshakes / sec) you are seeing on HAproxy.

The alternative, fanning out your SSL termination to your IIS endpoints, unfortunately means running HAproxy at L2, so you lose all your nice logging.

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#188

Post by mistermann »

jacques_chester wrote: post #169
jc4p wrote: post #34

Earlier quoted context omitted.

The things that obviously should have tests have tests. That means most of the things that touch money on our Careers product, and easily unit-testable features on the Core end (things with known inputs, e.g. flagging, our new top bar, etc), for most other things we just do a functionality test by hand and push it to our incubating site (formerly meta.stackoverflow, now meta.stackexchange). You can look at reported b…

You should come down our way and do a tech talk about it. If only for the wailing, gnashing of teeth and rending of garments.

Indeed, they seem to have achieved what many have said is literally not possible.

Also, some might remember when they had Uncle Bob on the podcast, the topic being unit testing, and iirc Bob preaching the gospel (~ you must write plentiful unit tests, or else) and Jeff and Joel (especially) more or less saying they don't quite get why.

(Am I remembering this story correctly? I swear that's how I remember it.)

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#189

Post by iancarroll »

r00fus wrote: post #126
JelteF wrote: post #53

Earlier quoted context omitted.

You do need to have more than 200 rep on one site though.

Which one? I'm a full member on all the StackExchange sites I care about for now with the +100 cross-membership bonus.

In order to get the +100 bonus on all sites, you must have one profile on the network with >200 reputation.

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#190

Post by mistermann »

bjourne wrote: post #141
numbsafari wrote: post #119

Earlier quoted context omitted.

My guess is that they feel that the layers of indirection and abstraction often needed to make TDD work result in an object creation pattern that results in heavy GC load during normal operation. The references to "using static methods" is probably related to this. ps. That's my guess, but I'd encourage you to post your question to the meta site for SO.

IME that always happens when you try to perform tdd in combination with Javaesque encapsulation. The good solution to the problem is to not be so afraid of classes seeing each others internals. The bad solution is to add, factory patterns, dependency injectors and other useless layers just to try and keep your design both well encapsulated and testable.

> The bad solution is to add, factory patterns, dependency injectors and other useless layers just to try and keep your design both well encapsulated and testable.

Not doing this will cause much self-righteous snickering from some.

Post reply on HN