Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

StackOverflow Update: 560M Pageviews a Month, 25 Servers

highscalability.com

Post reply on HN
Print view View original

141–150 of 278 posts

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#141

Post by bjourne »

numbsafari wrote: post #119
ternaryoperator wrote: post #113

"SO goes to great lengths to reduce garbage collection costs, skipping practices like TDD, avoiding layers of abstraction, and using static methods." I don't understand this at all. What does TDD have to do with reducing garbage collection?

My guess is that they feel that the layers of indirection and abstraction often needed to make TDD work result in an object creation pattern that results in heavy GC load during normal operation. The references to "using static methods" is probably related to this. ps. That's my guess, but I'd encourage you to post your question to the meta site for SO.

IME that always happens when you try to perform tdd in combination with Javaesque encapsulation. The good solution to the problem is to not be so afraid of classes seeing each others internals. The bad solution is to add, factory patterns, dependency injectors and other useless layers just to try and keep your design both well encapsulated and testable.

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#143

Post by jacquesm »

Alupis wrote: post #87
ufo wrote: post #27

Earlier quoted context omitted.

Anyone who knows Joel Spolsky would expect a MS stack :)

It's difficult to take pragmatic advice from someone who took the least pragmatic (and most expensive) route for a startup web company.

Programmer time is a factor that comes into play when you build something and re-training your people to use a stack they are not familiar with + the associated mistakes can get expensive in a hurry.

If you happen to be familiar with the MS stack and are able to factor the license fees into your business plan there is no strong reason against and many for using that particular stack.

MS programmers tend to be a bit more expensive than people working with one of the many Linux offerings but that's not such a huge difference that it would become the deciding factor and if anything the MS stack is more performant than the Linux one on the same hardware (and I write that as an anti-MS guy, I strongly believe that politics and tech should be kept separate when it comes to discussing systems relative to each other).

So even if there are lots of reasons why I would not have made that particular choice I can completely understand why the SO people made their choice the other way.

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#144

Post by adventured »

ZachPruckowski wrote: post #69
mrb wrote: post #29

Earlier quoted context omitted.

$4k is all you need to build a whitebox 256GB machine: dual-socket G34 motherboard with 16 DDR3 slots lke the ASUS KGPE-D16 ($500) + two Opteron 6320 (2 * $300) + sixteen 16GB registered ECC DDR3-1600 (16 * $160) + chassis/PSU ($500) = $4160. Doing this on Amazon r3.8xlarge ($2.8/hr) would cost you $8100+ over 4 months. A DIY server can make sense in some cases. Eg. you are a very early-stage startup with almost no f…

Don't forget to factor in the costs of running, cooling, and maintaining that much hardware (also bandwidth costs). I'm not saying it's $2k/month, I'm just saying don't compare the physical box and AWS without factoring everything in. At the end of the day, if you don't need the support, you're probably right that a dedicated box makes sense there.

AWS has among the highest bandwidth costs you're going to run into, if we're talking co-location, dedicated, or self-hosting. Their bandwidth costs are the sole reason I won't go anywhere near AWS yet, I consider their prices outrageously high.

10tb = roughly $1,000 per month

They're between 5 and 50 times more expensive on bandwidth than the options in dedicated / colo / self hosting.

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#145

Post by threeseed »

sklivvz1971 wrote: post #99
MichaelGG wrote: post #55

So this is what, a 2000 request/sec peak? Over 11 servers, that's like 200 requests/sec peak per frontend? The problem with scale-up is if you actually have to get a few times larger, it becomes super expensive. But fortunately hardware is increasing so much that you can probably just get away with it now. There's probably a crossover point we're rapidly approaching where even global-scale sites can just do all their…

Yes, the idea is to scale up as much as economically makes sense and only then scale out, but due to Moore's law we're still scaling up. Not everyone has google-like problems that are betters solved by a battery of cheap boxes.

Actually most of the websites today are better solved by a battery of cheap boxes.

You simply put a load balancer (Nginx/ELB/HAProxy etc) in front of a fleet of smaller web/application servers that dynamically scale depending on traffic. That way it is cost effective, far more reliable, easier to scale and you can tolerate DC outages better.

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#146

Post by fideloper »

I love that all the traffic is running through a single HAProxy box - a software load balancer (vs a hardware LB like F5).

And they've moved SSL termination to it.

That's a great quality product, and easy to setup.

Edit: I work I'm software that supports MySQL,PG and SqlServer. SqlServerr seems to be the most stable and consistent in performance - they're hard to kill! One of my few liked MS products :D

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#147

Post by Swannie »

lemcoe9 wrote: post #3

Makes me wonder why smaller website teams need dozens of engineers to keep their infinitesimally smaller app running.

From my limited experience, it's a lack of the kind of systems thinking present in companies like Stack Exchange, and present on the High Scalability blog, that results in a poor architecture.

And this is compounded by people who have little ability to troubleshoot performance issues. It's quite easy to hunt down the cruddy SQL queries in a DB, or realize that you spinning rust is too slow. But when it starts to come down to things like a blocking network fabric, which has some big fat buffer between two servers, that is killing your transaction speed - many will just start to blame the devs.

Of course, the second point is compounded by the first - the less systems thinking that went into the design in the first place, the harder it is to produce accurate hypotheses about the system in order to troubleshoot.

Whilst there is a lot to be said for the "right tool for the job", if you watch an artisan crafting something, you'll realize that despite having a huge number of tools, they actually get by with relatively few. This is a generalization, but they only use the full range when doing something new, solving a particularly tricky problem, etc. "All the gear and no idea" is certainly applicable in many start ups.

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#148

Post by skeletonjelly »

rakoo wrote: post #64

What I'd like to see is how much cache hit they add on each level of cache. I remember some presentation of the facebook images stack where less than 10% (if I remember well) of the requests actually hit the disks; it would be interesting to see the patterns for the whole SE galaxy.

As someone mentioned above, SQL Server hits memory first, and as they have 384GB of memory, most of the requests would sit there. That's just on the DB server. In the linked article it all authenticated requests hit the DB, with anon users getting a cached copy.

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#149

Post by skeletonjelly »

stonemetal wrote: post #37
UVB-76 wrote: post #18

Anyone else not quite expecting StackExchange to be using a Microsoft stack?

Joel worked at MS on office, Excel if I recall correctly. Jeff's blog is named for a feature from a book published by MS. I would be more surprised if it was MS free.

Trello (by Joel's company) is OS agnostic (running on MongoDB and node.js). But from memory they did this to learn new tools.

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#150

Post by Alupis »

jacquesm wrote: post #143
Alupis wrote: post #87

Earlier quoted context omitted.

It's difficult to take pragmatic advice from someone who took the least pragmatic (and most expensive) route for a startup web company.

Programmer time is a factor that comes into play when you build something and re-training your people to use a stack they are not familiar with + the associated mistakes can get expensive in a hurry. If you happen to be familiar with the MS stack and are able to factor the license fees into your business plan there is no strong reason against and many for using that particular stack. MS programmers tend to be a bit m…

> the MS stack is more performant than the Linux one on the same hardware

This is provably not true.

http://w3techs.com/technologies/overview/web_server/all

http://commons.wikimedia.org/wiki/File:Operating_systems_use...

http://w3techs.com/technologies/overview/operating_system/al...

There is a good reason majority of the World-Wide-Web is run on 'Nix stacks -- and there is a good reason majority of servers in general are 'Nix stacks. Also, an overwhelming percentage of super-computers (I know we are talking about webservers, but it illustrates performance capabilities) are running a 'Nix stack.

There is also cost involved, as well as flexibility of the stack. With a 'Nix stack, it's infinitely flexible, not so much the case with a Microsoft stack. Also the default MS Stack comes with a lot of additional OS overhead that is not present in the 'Nix stack, which reduces any said box's scalability.

Without trying to turn this into some sort of flame war - I was merely trying to suggest that the MS stack was not/is not the best choice for a highly scalable website. Take the top 10 websites -- they all run on 'Nix. The top cloud providers (except MS Azure), all 'Nix. These are companies that can easily afford MS licensing, so that's not part of the equation.

A MS stack at this scale is unusual to say the least.

That's not to say it won't scale (as evidenced by the SE team), but it doesn't mean there isn't a better alternative that saves more money and scales better with less hardware, etc.

Post reply on HN