Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

StackOverflow Update: 560M Pageviews a Month, 25 Servers

highscalability.com

Post reply on HN
Print view View original

191–200 of 278 posts

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#191

Post by mistermann »

Nick-Craver wrote: post #164
pbz wrote: post #102

Earlier quoted context omitted.

Dapper is nice, but if you want to have the compiler check your queries, and still run fast, I recommend LINQ to DB: https://github.com/linq2db/linq2db (not to be confused with LINQ to SQL from Microsoft)

To be fair, the compiler is checking against what you told it the database looks like . This can help with lots of things, but still has the fundamental disconnect problem you'll get at runtime with either approach. A huge problem with the tradeoff we had (well, still have in some areas) is the generated SQL is nasty, and finding the original code it came from is often non-trivial. Lack of ability to hint queries, co…

I'm curious if the comments above exist in the source code, or have been added here for the benefit of readers on HN?

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#192

Post by virmundi »

louthy wrote: post #135
guiomie wrote: post #133

Can someone help me understand what they mean by: "Garbage collection driven programming. SO goes to great lengths to reduce garbage collection costs, skipping practices like TDD, avoiding layers of abstraction, and using static methods. While extreme, the result is highly performing code."

I assume it means using C# more as a functional language than an OO language, so that most garbage collection is of short-lifetime objects and therefore cheaper to collect (generation 0).

I take that statement to mean C# as C. Functional languages are terrible for garbage collection. Static methods are great since they are static, loaded once and probably in-lineable by the compiler/JIT.

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#193

Post by yazaddaruvala »

mseebach wrote: post #168
tomblomfield wrote: post #4

"One problem is not many tests. Tests aren’t needed because there’s a great community... If users find any problems with it they report the bugs that they’ve found." I'm often surprised at the paucity of test-coverage in relatively large companies. http://nerds.airbnb.com/testing-at-airbnb/

I think we're going trough a thesis/anti-thesis cycle on tests - in the beginning, there was militant testing, 100% coverage, testing getters and setters etc (as well as more complex stuff, obviously). Then some people started coming around to the idea that there are actually large swathes of code that is simple enough that testing doesn't actually add much value especially compared to the effort of writing them, the…

It really also does depend on the language, or rather compiler. Specifically, tests are far more useful when static analysis doesn't exist.

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#194

Post by mistermann »

Alupis wrote: post #123
stonemetal wrote: post #37

Earlier quoted context omitted.

Joel worked at MS on office, Excel if I recall correctly. Jeff's blog is named for a feature from a book published by MS. I would be more surprised if it was MS free.

You mean to tell me Joel is partially responsible for the abomination known as Excel 2007 where I can't pull-drag the damn window from screen to screen (but all of the other Office 2007 products do!)

WIN+SHIFT+ArrowKey

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#195

Post by mistermann »

Alupis wrote: post #87
ufo wrote: post #27

Earlier quoted context omitted.

Anyone who knows Joel Spolsky would expect a MS stack :)

It's difficult to take pragmatic advice from someone who took the least pragmatic (and most expensive) route for a startup web company.

How in this scenario is the MS stack the "least pragmatic" route?

prag·mat·ic adjective \prag-ˈma-tik\

: dealing with the problems that exist in a specific situation in a reasonable and logical way instead of depending on ideas and theories

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#196

Post by mistermann »

Alupis wrote: post #85
UVB-76 wrote: post #18

Anyone else not quite expecting StackExchange to be using a Microsoft stack?

One must wonder, how much better this setup would of scaled had it been a more appropriate webserver stack such as Linux/BSD + Apache/Nginx or similar. Perhaps less boxes would be needed, or with the same number of boxes, more concurrent users. > Microsoft infrastructure works and is cheap enough not as cheap as just paying your team to maintain the boxes. I wonder how many times in the past few years SE has needed t…

When you encounter people who don't respect your judgement as much as they "should", I'd suggest it may be because "how well it scales" doesn't guarantee "appropriateness". In most real world situations, the specifics of the chosen stack are rather inconsequential - both MS and *nix stacks perform more than adequately. In my opinion, it is unwise to take the advice of zealots who refuse to acknowledge this reality.

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#197

Post by shurcooL »

icebraining wrote: post #114
ProAm wrote: post #79

Earlier quoted context omitted.

I agree. Diffuses the quality of the related sites. Means you have to deal with twice as many moderators. I dont see why they couldnt have done a "subreddit' approach where you subscribe to the SE category you want and it shows on your homepage.

Hum, each subreddit has its own moderators as well. They also have a front page with questions from the different sites[1], though it's not filtered by the ones you have an account on. [1] http://stackexchange.com/

But you don't need to create new accounts with different passwords, and login/logout separately whenever you navigate between subreddits.

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#198

Post by kmontrose »

mistermann wrote: post #191
Nick-Craver wrote: post #164

Earlier quoted context omitted.

To be fair, the compiler is checking against what you told it the database looks like . This can help with lots of things, but still has the fundamental disconnect problem you'll get at runtime with either approach. A huge problem with the tradeoff we had (well, still have in some areas) is the generated SQL is nasty, and finding the original code it came from is often non-trivial. Lack of ability to hint queries, co…

I'm curious if the comments above exist in the source code, or have been added here for the benefit of readers on HN?

That's a straight copy/paste from the Stack Overflow codebase (Helpers\SqlMapper.cs:332 at time of writing, if you care). Now the typos and snark are out there for everyone to see.

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#199

Post by superuser2 »

bradyd wrote: post #14
habosa wrote: post #5

> With their SQL Servers loaded with 384 GB of RAM and 2TB of SSD, AWS would cost a fortune. I have next to zero experience with server administration, but 384GB seems like a lot to me. Is that common for production servers for popular web services? Do you need a customized OS to address that much memory? Seems like you'd really need to beef up the cache hierarchy make 0.38TB of RAM fast.

I don't believe that much RAM is uncommon for large scale database servers. 384GB RAM is only about $5000 from Dell. They also have a new server model coming out that supports up to 6TB of RAM [0]. [0] http://www.dell.com/us/business/p/poweredge-r920/pd

How common is it to buy one extremely powerful database server? Wouldn't you need the high availability properties of something like a Galera cluster? Or are most people accepting the single point of failure at huge database server?

(I honestly don't know, it's just surprising.)

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#200

Post by AlisdairO »

samstave wrote: post #138
AlisdairO wrote: post #105

Stack Overflow is the example I'm forever using when arguing against premature scale-out. For non-trivial applications scale-out has substantial complexity costs attached to it, and the overwhelming majority of applications will never truly need it . It's frustrating to see time wasted obsessing over trying to maintain eventual consistency (or chasing bugs when you failed to do so) on systems that could quite happily…

> For non-trivial applications scale-out has substantial complexity costs attached to it Forgie me if I am misunderstanding you - but non-trivial applications can actually require scale out. From my perspective, StackExchange is not techinically that complex. They have built a very efficient, cost-effective and performant stack for their singular application and that works very well for them, but the complexity of th…

By nontrivial I mean 'application which, in your document database, requires multi-document updates to perform some individual logical operations'. This is a rather low bar :-).

The fact is, the vast majority of projects that programmers are working on are less computationally complex than stack overflow. That's not to say that forum software is all that complex, more that most problems are pretty simple. Of course there are real reasons to use scale out - I simply advocate thinking hard about whether your problem will ever truly need it before taking the substantial complexity hit of coding for it.

Post reply on HN