Live data from Hacker News

StackOverflow Update: 560M Pageviews a Month, 25 Servers

highscalability.com

161–170 of 278 posts

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#161

Earlier quoted context omitted.

I wrote a Dapper extension for working with SQL Server's geospatial queries and types a couple of years ago - have they added anything like that yet? Otherwise I'd be happy to add it.

Marc Gravell added in the ability to put in pretty much any custom type without adding any dependency weight about a month ago. You can see the commit here: https://github.com/StackExchange/dapper-dot-net/commit/e26ee... Look towards the end at the tests for example of how to hook up a custom type (it's pretty simple).

Cool - the one I wrote was years ago. Looks like it's not necessary any more.

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#162

Earlier quoted context omitted.

Don't forget to factor in the costs of running, cooling, and maintaining that much hardware (also bandwidth costs). I'm not saying it's $2k/month, I'm just saying don't compare the physical box and AWS without factoring everything in. At the end of the day, if you don't need the support, you're probably right that a dedicated box makes sense there.

AWS has among the highest bandwidth costs you're going to run into, if we're talking co-location, dedicated, or self-hosting. Their bandwidth costs are the sole reason I won't go anywhere near AWS yet, I consider their prices outrageously high. 10tb = roughly $1,000 per month They're between 5 and 50 times more expensive on bandwidth than the options in dedicated / colo / self hosting.

You can negotiate with AWS.

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#163
post #5

> With their SQL Servers loaded with 384 GB of RAM and 2TB of SSD, AWS would cost a fortune. I have next to zero experience with server administration, but 384GB seems like a lot to me. Is that common for production servers for popular web services? Do you need a customized OS to address that much memory? Seems like you'd really need to beef up the cache hierarchy make 0.38TB of RAM fast.

The simplest rule of computing is, if you have a problem that can be solved economically by just throwing more RAM into a single machine at it, you should do it (instead of any other dimension, and assuming you don't have other constraints).

No, that's no common for production servers for popular web services, though- they tend to shard across other dimensions because the web services are often CPU or network-bound.

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#164
post #102
post #2

I've recently started using their micro-ORM Dapper, and I like it a lot. I get the performance of hand-coded SQL, but without the tedious mapping from SqlDataReader to my entity.

Dapper is nice, but if you want to have the compiler check your queries, and still run fast, I recommend LINQ to DB: https://github.com/linq2db/linq2db (not to be confused with LINQ to SQL from Microsoft)

To be fair, the compiler is checking against what you told it the database looks like. This can help with lots of things, but still has the fundamental disconnect problem you'll get at runtime with either approach.

A huge problem with the tradeoff we had (well, still have in some areas) is the generated SQL is nasty, and finding the original code it came from is often non-trivial. Lack of ability to hint queries, control parameterization, etc. is also a big issue when trying to optimize queries. For example we (and by "we" I mean, "I made Marc Gravell do it") added literal replacement to Dapper to help with query parameterization which allows you to use things like filtered indexes. In dapper we also intercept the SQL calls to dapper and add add exactly where it came from. Here's what that looks like, replicate for the other methods:

    public static IEnumerable Query(this DataContext db, SqlBuilder.Template template, bool buffered = true, int? commandTimeout = null, IDbTransaction transaction = null, [CallerFilePath]string fromFile = null, [CallerLineNumber]int onLine = 0, string comment = null)
    {
        using (db.Connection.EnsureOpen())
        {
            return SqlMapper.Query(db.Connection, MarkSqlString(template.RawSql, fromFile, onLine, comment), template.Parameters as object, transaction ?? db.Transaction, buffered, commandTimeout);
        }
    }
And here's the our marking method tailored for our code, but you get the idea:

    private static string MarkSqlString(string sql, string path, int lineNumber, string comment)
    {
        if (path.IsNullOrEmpty() || lineNumber == 0)
        {
            return sql;
        }

        var commentWrap = " ";
        var i = sql.IndexOf(Environment.NewLine);

        // if we didn't find \n, or it was the very end, go to the first space method
        if (i 
This results in a comment at the top of the query like this:

    /* Models\Post.LinkedQuestions.cs@105 */ 
    select top (@top) 
           p.Id 
         , p.Title
         , p.Score
It's such a simple approach using the caller member attributes, but it saves so much time tracking things down.

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#165
post #134

This is the fascinating part to me, their SSD have not failed: Failures have not been a problem, even with hundreds of intel 2.5" SSDs in production, a single one hasn’t failed yet. One or more spare parts are kept for each model, but multiple drive failure hasn't been a concern.

I would agree that hundreds of drives is a large enough sample that you would expect to see some failures.

That said, all the SSDs I've had have taken repeated pounding with no complaints. I accidentally swapped a few terabytes to swap on an SSD, and was simply surprised that the job I was doing finished faster.

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#166

Earlier quoted context omitted.

But who in their right mind would do the second one? Oh abritishguy, I can assure you that the world is awash in such magical code snippets.

> competent programmer

> true scotsman

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#167
post #9
post #5

> With their SQL Servers loaded with 384 GB of RAM and 2TB of SSD, AWS would cost a fortune. I have next to zero experience with server administration, but 384GB seems like a lot to me. Is that common for production servers for popular web services? Do you need a customized OS to address that much memory? Seems like you'd really need to beef up the cache hierarchy make 0.38TB of RAM fast.

Linux (SuSE enterprise something, possibly with a custom kernel, I’m just a user, not admin) runs just fine on 5.2TB machine, though I can easily imagine you’d have problems if you tried it with, say, DOS :) The system obviously behaves a bit different from a standard desktop machines, e.g. different areas of RAM are differently fast, depending on the core on which your current process runs, you need to disable indiv…

> e.g. different areas of RAM are differently fast, depending on the core on which your current process runs, you need to disable individual CPU lines displayed in top etc.

this is NUMA, in case anyone 'new to sysadmin' or to architecture is trying to Google this

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#168

"One problem is not many tests. Tests aren’t needed because there’s a great community... If users find any problems with it they report the bugs that they’ve found." I'm often surprised at the paucity of test-coverage in relatively large companies. http://nerds.airbnb.com/testing-at-airbnb/

I think we're going trough a thesis/anti-thesis cycle on tests - in the beginning, there was militant testing, 100% coverage, testing getters and setters etc (as well as more complex stuff, obviously). Then some people started coming around to the idea that there are actually large swathes of code that is simple enough that testing doesn't actually add much value especially compared to the effort of writing them, then that probably got a bit out of hand (to what you're referring to). Maybe the pendulum will swing back and we'll find a good heuristic for just how much testing is the right amount that isn't all or nothing?

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#169
post #34
post #30

Earlier quoted context omitted.

That bullet point surprised me. It comes right after "110K lines of code. A small number given what it does"; to me this reads like they didn't write tests because that would add complexity. Can any SO devs give us more details on "not many tests"? Or how many bug reports get filed vs rate of change of software?

The things that obviously should have tests have tests. That means most of the things that touch money on our Careers product, and easily unit-testable features on the Core end (things with known inputs, e.g. flagging, our new top bar, etc), for most other things we just do a functionality test by hand and push it to our incubating site (formerly meta.stackoverflow, now meta.stackexchange). You can look at reported b…

You should come down our way and do a tech talk about it.

If only for the wailing, gnashing of teeth and rending of garments.

Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers

#170

"SO goes to great lengths to reduce garbage collection costs, skipping practices like TDD, avoiding layers of abstraction, and using static methods." I don't understand this at all. What does TDD have to do with reducing garbage collection?

My guess is that they feel that the layers of indirection and abstraction often needed to make TDD work result in an object creation pattern that results in heavy GC load during normal operation. The references to "using static methods" is probably related to this. ps. That's my guess, but I'd encourage you to post your question to the meta site for SO.

Insofar as this is your guess, it's down to the language. In Ruby or Javascript it's very easy to TDD. In Java, much less so. (I'm not sure where C# lies on the spectrum).
Post reply on HN