Earlier quoted context omitted.
I wrote a Dapper extension for working with SQL Server's geospatial queries and types a couple of years ago - have they added anything like that yet? Otherwise I'd be happy to add it.
Marc Gravell added in the ability to put in pretty much any custom type without adding any dependency weight about a month ago. You can see the commit here: https://github.com/StackExchange/dapper-dot-net/commit/e26ee... Look towards the end at the tests for example of how to hook up a custom type (it's pretty simple).
StackOverflow Update: 560M Pageviews a Month, 25 Servers
161–170 of 278 posts
Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers
#162Earlier quoted context omitted.
Don't forget to factor in the costs of running, cooling, and maintaining that much hardware (also bandwidth costs). I'm not saying it's $2k/month, I'm just saying don't compare the physical box and AWS without factoring everything in. At the end of the day, if you don't need the support, you're probably right that a dedicated box makes sense there.
AWS has among the highest bandwidth costs you're going to run into, if we're talking co-location, dedicated, or self-hosting. Their bandwidth costs are the sole reason I won't go anywhere near AWS yet, I consider their prices outrageously high. 10tb = roughly $1,000 per month They're between 5 and 50 times more expensive on bandwidth than the options in dedicated / colo / self hosting.
Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers
#163> With their SQL Servers loaded with 384 GB of RAM and 2TB of SSD, AWS would cost a fortune. I have next to zero experience with server administration, but 384GB seems like a lot to me. Is that common for production servers for popular web services? Do you need a customized OS to address that much memory? Seems like you'd really need to beef up the cache hierarchy make 0.38TB of RAM fast.
No, that's no common for production servers for popular web services, though- they tend to shard across other dimensions because the web services are often CPU or network-bound.
Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers
#164I've recently started using their micro-ORM Dapper, and I like it a lot. I get the performance of hand-coded SQL, but without the tedious mapping from SqlDataReader to my entity.
Dapper is nice, but if you want to have the compiler check your queries, and still run fast, I recommend LINQ to DB: https://github.com/linq2db/linq2db (not to be confused with LINQ to SQL from Microsoft)
A huge problem with the tradeoff we had (well, still have in some areas) is the generated SQL is nasty, and finding the original code it came from is often non-trivial. Lack of ability to hint queries, control parameterization, etc. is also a big issue when trying to optimize queries. For example we (and by "we" I mean, "I made Marc Gravell do it") added literal replacement to Dapper to help with query parameterization which allows you to use things like filtered indexes. In dapper we also intercept the SQL calls to dapper and add add exactly where it came from. Here's what that looks like, replicate for the other methods:
public static IEnumerable Query(this DataContext db, SqlBuilder.Template template, bool buffered = true, int? commandTimeout = null, IDbTransaction transaction = null, [CallerFilePath]string fromFile = null, [CallerLineNumber]int onLine = 0, string comment = null)
{
using (db.Connection.EnsureOpen())
{
return SqlMapper.Query(db.Connection, MarkSqlString(template.RawSql, fromFile, onLine, comment), template.Parameters as object, transaction ?? db.Transaction, buffered, commandTimeout);
}
}
And here's the our marking method tailored for our code, but you get the idea: private static string MarkSqlString(string sql, string path, int lineNumber, string comment)
{
if (path.IsNullOrEmpty() || lineNumber == 0)
{
return sql;
}
var commentWrap = " ";
var i = sql.IndexOf(Environment.NewLine);
// if we didn't find \n, or it was the very end, go to the first space method
if (i
This results in a comment at the top of the query like this: /* Models\Post.LinkedQuestions.cs@105 */
select top (@top)
p.Id
, p.Title
, p.Score
It's such a simple approach using the caller member attributes, but it saves so much time tracking things down.Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers
#165This is the fascinating part to me, their SSD have not failed: Failures have not been a problem, even with hundreds of intel 2.5" SSDs in production, a single one hasn’t failed yet. One or more spare parts are kept for each model, but multiple drive failure hasn't been a concern.
That said, all the SSDs I've had have taken repeated pounding with no complaints. I accidentally swapped a few terabytes to swap on an SSD, and was simply surprised that the job I was doing finished faster.
Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers
#166Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers
#167> With their SQL Servers loaded with 384 GB of RAM and 2TB of SSD, AWS would cost a fortune. I have next to zero experience with server administration, but 384GB seems like a lot to me. Is that common for production servers for popular web services? Do you need a customized OS to address that much memory? Seems like you'd really need to beef up the cache hierarchy make 0.38TB of RAM fast.
Linux (SuSE enterprise something, possibly with a custom kernel, I’m just a user, not admin) runs just fine on 5.2TB machine, though I can easily imagine you’d have problems if you tried it with, say, DOS :) The system obviously behaves a bit different from a standard desktop machines, e.g. different areas of RAM are differently fast, depending on the core on which your current process runs, you need to disable indiv…
this is NUMA, in case anyone 'new to sysadmin' or to architecture is trying to Google this
Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers
#168"One problem is not many tests. Tests aren’t needed because there’s a great community... If users find any problems with it they report the bugs that they’ve found." I'm often surprised at the paucity of test-coverage in relatively large companies. http://nerds.airbnb.com/testing-at-airbnb/
Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers
#169Earlier quoted context omitted.
That bullet point surprised me. It comes right after "110K lines of code. A small number given what it does"; to me this reads like they didn't write tests because that would add complexity. Can any SO devs give us more details on "not many tests"? Or how many bug reports get filed vs rate of change of software?
The things that obviously should have tests have tests. That means most of the things that touch money on our Careers product, and easily unit-testable features on the Core end (things with known inputs, e.g. flagging, our new top bar, etc), for most other things we just do a functionality test by hand and push it to our incubating site (formerly meta.stackoverflow, now meta.stackexchange). You can look at reported b…
If only for the wailing, gnashing of teeth and rending of garments.
Re: StackOverflow Update: 560M Pageviews a Month, 25 Servers
#170"SO goes to great lengths to reduce garbage collection costs, skipping practices like TDD, avoiding layers of abstraction, and using static methods." I don't understand this at all. What does TDD have to do with reducing garbage collection?
My guess is that they feel that the layers of indirection and abstraction often needed to make TDD work result in an object creation pattern that results in heavy GC load during normal operation. The references to "using static methods" is probably related to this. ps. That's my guess, but I'd encourage you to post your question to the meta site for SO.