Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The architecture of Stack Overflow [video]

dev-metal.com

Post reply on HN
Print view View original

21–30 of 49 posts

Re: The architecture of Stack Overflow [video]

#21

Post by edwinnathaniel »

vladimirralev wrote: post #19
merak136 wrote: post #9

Some points that I find interesting: [1] StackOverflow has VERY FEW tests. He says that StackOverflow doesn't use many unit tests because of their active community and heavy usage of static code. [2] Most StackOverflow employees work remotely. This is very different than a lot of companies that are now trying to force employees back into an office. [3] Heavy usage of Static classes and methods. His main argument is t…

I see more and more static methods and classes last 2 years maybe. It's probably more about the stateless design and less side effects, but it definitely helps garbage collection if you avoid classes at session scope or smaller. In OOP there is another pattern that helps - object pools, but it's a lot of work to get it to work correctly and it's not as efficient.

If most of your classes are small, I don't see why people have to resort to static methods.

If your methods are static, there are tendency/lust to use static member variables (hence stateful) which will cause side effects.

Don't forget the following points too:

1) You still have pooled objects somewhere (stateless business logic classes like XYZServices, repository classes that may be backed by pooled DB connections and Transaction Managers) provided/managed by your Application Server or by 3rd-party framework (Spring does this).

2) Your Application Server tend to have beefy hardware, good enough not to care of GC hiccups.

There are other reasons to use static methods but I don't think they're strong enough in this case.

Re: The architecture of Stack Overflow [video]

#22

Post by kmontrose »

zero1zero wrote: post #20
y0ghur7_xxx wrote: post #11

I would love to know more about the Databases: - Are they used for different things on the sites? - Is data partitioned across tables? - Are they all SQL Server instances?

I would like to know more about this as well. It sounds like they are all SQL Server instances. However, he made it seem like they are reproducing the schema once per site? I.e., a separate database per site rather than sharding the shared data to multiple hosts per site. Did I hear this right in the question/answer portion?

Stack Exchange has one database per-site, so Stack Overflow gets on, Super User gets one, Server Fault gets one, and so on. The schema for these is the same.

There are a few wrinkles. There is one "network wide" database which has things like login credentials, and aggregated data (mostly exposed through stackexchange.com user profiles, or APIs). Careers Stack Overflow, stackexchange.com, and Area 51 all have their own unique database schema.

All databases are MS SQL Server.

Re: The architecture of Stack Overflow [video]

#23

Post by vladimirralev »

edwinnathaniel wrote: post #21
vladimirralev wrote: post #19

Earlier quoted context omitted.

I see more and more static methods and classes last 2 years maybe. It's probably more about the stateless design and less side effects, but it definitely helps garbage collection if you avoid classes at session scope or smaller. In OOP there is another pattern that helps - object pools, but it's a lot of work to get it to work correctly and it's not as efficient.

If most of your classes are small, I don't see why people have to resort to static methods. If your methods are static, there are tendency/lust to use static member variables (hence stateful) which will cause side effects. Don't forget the following points too: 1) You still have pooled objects somewhere (stateless business logic classes like XYZServices, repository classes that may be backed by pooled DB connections…

Small classes are actually worse GC-wise. Because they will fill up the GC graph with many small nodes as opposed to fewer large nodes, which are released in bulk with little fragmentation. Small and large nodes have the same GC overhead essentially. In general you want your objects to be large. When they are small, once the GC realizes what's going on (usually at some high threshold 90% or so), it will have to run a some O(n^x) graph reduction algorithm or defragmenataion. Special tuning is required for such cases. Beefy hardware doesn't help in many cases due to locks. There are very few production-ready lock-less GCs.

Re: The architecture of Stack Overflow [video]

#25

Post by skittles »

He mentioned that they use the servicestack.text library. I've looked into servicestack recently (using the nuget packages), but then found the library to be pay-to-play. There's an older version (v3) that is BSD licensed that is being maintained. Do any of you have experience with it? I have grown tired of Microsoft pushing new solutions to the same problem (REST service with WCF and then Asp.net web api).

Re: The architecture of Stack Overflow [video]

#26

Post by JasonPunyon »

merak136 wrote: post #9

Some points that I find interesting: [1] StackOverflow has VERY FEW tests. He says that StackOverflow doesn't use many unit tests because of their active community and heavy usage of static code. [2] Most StackOverflow employees work remotely. This is very different than a lot of companies that are now trying to force employees back into an office. [3] Heavy usage of Static classes and methods. His main argument is t…

It's not that no one follows the norms or tests. On the Careers team we do much more automated testing because there's money and literally people's jobs at stake. We have unit tests, integration tests and UI tests that all run on every push. All the tests must succeed before a production build run is even possible.

Re: The architecture of Stack Overflow [video]

#27

Post by edwinnathaniel »

vladimirralev wrote: post #23
edwinnathaniel wrote: post #21

Earlier quoted context omitted.

If most of your classes are small, I don't see why people have to resort to static methods. If your methods are static, there are tendency/lust to use static member variables (hence stateful) which will cause side effects. Don't forget the following points too: 1) You still have pooled objects somewhere (stateless business logic classes like XYZServices, repository classes that may be backed by pooled DB connections…

Small classes are actually worse GC-wise. Because they will fill up the GC graph with many small nodes as opposed to fewer large nodes, which are released in bulk with little fragmentation. Small and large nodes have the same GC overhead essentially. In general you want your objects to be large. When they are small, once the GC realizes what's going on (usually at some high threshold 90% or so), it will have to run a…

We're talking in the context of stateless Request Response of the Web-Application nature here.

When a Request comes in, the App-Server will allocate (or use from the pool) a thread to serve that Request (in .NET/JVM world, Ruby/Python uses Processes unless you use different App Server).

If you create small objects within the scope of that Request (which usually lives inside a method) and that objects are contained and don't hold references to any long-lived objects, they will be GC-ed quickly (and potentially way quicker) once the method is finished.

Thread is GC-ed as well once it's finished (unless you wish to release them back to the 'unused' pool).

My feeling is that their use of static methods have nothing to do at all with GC.

Re: The architecture of Stack Overflow [video]

#28

Post by welegan »

sklivvz1971 wrote: post #18
BrandonY wrote: post #14

Earlier quoted context omitted.

User community as testers presents some interesting pros and cons. Pros: * Tests are self-updating. Add a new feature: tests come in for free. Change a feature: tests automatically update. Fail to document a change: tests fail. * Tests are unusually thorough * Eventually consistent testing. If nobody ever complains, it probably wasn't a bug worth fixing. Cons: * Tests cannot be run offline. Feature must be committed…

Remember that our community writes bug reports but also vets bug reports. We rarely have to deal with bad reports. Interestingly, large quantities of false negatives are a non-issue.

Presumably the same reason why they don't have a ton of bad questions on stack overflow: their community scoring would apply just as much to bug reports

Re: The architecture of Stack Overflow [video]

#29

Post by sklivvz1971 »

skittles wrote: post #25

He mentioned that they use the servicestack.text library. I've looked into servicestack recently (using the nuget packages), but then found the library to be pay-to-play. There's an older version (v3) that is BSD licensed that is being maintained. Do any of you have experience with it? I have grown tired of Microsoft pushing new solutions to the same problem (REST service with WCF and then Asp.net web api).

We used it at the time I gave that talk, we don't anymore. We only used JSON serialization and we have rolled out our own free solution, Jil.

https://github.com/kevin-montrose/Jil

Post reply on HN