Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stack Overflow is a cacheless, 9-server on-prem monolith

twitter.com

Post reply on HN
Print view View original

111–120 of 120 posts

Re: Stack Overflow is a cacheless, 9-server on-prem monolith

#111

Post by Nextgrid »

kichik wrote: post #56
Spooky23 wrote: post #49

Earlier quoted context omitted.

That’s an engineering choice not cloud vs. cloud. How many services are down when AWS us-east has a problem?

True. But cloud makes it a lot easier. In some cases it's built-in, like S3. In others it's a checkbox like RDS Multi-AZ. And if you need to roll your own, multi-AZ or even multi-region is much more straightforward than renting another rack somewhere. I have personally seen Stack Overflow be "under maintenance" or straight up down a lot more than I have seen entire us-east-1 down.

Keep in mind that the "cloud" relies on an opaque control plane with undocumented failure modes (that sometimes even the provider does not know).

Just because you tick a checkbox doesn't mean it'll actually work as planned, and unlike infrastructure within your control that you can actually test (pull the network or power cable from a live server if you need to), you can't simulate a cloud provider outage.

> multi-AZ or even multi-region is much more straightforward than renting another rack somewhere.

Assuming that enough of the AWS control plane is alive to actually allow you to login and administer the services in your backup region.

Furthermore, cloud providers are their own businesses and are constantly in motion (introducing new features, etc). That's good for their business but bad for yours, as it means they might be doing risky changes that could affect you should it go wrong.

Re: Stack Overflow is a cacheless, 9-server on-prem monolith

#112

Post by Yeroc »

ctvo wrote: post #85

The folks over at SO picked a stack (C#, SQL Server, IIS), and optimized the heck out of it to keep this "simplicity". Much of SO is custom built from the ground up to push performance and stay within the purity of the canonical .net stack. It isn't clear to me this is a model that would work elsewhere, or should be held up as something to be replicated. Did they save time? Did they save money? Did this help make SO…

It's worth reminding people what is actually possible with a relatively simple architecture. There's a vast number of websites and services with a very small fraction of the traffic of Stack Overflow with a much more complicated architecture simply because everyone thinks you need Kubernetes etc to scale out.

Re: Stack Overflow is a cacheless, 9-server on-prem monolith

#113

Post by Spooky23 »

Nextgrid wrote: post #111
kichik wrote: post #56

Earlier quoted context omitted.

True. But cloud makes it a lot easier. In some cases it's built-in, like S3. In others it's a checkbox like RDS Multi-AZ. And if you need to roll your own, multi-AZ or even multi-region is much more straightforward than renting another rack somewhere. I have personally seen Stack Overflow be "under maintenance" or straight up down a lot more than I have seen entire us-east-1 down.

Keep in mind that the "cloud" relies on an opaque control plane with undocumented failure modes (that sometimes even the provider does not know). Just because you tick a checkbox doesn't mean it'll actually work as planned, and unlike infrastructure within your control that you can actually test (pull the network or power cable from a live server if you need to), you can't simulate a cloud provider outage. > multi-AZ…

Exactly. I run a large enterprise service in a single datacenter with 5 years of 100% uptime. Our design goal is 99.97% measured monthly.

We have that because we have complete control end to end. We made an engineering decision not to have geo-redundancy because many of the dependent services aren’t available that way either.

Because of the compute requirements, running that service in AWS or GCP would cost about 80% more, inclusive of all costs (equipment, labor, utilities, etc)

Re: Stack Overflow is a cacheless, 9-server on-prem monolith

#114

Post by inhumantsar »

Yeroc wrote: post #112
ctvo wrote: post #85

The folks over at SO picked a stack (C#, SQL Server, IIS), and optimized the heck out of it to keep this "simplicity". Much of SO is custom built from the ground up to push performance and stay within the purity of the canonical .net stack. It isn't clear to me this is a model that would work elsewhere, or should be held up as something to be replicated. Did they save time? Did they save money? Did this help make SO…

It's worth reminding people what is actually possible with a relatively simple architecture. There's a vast number of websites and services with a very small fraction of the traffic of Stack Overflow with a much more complicated architecture simply because everyone thinks you need Kubernetes etc to scale out.

That's the point though. If you want to focus your engineering time on optimization and code quality, then of course you can scale to SO's size with 9 servers and a simple architecture.

If you're still growing and more interested in delivering tons of features quickly, and/or don't have the ability to attract world leading talent, then a more complicated architecture with clear boundaries is often a better call than delivering relatively few features with obsessive rigor in a monolithic codebase.

Re: Stack Overflow is a cacheless, 9-server on-prem monolith

#116

Post by Yeroc »

inhumantsar wrote: post #114
Yeroc wrote: post #112

Earlier quoted context omitted.

It's worth reminding people what is actually possible with a relatively simple architecture. There's a vast number of websites and services with a very small fraction of the traffic of Stack Overflow with a much more complicated architecture simply because everyone thinks you need Kubernetes etc to scale out.

That's the point though. If you want to focus your engineering time on optimization and code quality, then of course you can scale to SO's size with 9 servers and a simple architecture. If you're still growing and more interested in delivering tons of features quickly, and/or don't have the ability to attract world leading talent, then a more complicated architecture with clear boundaries is often a better call than…

You'll only need the expertise to perform in-depth optimization if you're scaling to the level of Stack Overflow though. For the vast majority of sites it won't be a concern. The simpler architecture should be the default because local method calls are easily 2 orders of magnitude faster than a network roundtrip (yes I know that's not the whole story). I'm not sure how or why creating clear boundaries without the RPC crutch suddenly became insurmountable.

Re: Stack Overflow is a cacheless, 9-server on-prem monolith

#117

Post by stby »

infomaniac wrote: post #71
Fire-Dragon-DoL wrote: post #69

Isn't stackoverflow, incidentally, one of the websites who would benefit the most from caching, given their content supposedly is going to be static the majority of the time?

This is addressed in one of the linked tweets.

This one: https://twitter.com/sahnlam/status/1629713961951330304?s=20

Re: Stack Overflow is a cacheless, 9-server on-prem monolith

#118

Post by bluedino »

Not that long ago (2016) they had:

  Servers:

  SQL Servers (Stack Overflow Cluster)
   2 Dell R720xd Servers
  SQL Servers (Stack Exchange “…and everything else” Cluster)
   2 Dell R730xd Servers, each with:
  Web Servers
   11 Dell R630 Servers
  Service Servers (Workers)
   2 Dell R630 Servers
   1 Dell R620 Server
  Elasticsearch Servers (Search)
   3 Dell R620 Servers
  HAProxy Servers (Load Balancers)
   2 Dell R620 Servers
  Redis Servers (Cache)
   2 Dell R630 Servers
  VM Servers (VMWare, Currently)
   2 Dell FX2s Blade Chassis, each with 2 of 4 blades populated
   4 Dell FC630 Blade Servers (2 per chassis)
   2 Equalogic SAN PS6000-series
  Machine Learning Servers (Providence)
   2 Dell R620 Servers
  Machine Learning Redis Servers (Still Providence)
   3 Dell R720xd Servers
  LogStash Servers
   6 Dell R720xd Servers
  HTTP Logging SQL Server
   1 Dell R730xd 
  Development SQL Server
   1 Dell R620 

  Network:

  2x Cisco Nexus 5596UP core switches (96 SFP+ ports each)
  10x Cisco Nexus 2232TM Fabric Extenders (2 per rack)
  2x Fortinet 800C Firewalls
  2x Cisco ASR-1001 Routers
  2x Cisco ASR-1001-x Routers
  6x Cisco 2960S-48TS-L Management network switches (1 Per Rack)
https://nickcraver.com/blog/2016/03/29/stack-overflow-the-ha...

Re: Stack Overflow is a cacheless, 9-server on-prem monolith

#119

Post by tcgv »

jameshart wrote: post #44
atonse wrote: post #14

Even though I love their simplicity as an example of how to be pragmatic and not over-engineer, do remember that they’ve tuned their code to the point that they built an ORM that is one of the fastest in the NET world. I used it and it was awesomely lightweight. It’s as much an example of how far world class talent can go, as it is about doing more with less.

Right - Marc Gravell and Tim Craver, who worked on the core architecture of Stack Overflow, were both so obsessive about extracting performance from .net web applications that when they couldn’t do any more from the outside, they both quit and went to work for Microsoft on performance improvements in the framework itself. I feel like it’s similar to how people point to Craigslist as evidence that you can still build…

> Running highly scalable monoliths is easy! As long as you’re willing to hire some of the five to ten people in the world who are capable of advancing the state of the art of development on that technology stack…

I truly believe that being able to design and run a modular monolith application effectively (not talking about the 'hyperscale' scenario here) should be a prerequisite for designing and running a set of interconnected microsservices. The challenge is similar, but dealing with modular monoliths has the advantage of not having to deal with the uncertainty of networking programming (i.e. remote calls, network error handling, distributed transactions).

Re: Stack Overflow is a cacheless, 9-server on-prem monolith

#120

Post by edude03 »

MonkeyMalarky wrote: post #41
threeseed wrote: post #34

Earlier quoted context omitted.

Microservices became a synonym for Services Orientated Architecture years ago. It's almost always relatively normal sized services split by functional area e.g. Auth, Cache etc.

Hehehe. This is absolutely true and trying to explain it to anyone around me makes me feel like a Cassandra quoting battlestar galactica: "All of this has happened before. All of this will happen again". I'm not even old, I just happened to witness the tail end of SOA when I was still a student, followed by the rise of micro-services afterwards. Your service mesh is just someone else's message bus. Sure, it's differe…

> service mesh is just someone else's message bus

I don't think you mean service mesh here, or at least not Istio/Linkerd style service mesh

Post reply on HN