Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stack Exchange's monitoring system is now open source

github.com

Post reply on HN
Print view View original

41–50 of 57 posts

Re: Stack Exchange's monitoring system is now open source

#41

Post by EnderMB »

Styck wrote: post #35
stevepotter wrote: post #29

The anti .net kneejerk reactions on HN really disturbs me. You spend all day on stackoverflow then blindly bash their tech stack. Developing, deploying, and hosting .net apps is just fine. Many brilliant people choose .net and are plenty happy with it. Maybe rather than jumping to conclusions, you could give it a shot.

Many of us have .NET experience, really like C# but have sensible reasons for not recommending it. A Microsoft only stack is nowhere near as flexible as Linux and if you were to reach Stack Overflow's levels of traffic you'll not be able to avoid the need to run a part of your infrastructure on Linux. I seriously doubt Stack Overflow is running Redis, ElasticSearch & HAProxy* on Windows. Unless you already have a tea…

I imagine that 99% of the users on here won't work on any projects that receive the same levels of traffic that Stack Overflow will get, and as such the worries of scaling to this level aren't really an issue.

For the vast majority of websites that developers will build, the .NET stack is absolutely fine. My only gripe with being a .NET developer is that if your professional experience is limited to a Windows based stack you may find it hard to move over to a Unix-based stack.

Re: Stack Exchange's monitoring system is now open source

#42

Post by Nick-Craver »

jamesRaybould wrote: post #5

Having a quick poke around and I notice that they are storing all exceptions in an SQL database. I've been looking at storing all the errors we get in our various applications in a central repository and was wondering what the general consensus was? Currently I'm going a centralised logstash server and using a logstash shipper on each of my servers to push the exceptions, from a standard logfile to it. I was toying w…

We indeed use SQL for errors (though Exceptional has a couple of stores...and you can add a new one). The reason a SQL outage isn't an issue is a) we have other alarms for that, and b) Exceptional will fall back to an in-memory exception queue and flush to the DB when it's available again.

In the event of a store loss (file share, SQL server...whatever your store is) then it queues exceptions in memory with rollups to reduce memory usage, and will flush to the store when it's available again. It does a connectivity test every 2 seconds in the event of failure.

Exceptional is open source and is the basis for what's used in Opserver...the UI is even very, very similar it's just that Opserver has more features for a multi-application view. You can see the source here: https://github.com/NickCraver/StackExchange.Exceptional

Re: Stack Exchange's monitoring system is now open source

#43

Post by Nick-Craver »

Styck wrote: post #35
stevepotter wrote: post #29

The anti .net kneejerk reactions on HN really disturbs me. You spend all day on stackoverflow then blindly bash their tech stack. Developing, deploying, and hosting .net apps is just fine. Many brilliant people choose .net and are plenty happy with it. Maybe rather than jumping to conclusions, you could give it a shot.

Many of us have .NET experience, really like C# but have sensible reasons for not recommending it. A Microsoft only stack is nowhere near as flexible as Linux and if you were to reach Stack Overflow's levels of traffic you'll not be able to avoid the need to run a part of your infrastructure on Linux. I seriously doubt Stack Overflow is running Redis, ElasticSearch & HAProxy* on Windows. Unless you already have a tea…

Indeed we run all of these things (and more) on linux. We use the most appropriate architecture we can come up with at the time. When a better overall option appears, we do that. New information and technology makes our decisions change, that's how brains are supposed to work, I think.

We use linux for: redis, elasticsearch, HAProxy, DNS (bind), nginx, mail (exim) apache, wordpress, mysql, nexpose, backups, puppet, asterix, android builds and our internal mercurial.

As Opserver grows we will be monitoring Windows and Linux with our solution, but simple to setup via polling, or more advanced monitoring via an agent (puppet and DSC configurable/installable). We plan to have agents for both Windows and Linux open sourced, both using a standard communication format so that anyone can write additional agents, or add to them, or...whatever really. We haven't started building this yet, a complete monitoring solution is what we'll be working on over the next 6-12 months. It will be in the open as we go, with lots of internal dogfooding to prove things out.

Re: Stack Exchange's monitoring system is now open source

#44

Post by emillon »

Nick-Craver wrote: post #36
emillon wrote: post #34

Earlier quoted context omitted.

I think that the commit history says a lot about the development process and the company culture. First, is that it's interesting to see how these companies work internally. Second, it's easier to dive into the code by reading the latest commits. You can see what's going on and what pieces are changed together. Finally, there is a substantial difference between: - a repository that's handled as a first class "source…

Guy who wrote it here...the only reason this isn't public is security, at a few points there were various passwords in the repository. Also, the internal repository is Hg (still is, though this may change). I still want to dogfood major changes before breaking others. We could convert this history, sure...but the security reason remains. There were many commits that don't make a lot of sense to anyone but me probably…

Thanks!

Re: Stack Exchange's monitoring system is now open source

#45

Post by viraptor »

beck5 wrote: post #12

Could someone please list the equivalent systems this is could replace? (linux or windows)

I see parts of cacti, nagios/icinga + graphing, munin, zabbix, sentry (exceptions/errors), shinken, etc. There are many similar projects, but nothing so much integrated as far as I know.

Re: Stack Exchange's monitoring system is now open source

#46

Post by lmm »

stevepotter wrote: post #29

The anti .net kneejerk reactions on HN really disturbs me. You spend all day on stackoverflow then blindly bash their tech stack. Developing, deploying, and hosting .net apps is just fine. Many brilliant people choose .net and are plenty happy with it. Maybe rather than jumping to conclusions, you could give it a shot.

Stack Overflow (which I avoid as much as I can fwiw) is not a strong argument for the .net stack given the people behind it have moved to a different stack for their next project (http://www.codinghorror.com/blog/2013/03/why-ruby.html). That's not an isolated incident.

I've several friends working in that stack. While many defend it as the correct decision at the time (and I don't disagree with that), all would rather be using something else now.

Re: Stack Exchange's monitoring system is now open source

#47

Post by spion »

stevepotter wrote: post #29

The anti .net kneejerk reactions on HN really disturbs me. You spend all day on stackoverflow then blindly bash their tech stack. Developing, deploying, and hosting .net apps is just fine. Many brilliant people choose .net and are plenty happy with it. Maybe rather than jumping to conclusions, you could give it a shot.

A problem with the Windows tech stack is the culture of automation (or lack thereof). Sure there is PowerShell, but often times critical tools insist on giving you a GUI instead of providing something that can be incorporated into a script. And even when they do offer scriptability, its much less documented than the GUI.

Re: Stack Exchange's monitoring system is now open source

#49

Post by Nick-Craver »

lmm wrote: post #46
stevepotter wrote: post #29

The anti .net kneejerk reactions on HN really disturbs me. You spend all day on stackoverflow then blindly bash their tech stack. Developing, deploying, and hosting .net apps is just fine. Many brilliant people choose .net and are plenty happy with it. Maybe rather than jumping to conclusions, you could give it a shot.

Stack Overflow (which I avoid as much as I can fwiw) is not a strong argument for the .net stack given the people behind it have moved to a different stack for their next project ( http://www.codinghorror.com/blog/2013/03/why-ruby.html ). That's not an isolated incident. I've several friends working in that stack. While many defend it as the correct decision at the time (and I don't disagree with that), all would rat…

Most of the people behind Stack Overflow are still here...and we haven't changed our stack and are quite happy with the performance we get...and there's always more to squeeze out.

We can and sometimes do run Stack Overflow (currently 3.3 billions hit a month) from 2 web servers and 1 SQL server...I think that's pretty good for any stack.

Re: Stack Exchange's monitoring system is now open source

#50

Post by lmm »

Nick-Craver wrote: post #49
lmm wrote: post #46

Earlier quoted context omitted.

Stack Overflow (which I avoid as much as I can fwiw) is not a strong argument for the .net stack given the people behind it have moved to a different stack for their next project ( http://www.codinghorror.com/blog/2013/03/why-ruby.html ). That's not an isolated incident. I've several friends working in that stack. While many defend it as the correct decision at the time (and I don't disagree with that), all would rat…

Most of the people behind Stack Overflow are still here...and we haven't changed our stack and are quite happy with the performance we get...and there's always more to squeeze out. We can and sometimes do run Stack Overflow (currently 3.3 billions hit a month) from 2 web servers and 1 SQL server...I think that's pretty good for any stack.

Performance isn't the reason I see people wanting to move away from that stack; it's more about library availability, tool ecosystem (including things like monitoring, so this is an improvement - though I doubt it will stem the tide), and language productivity.

(Not that that your performance sounds like a compelling advantage for that stack. That's what, an average of 700 requests/second/web server? So the peak is probably around 1500? Pretty good indeed, but not outstandingly so - where I worked 3 years ago our system handled peaks of 600 requests/second/web server on a JVM-based stack. And I don't see .net topping the charts on http://www.techempower.com/benchmarks/).

Post reply on HN