Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sentry.io outage

status.sentry.io

Post reply on HN
Print view View original

41–50 of 84 posts

Re: Sentry.io outage

#41

Post by luismedel »

the_gipsy wrote: post #28
rtpg wrote: post #22

Earlier quoted context omitted.

It’s likely that the failure is around Sentry’s “signal a release and upload your source code out of band to us” feature. In theory things shouldn’t break but if your CI process involves signaling a third party, it can fail from that (hopefully you can temporarily disable it or something of the sort)

Yes the CI should fail, or else you would be deploying something that can't match your errors to your source code (via the sourcemaps which is what presumably gets uploaded to sentry on the CI run).

Take in advance that I don't know Sentry, nor webpack et al but, in my mind, I'd like to have an opportunity to continue my build process with a big, red warning.

I prefer a 98% accurate error reporting better than a 100% one I can't push to production.

Or maybe I'm not understanding well the value Sentry offers, of course.

Re: Sentry.io outage

#42

Post by kenrick95 »

drej wrote: post #38
Klathmon wrote: post #7

Earlier quoted context omitted.

Has anyone here used their self hosted version? I always assume that running an on-prem version of a SAAS offering is going to be a shit show, but I'm curious if anyone here uses it?

We used to run it self hosted, but the architecture is rather complex, so we ended up switching to their SaaS offering. I tried advocating [0] for a leaner architecture for simpler setups + to run integration tests against... but it wasn't met with much enthusiasm. I imagine 99% of installations (and certainly CI pipelines) would be fine with their wsgi webserver and sqlite instead of Clickhouse, Relay, memcached, ng…

This has been the experience at my company as well. The previous version of Sentry Server was okay-ish to self-host, but the newest version requires setting up some more services that our infra unwilling to setup/maintain as they are different from the tech stack that our devs use in the company. We ended up with SaaS too

Re: Sentry.io outage

#43

Post by AbacusAvenger »

Klathmon wrote: post #7
dijit wrote: post #6

Sentry is a great product, I love even more that you can self-host it; otherwise I would be one of the people denigrating everyone for running their entire operation on SaaS products. As it stands if their reliability leaves something to be desired: run your own. These things happen, they’re more common the more complicated or huge your setup. Good luck to them in getting it resolved.

Has anyone here used their self hosted version? I always assume that running an on-prem version of a SAAS offering is going to be a shit show, but I'm curious if anyone here uses it?

I've been running it for almost a year, using docker-compose. It seems rather unstable for me. Every week or so, Sentry will just stop handling incoming events, and the request queues just keep growing. And if you try to upload symbols in this state, it will pause forever when trying to process them. So I've got a script[0] I can run that will unbreak it, but I don't fully understand what it actually does (I cobbled that script together based on a GitHub issue[1] that described the same problem). The Sentry architecture is complicated and not trivial to debug.

When it works, it's pretty good. I'm using it with sentry-native on the application side, which uses Crashpad to capture stack traces of native binaries (x86/ARM, Windows/Mac/Linux, whatever). It often doesn't deduplicate events properly, and the stack trace qualities vary dramatically by platform. Sometimes the stack traces it provides are total nonsense, but it does allow downloading the minidump files, so I can dig at them in Visual Studio and see what's really going on. I have discovered and solved many real bugs using it, so I've put up with the frustrating stability issues.

[0] https://gist.github.com/tycho/4279ce2ca47b293a85696695968263...

[1] https://github.com/getsentry/self-hosted/issues/478

Re: Sentry.io outage

#44

Post by sofixa »

Nextgrid wrote: post #27
Cthulhu_ wrote: post #15

Earlier quoted context omitted.

> run your own. Is there a good name for this theory that self-hosting something is more reliable? I mean yeah, if you have an outage it only affects you and your customers so I guess it's an improvement in that regard, but thinking you're better at running software than a SaaS is hubris.

> but thinking you're better at running software than a SaaS is hubris. Disagreed on several points. First off, if you're using something like Sentry, chances are that building and running software is your job so it should not be a problem. If self-hosting Sentry is a problem I would start doubting the skills of your tech team. Second, running a service for your own use is very different to running a service that has…

> First off, if you're using something like Sentry, chances are that building and running software is your job so it should not be a problem. If self-hosting Sentry is a problem I would start doubting the skills of your tech team

Sentry is a pretty complex piece of software with a lot of moving parts. They have good orchestration around it, either Helm charts for Kube deployments or a giant bash script managing docker-compose for mono-machine, and it just works. However understanding either of them, and being capable of debugging aren't skills i expect your average developer to have. It's more for SRE/Infra/Platform/etc. folk. Like i don't expect the average developer to be capable of debugging Kubernetes on their own.

Of course, many of them absolutely can, but many more would prefer that to people who do that for a living. The type of developer using Heroku and other PaaSes precisely to avoid getting their hands too deep into infrastructure stuff.

Re: Sentry.io outage

#45

Post by sofixa »

tehbeard wrote: post #35
ohnoesjmr wrote: post #32

Earlier quoted context omitted.

I run it on k8s for a medium sized company, as running via compose didn't seem to scale well for us. I even upgraded from 9.11 which ran a handful of containers to the most recent version which runs 15+ containers. You get used to the quirks of running it and then it just works, but it did take some time polishing the health checks etc to get it HA and scalable. None of the helm charts I tried actually worked or prov…

Pardon my ignorance but how on earth does it need 15+ containers? Is that just a scaling thing for your company? Or are their 15 discrete services under the hood to drive it?

It really does run a ton of different services under the hood. It's designed for their type of scale, a small-scale deployment would probably work OK without Zookeeper and similar.

https://develop.sentry.dev/architecture/

Re: Sentry.io outage

#46

Post by jrockway »

that_guy_iain wrote: post #29
nicoburns wrote: post #5

Obviously it's never good when a service has an outage, but it looks like Sentry's status page reports the actual status which makes a nice change.

Plus, it's literally the first time I've seen them having an outage over the years so it seems like they're generally doing well on staying up.

I think this was their Big One: https://blog.sentry.io/2015/07/23/transaction-id-wraparound-...

Re: Sentry.io outage

#47

Post by bredren »

heipei wrote: post #21
Klathmon wrote: post #7

Earlier quoted context omitted.

Has anyone here used their self hosted version? I always assume that running an on-prem version of a SAAS offering is going to be a shit show, but I'm curious if anyone here uses it?

I am using it, but it is not fun to use. It spawns 28 Docker containers (yes, that's the actual number), including things like nginx, postgres, memcached, redis, kafka, zookeeper, clickhouse. I've tried upgrading it once and it failed miserably, after which I simply started from scratch. I would like to use their hosted version, or a more stable paid on-prem version. Pricing is not the issue for me, but I don't want…

Would you mind sharing which other saas products you are so posting?

Re: Sentry.io outage

#48

Post by nik736 »

jhiesey wrote: post #24

The domain sentry.io resolves (for me at least) to a Google Cloud ip address in the us-central1 region, which is having an outage as well: https://status.cloud.google.com/incidents/4Qvmd4q81VnA9RirCM...

Are they using a single zone though? Only a single zone seems to be down.

Re: Sentry.io outage

#49

Post by boesboes »

Cthulhu_ wrote: post #15
dijit wrote: post #6

Sentry is a great product, I love even more that you can self-host it; otherwise I would be one of the people denigrating everyone for running their entire operation on SaaS products. As it stands if their reliability leaves something to be desired: run your own. These things happen, they’re more common the more complicated or huge your setup. Good luck to them in getting it resolved.

> run your own. Is there a good name for this theory that self-hosting something is more reliable? I mean yeah, if you have an outage it only affects you and your customers so I guess it's an improvement in that regard, but thinking you're better at running software than a SaaS is hubris.

It's not going to be more reliable probably, especially if you are trying to host a app with tens of services talking to each other. Scale/load might make a difference sometimes. I'm thinking github's recent issues here. But I wouldn't want to self-host github/gitlab, that would become a full time job probably :) If we decided github was not reliable enough, I'd just make sure we would not be dependend on it for normal work and maybe run my own git server and mirror it to GH.

I am however a proponent of self-hosting open source things you could get as a managed service. I.e databases, elastic search, redis etc. Provided you have the expertise to do that and benifit from it. For example, I used to run a redis setup that would failover in <5s on crashes and with basically 0 downtime for upgrades etc. Now we use ElastiCache and an upgrade will cause serveral minutes of downtime. I haven't had redis crash ever, but I have had elasticache & rds instances dissapear. Sometimes failover works, sometimes not and you have little to no wayt ot find out what is going on.

Post reply on HN