Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sentry.io outage

status.sentry.io

Post reply on HN
Print view View original

71–80 of 84 posts

Re: Sentry.io outage

#71

Post by mekster »

raunometsa wrote: post #55

I wake up to discover that my site https://remotehunt.com is super slow. First thing I do is visit HN to see what's up and I instantly see that Sentry is down. I'm using Sentry to monitor logs and it now makes sense. Ok, so I remove Sentry from Laravel's error handler but nothing changes. And it's weird because sometimes it works, sometimes not. I tweak some things on Cloudflare (turning on Under Attack mode etc). I…

You post not knowing what happened just to say you don't use sentry in the end?

Re: Sentry.io outage

#72

Post by mekster »

Klathmon wrote: post #7
dijit wrote: post #6

Sentry is a great product, I love even more that you can self-host it; otherwise I would be one of the people denigrating everyone for running their entire operation on SaaS products. As it stands if their reliability leaves something to be desired: run your own. These things happen, they’re more common the more complicated or huge your setup. Good luck to them in getting it resolved.

Has anyone here used their self hosted version? I always assume that running an on-prem version of a SAAS offering is going to be a shit show, but I'm curious if anyone here uses it?

It works if you can live with 28 containers (messing up "docker container ls" output that I always have to "grep -v sentry") literally eating 2GB+ memory just to have it launched.

Ideally, things run as a single binary with bring-your-own-database but that's not the case here.

I wish I could use my already installed ClickHouse to save some memory but the version constraint and some very complicated docker compose setup on their end wasn't worth trying.

But I still love the product itself, so I'll let it chew my memory.

Upgrade has been fine as the manual says, you pull their git repo, run the installer and worked fine on all my past several occasions.

Re: Sentry.io outage

#73

Post by metafunctor »

endemic wrote: post #19
Klathmon wrote: post #7

Earlier quoted context omitted.

Has anyone here used their self hosted version? I always assume that running an on-prem version of a SAAS offering is going to be a shit show, but I'm curious if anyone here uses it?

A previous company did. Reliability seemed fine; it was a way to ensure no PHI/PII was accidentally sent to a 3rd party as part of an error report.

Not just PII, but access tokens to 3rd party systems in case of integrations and whatnot.

A Sentry data leak could include valid access tokens to, say, all data on a customer's Docusign account. We try to make sure data like that is scrubbed before sending to Sentry… but mistakes can happen. This is simply not a risk we are willing to take.

Thus, we self-host Sentry in our own secure infrastructure (we are a SaaS provider ourselves), and accept all the maintenance burden that it entails.

Re: Sentry.io outage

#74

Post by s_dev »

krono wrote: post #60
nicoburns wrote: post #5

Obviously it's never good when a service has an outage, but it looks like Sentry's status page reports the actual status which makes a nice change.

If even their own operational telemetry were misrepresented, one might have started wondering about the accuracy of what they're reporting to their customers. So, I agree, but it's not like they really had a choice :)

It's well known that the AWS status page doesn't reflect the actual uptime. Anytime an massive outage is repored here on HN the AWS status page shows all green.

Anytime this debate comes up there are two camps of thought -- the AWS camp where "measuring service degradation" is one of the most insanely complex problems that we lesser devs will never understand -- and the other camp is StatusGator a cheap service that seems to be easily able to tell you when one of your services you use is having trouble - including AWS.

Re: Sentry.io outage

#76

Post by mekster »

fbrncci wrote: post #9

Why do these outages come at just the right time, when you are showing friends or colleagues sentry; in relation to preventing outages.

Because someone is doing what you're doing at every moment?

Re: Sentry.io outage

#77

Post by darknavi »

hadrien01 wrote: post #70
raunometsa wrote: post #55

I wake up to discover that my site https://remotehunt.com is super slow. First thing I do is visit HN to see what's up and I instantly see that Sentry is down. I'm using Sentry to monitor logs and it now makes sense. Ok, so I remove Sentry from Laravel's error handler but nothing changes. And it's weird because sometimes it works, sometimes not. I tweak some things on Cloudflare (turning on Under Attack mode etc). I…

Shouldn't Sentry be called asynchronously?

Can still bog down job queues and cause odd hiccups.

Re: Sentry.io outage

#79

Post by krono »

s_dev wrote: post #74
krono wrote: post #60

Earlier quoted context omitted.

If even their own operational telemetry were misrepresented, one might have started wondering about the accuracy of what they're reporting to their customers. So, I agree, but it's not like they really had a choice :)

It's well known that the AWS status page doesn't reflect the actual uptime. Anytime an massive outage is repored here on HN the AWS status page shows all green. Anytime this debate comes up there are two camps of thought -- the AWS camp where "measuring service degradation" is one of the most insanely complex problems that we lesser devs will never understand -- and the other camp is StatusGator a cheap service that…

Nevertheless, considering telemetry is Sentry's bread and butter it wouldn't be a good look on them in many a manager's eyes.

Re: Sentry.io outage

#80

Post by scottlamb »

dijit wrote: post #62
nik736 wrote: post #48

Earlier quoted context omitted.

Are they using a single zone though? Only a single zone seems to be down.

I think it can be unfair to characterise single zone failures as being an failure to adequately deploy or architect. There's many opportunities for failure even if only a single zone goes away; most (if not nearly all) database solutions elect leaders for example, and "brown-outs" (as in, not total failures) can lead to the leader maintaining leadership status, or at least messing with quorum. other situations can ex…

> I believe multi-zone is less possible than we would like it to be, there are many cases where you can commit no error but still be completely at the mercy of a single zone going away.

There are many understandable ways to accidentally have a single point of failure. But if your conclusion after the outage is that there was no mistake, you have made two of them, and the second is much less understandable.

Post reply on HN