Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sentry.io outage

status.sentry.io

Post reply on HN
Print view View original

61–70 of 84 posts

Re: Sentry.io outage

#61

Post by obituary_latte »

raunometsa wrote: post #55

I wake up to discover that my site https://remotehunt.com is super slow. First thing I do is visit HN to see what's up and I instantly see that Sentry is down. I'm using Sentry to monitor logs and it now makes sense. Ok, so I remove Sentry from Laravel's error handler but nothing changes. And it's weird because sometimes it works, sometimes not. I tweak some things on Cloudflare (turning on Under Attack mode etc). I…

Are you using Google cloud? https://status.cloud.google.com/incidents/4Qvmd4q81VnA9RirCM...

Re: Sentry.io outage

#62

Post by dijit »

nik736 wrote: post #48
jhiesey wrote: post #24

The domain sentry.io resolves (for me at least) to a Google Cloud ip address in the us-central1 region, which is having an outage as well: https://status.cloud.google.com/incidents/4Qvmd4q81VnA9RirCM...

Are they using a single zone though? Only a single zone seems to be down.

I think it can be unfair to characterise single zone failures as being an failure to adequately deploy or architect.

There's many opportunities for failure even if only a single zone goes away; most (if not nearly all) database solutions elect leaders for example, and "brown-outs" (as in, not total failures) can lead to the leader maintaining leadership status, or at least messing with quorum.

other situations can exist where the migration out of a zone leads to hardware becoming unavailable for consumption for other people, after all, the cloud is not magic and if peoples workloads auto shift to the surrounding (unaffected) zones then it will impact peoples ability to do the same migration as all the free hardware could be used up.

I can think of dozens of examples honestly where even if you had built everything multi-zonal you could be down due to a single zone; for instance if some unknown subsystem was zonal (like IAM?) or you use regionally available persistent disks and now they suddenly perform extremely bad with writes because they can't sync to the unavailable datacenter.

I believe multi-zone is less possible than we would like it to be, there are many cases where you can commit no error but still be completely at the mercy of a single zone going away.

Re: Sentry.io outage

#63

Post by np_tedious »

luismedel wrote: post #41
the_gipsy wrote: post #28

Earlier quoted context omitted.

Yes the CI should fail, or else you would be deploying something that can't match your errors to your source code (via the sourcemaps which is what presumably gets uploaded to sentry on the CI run).

Take in advance that I don't know Sentry, nor webpack et al but, in my mind, I'd like to have an opportunity to continue my build process with a big, red warning. I prefer a 98% accurate error reporting better than a 100% one I can't push to production. Or maybe I'm not understanding well the value Sentry offers, of course.

I kind of agree. Their source map thing to get better traces is a nicety, but few would find it release blocking to forgo it

Edit due to other comment: this assumes CI/CD. If you were deploying only weekly then this might be good reason to delay

Re: Sentry.io outage

#64

Post by zeeg »

Service should be nearly restored at this point. We apologize for the significant interrupt to everyones days. As most of you all figured out at this point the issue was with a regional failure in GCP (central, which is our primary zone). We're going to be exploring what we can do to lessen a single zone failure, and more importantly, reduce/remove any impact of the storage services being unavailable.

<3

Re: Sentry.io outage

#65

Post by throwaway787544 »

dijit wrote: post #62
nik736 wrote: post #48

Earlier quoted context omitted.

Are they using a single zone though? Only a single zone seems to be down.

I think it can be unfair to characterise single zone failures as being an failure to adequately deploy or architect. There's many opportunities for failure even if only a single zone goes away; most (if not nearly all) database solutions elect leaders for example, and "brown-outs" (as in, not total failures) can lead to the leader maintaining leadership status, or at least messing with quorum. other situations can ex…

Yeah but it looks bad

Re: Sentry.io outage

#66

Post by samstave »

dijit wrote: post #62
nik736 wrote: post #48

Earlier quoted context omitted.

Are they using a single zone though? Only a single zone seems to be down.

I think it can be unfair to characterise single zone failures as being an failure to adequately deploy or architect. There's many opportunities for failure even if only a single zone goes away; most (if not nearly all) database solutions elect leaders for example, and "brown-outs" (as in, not total failures) can lead to the leader maintaining leadership status, or at least messing with quorum. other situations can ex…

I remember when it was the control plane at AWS US-West that went out - causing mass havoc for many regardless of your architecture.

Re: Sentry.io outage

#67

Post by zeeg »

zeeg wrote: post #64

Service should be nearly restored at this point. We apologize for the significant interrupt to everyones days. As most of you all figured out at this point the issue was with a regional failure in GCP (central, which is our primary zone). We're going to be exploring what we can do to lessen a single zone failure, and more importantly, reduce/remove any impact of the storage services being unavailable. <3

For folks reading this - and we'll solve this at a global scale - check your `SENTRY_DSN` setting and if its not using something like `oXXXXX.ingest.sentry.io` you should consider rolling out the updated value (located in your project settings). Our master domain routing layer is a more complicated failure point, and moving to the ingest based domains pushes you to our edge layer which _should_ be extremely reliable as well as much reduced global latency.

Re: Sentry.io outage

#70

Post by hadrien01 »

raunometsa wrote: post #55

I wake up to discover that my site https://remotehunt.com is super slow. First thing I do is visit HN to see what's up and I instantly see that Sentry is down. I'm using Sentry to monitor logs and it now makes sense. Ok, so I remove Sentry from Laravel's error handler but nothing changes. And it's weird because sometimes it works, sometimes not. I tweak some things on Cloudflare (turning on Under Attack mode etc). I…

Shouldn't Sentry be called asynchronously?
Post reply on HN