I wonder if they are on GCP which is also down this morning durning the same time frame.
Sentry.io outage
31–40 of 84 posts
Re: Sentry.io outage
#32Sentry is a great product, I love even more that you can self-host it; otherwise I would be one of the people denigrating everyone for running their entire operation on SaaS products. As it stands if their reliability leaves something to be desired: run your own. These things happen, they’re more common the more complicated or huge your setup. Good luck to them in getting it resolved.
Has anyone here used their self hosted version? I always assume that running an on-prem version of a SAAS offering is going to be a shit show, but I'm curious if anyone here uses it?
I run it on k8s for a medium sized company, as running via compose didn't seem to scale well for us. I even upgraded from 9.11 which ran a handful of containers to the most recent version which runs 15+ containers.
You get used to the quirks of running it and then it just works, but it did take some time polishing the health checks etc to get it HA and scalable.
None of the helm charts I tried actually worked or provided HA.
Re: Sentry.io outage
#33Sentry is a great product, I love even more that you can self-host it; otherwise I would be one of the people denigrating everyone for running their entire operation on SaaS products. As it stands if their reliability leaves something to be desired: run your own. These things happen, they’re more common the more complicated or huge your setup. Good luck to them in getting it resolved.
Has anyone here used their self hosted version? I always assume that running an on-prem version of a SAAS offering is going to be a shit show, but I'm curious if anyone here uses it?
We run the on-prem version, but looking to transition to SaaS soon. While one can point at the current outage and say "good thing we run it ourselves!" it is a chore to keep updated and running smoothly.
That said, it runs untouched for long stretches of time no problem.
Re: Sentry.io outage
#34I wonder if they are on GCP which is also down this morning durning the same time frame.
Yeah they use GCP for most of their stuff.
Not sure if their POP ingest servers are on it though.
Re: Sentry.io outage
#35Earlier quoted context omitted.
Has anyone here used their self hosted version? I always assume that running an on-prem version of a SAAS offering is going to be a shit show, but I'm curious if anyone here uses it?
I run it on k8s for a medium sized company, as running via compose didn't seem to scale well for us. I even upgraded from 9.11 which ran a handful of containers to the most recent version which runs 15+ containers. You get used to the quirks of running it and then it just works, but it did take some time polishing the health checks etc to get it HA and scalable. None of the helm charts I tried actually worked or prov…
Pardon my ignorance but how on earth does it need 15+ containers?
Is that just a scaling thing for your company? Or are their 15 discrete services under the hood to drive it?
Re: Sentry.io outage
#36Sentry is a great product, I love even more that you can self-host it; otherwise I would be one of the people denigrating everyone for running their entire operation on SaaS products. As it stands if their reliability leaves something to be desired: run your own. These things happen, they’re more common the more complicated or huge your setup. Good luck to them in getting it resolved.
Has anyone here used their self hosted version? I always assume that running an on-prem version of a SAAS offering is going to be a shit show, but I'm curious if anyone here uses it?
Their self-hosted version works well with the caveat that the upgrade process can be a pain and can be buggy. I used their docker-compose open source version which involved downtime to spin down the stack, then spin up the new stack. Considering that there's no enterprise support for on-prem, its nice that it works so well. If you're using their apm/observability you will need to keep an eye out on disk space and memory/cpu after turning it on, we ran into resources issues due to the amount of data we started sending. Otherwise it's rock solid and cost-efficient, we were paying 500ish bucks a month for infra and 1-2 days of dev time for something sentry.io quoted 50,000K/month for. I imagine the kubernetes installer is much less hands on in terms of upgrades. Would use again without any hesitation.
Re: Sentry.io outage
#37I just dealt with this in my laravel app and it was preventing the application from working. I was able to resolve by redeploying app with "SENTRY_LARAVEL_DSN=null" in the .env file
I'm curious if using Relay (https://docs.sentry.io/product/relay/) would prevent these problems?
Re: Sentry.io outage
#38Sentry is a great product, I love even more that you can self-host it; otherwise I would be one of the people denigrating everyone for running their entire operation on SaaS products. As it stands if their reliability leaves something to be desired: run your own. These things happen, they’re more common the more complicated or huge your setup. Good luck to them in getting it resolved.
Has anyone here used their self hosted version? I always assume that running an on-prem version of a SAAS offering is going to be a shit show, but I'm curious if anyone here uses it?
We used to run it self hosted, but the architecture is rather complex, so we ended up switching to their SaaS offering. I tried advocating [0] for a leaner architecture for simpler setups + to run integration tests against... but it wasn't met with much enthusiasm.
I imagine 99% of installations (and certainly CI pipelines) would be fine with their wsgi webserver and sqlite instead of Clickhouse, Relay, memcached, nginx, Postgres, Kafka and a host of other services. I wanted to take a stab at this myself, but given the complexity of the system, uncertainty of being able to merge it back, and, last but not least, their license, I decided against it.
Re: Sentry.io outage
#39The domain sentry.io resolves (for me at least) to a Google Cloud ip address in the us-central1 region, which is having an outage as well: https://status.cloud.google.com/incidents/4Qvmd4q81VnA9RirCM...
Seems you are right: https://twitter.com/mitsuhiko/status/1522550576734691328
Re: Sentry.io outage
#40The domain sentry.io resolves (for me at least) to a Google Cloud ip address in the us-central1 region, which is having an outage as well: https://status.cloud.google.com/incidents/4Qvmd4q81VnA9RirCM...
I am seeing some GKE control plane weirdness this morning too. API server endpoints going unreachable intermittently.