Oh lawd its starting to come back up! Pingdom monitor is flapping and we're starting to get some successful, albeit slow, logins!
Yes, looks like it started to work for us too a few minutes ago.
Auth0 Down
41–50 of 78 posts
Re: Auth0 Down
#42Don't forget to request your credit per their SLA[0]. You have 10 days to request your credit, which by my calculations should be 10% of this months' charge. Not a fair trade for leaving us dead in the water for 4 hours, but SLAs in general are worthless.
[0]: https://auth0.com/docs/support/services-level-descriptions
Re: Auth0 Down
#43Don't forget to request your credit per their SLA[0]. You have 10 days to request your credit, which by my calculations should be 10% of this months' charge. Not a fair trade for leaving us dead in the water for 4 hours, but SLAs in general are worthless. [0]: https://auth0.com/docs/support/services-level-descriptions
I'm willing to bet we can get at least the 20% return, they're still "officially" down. I bet it'll be a few hours till they say we're good.
Everyone make sure you've got detailed tickets into their service queue. Get your creds!
Re: Auth0 Down
#44We managed to get a reply from a C level. All we could get out of them was "something to do with our DB, but we don't know the root yet. Our fail-over process didn't work. This will never happen again". Also, it only took them 2 and a half hours to admit it was their entire system instead of "a small subset of users" lol.
"something to do with our DB"
Oof. Is there something about what they do that prevents you from having a completely separate second site? Or is this a case where "bad data" is being happily propagated to the redundant site?
As core as the service is, I imagined a panic button that reverted the database for site #2 to some specified point in time.
Re: Auth0 Down
#45Wow. The whole point of paying someone like Auth0 is to _not_ have this happen. This is basically their whole point, is it not? Really looking forward to the post-mortem, but I won't ever forget just how down this thing is right now.
The whole point for me is that I don't want to be responsible for user credentials. I don't trust the security of my app.
Re: Auth0 Down
#46What alternatives to Auth0 are worth looking into? Between this P0 (with no ability to check the status or file a ticket) and the Okta acquisition, I hesitate to continue using Auth0 as the default when spinning up new web apps.
What's the point of such services? Every web framework worth its salt supports all common authentication schemes and offers an easy interface to write your own.
Re: Auth0 Down
#47This type of incident is exactly why I dislike identity federation as a service. Yes it's difficult to get right, and you open yourself up to additional risk and technical complexity to do the federation yourself, but simultaneously how many businesses are currently completely down and just sitting on their hands waiting for Auth0's engineers to fix their systems?
Re: Auth0 Down
#48We managed to get a reply from a C level. All we could get out of them was "something to do with our DB, but we don't know the root yet. Our fail-over process didn't work. This will never happen again". Also, it only took them 2 and a half hours to admit it was their entire system instead of "a small subset of users" lol.
"something to do with our DB" Oof. Is there something about what they do that prevents you from having a completely separate second site? Or is this a case where "bad data" is being happily propagated to the redundant site? As core as the service is, I imagined a panic button that reverted the database for site #2 to some specified point in time.
From what I've seen on some internal email chains, there is a fail-over process for HA multi-region/site, but it didn't work right. Whomp whomp.
Re: Auth0 Down
#49Management console went down for us again. 502 service unavailable.
Re: Auth0 Down
#50What alternatives to Auth0 are worth looking into? Between this P0 (with no ability to check the status or file a ticket) and the Okta acquisition, I hesitate to continue using Auth0 as the default when spinning up new web apps.
Okta?