Render Is Down • Hacker News Board

Re: Render Is Down

#11

Post by anurag » Tue, Mar 26, 2024, 4:42 PM UTC

Sytten wrote: post #10

anurag wrote: post #6

(Render CEO) Things are recovering. We're still investigating, but the issue seems unrelated to Cloudflare. We'll post more updates on https://status.render.com .

I expect to have a decent post-mortem for this major outage...

We will share a full incident report soon.

Re: Render Is Down

#12

Post by agamble » Tue, Mar 26, 2024, 4:46 PM UTC

Our production Postgres instance is still down as of 16:46 London time.

Re: Render Is Down

#13

Post by anurag » Tue, Mar 26, 2024, 4:48 PM UTC

We're posting the latest updates at https://status.render.com. Databases are taking longer to recover but we're on it.

Re: Render Is Down

#14

Post by zegl » Tue, Mar 26, 2024, 4:55 PM UTC

anurag wrote: post #9

Sytten wrote: post #7

All our services and databases are still down

We're on it. Some databases are taking longer to recover.

Does "some databases" mean all? I have three databases on Render, all are currently down.

Re: Render Is Down

#15

Post by Sytten » Tue, Mar 26, 2024, 4:55 PM UTC

seabass wrote: post #5

Whenever there's an incident like this, it seems like the status pages take a while to reflect the problem. Why is that? The part of the status page that can be manually updated is now correct, but most of the automated checks still show 100% uptime and only "degraded performance" in some cases despite being fully offline. If I were implementing a status page, it might look something like a ping to some url from vari…

For me it is the speed of update, I posted an outage on our own status a good 15m before they did on their status. I had to email them before I got an ACK.

Re: Render Is Down

#16

Post by abi » Tue, Mar 26, 2024, 4:57 PM UTC

Our databases are all down :(

Re: Render Is Down

#17

Post by Sytten » Tue, Mar 26, 2024, 4:58 PM UTC

zegl wrote: post #14

anurag wrote: post #9

Earlier quoted context omitted.

We're on it. Some databases are taking longer to recover.

Does "some databases" mean all? I have three databases on Render, all are currently down.

Same for me, 3 database and they are all down for almost an hour now

Re: Render Is Down

#18

Post by Sytten » Tue, Mar 26, 2024, 5:13 PM UTC

anurag wrote: post #13

We're posting the latest updates at https://status.render.com . Databases are taking longer to recover but we're on it.

Please give us an update if you have one, are we to expect a multi-hour recovery?

Re: Render Is Down

#19

Post by paultannenbaum » Tue, Mar 26, 2024, 5:31 PM UTC

zegl wrote: post #14

anurag wrote: post #9

Earlier quoted context omitted.

We're on it. Some databases are taking longer to recover.

Does "some databases" mean all? I have three databases on Render, all are currently down.

Same here, we have several all in different datacenters. All down

Re: Render Is Down

#20

Post by anurag » Tue, Mar 26, 2024, 5:37 PM UTC

Sytten wrote: post #18

anurag wrote: post #13

We're posting the latest updates at https://status.render.com . Databases are taking longer to recover but we're on it.

Please give us an update if you have one, are we to expect a multi-hour recovery?

Recovery is well underway, but I don't have an ETA yet. Some regions like Ohio are recovering faster than others.