Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Update on Our Outage

blog.roblox.com

Post reply on HN
Print view View original

1–10 of 235 posts

Re: An Update on Our Outage

#2

Post by reilly3000 »

Did they run out of internal ipv4 address space somehow? I’ll be curious for the post-mortem, this is definitely the longest outage of a global web service I can remember in… forever? This incident puts them under 2 9’s.

Re: An Update on Our Outage

#3

Post by dannyw »

reilly3000 wrote: post #2

Did they run out of internal ipv4 address space somehow? I’ll be curious for the post-mortem, this is definitely the longest outage of a global web service I can remember in… forever? This incident puts them under 2 9’s.

Not a lot of details here but I'd suspect it's yet another example of autoscaling policies gone wrong.

Re: An Update on Our Outage

#4

Post by mr-karan »

Huh, strangely the link doesn't open for me. Seems like an infinite redirect loop.

    $ curl -i -L https://blog.roblox.com/2021/10/update-on-our-outage/
    
    HTTP/1.1 301 Moved Permanently
    Server: nginx
    Content-Type: text/html; charset=UTF-8
    Content-Length: 0
    X-Redirect-By: WordPress
    Location: https://blog.roblox.com/2021/10/update-on-our-outage/
    X-Powered-By: WP Engine
    X-Cacheable: bot
    X-Cache-Group: bot
    Strict-Transport-Security: max-age=63072000; includeSubDomains; preload
    Cache-Control: must-revalidate, max-age=7108
    Expires: Mon, 01 Nov 2021 05:39:59 GMT
    Date: Mon, 01 Nov 2021 03:41:31 GMT
    Connection: keep-alive

Re: An Update on Our Outage

#5

Post by nitinreddy88 »

mr-karan wrote: post #4

Huh, strangely the link doesn't open for me. Seems like an infinite redirect loop. $ curl -i -L https://blog.roblox.com/2021/10/update-on-our-outage/ HTTP/1.1 301 Moved Permanently Server: nginx Content-Type: text/html; charset=UTF-8 Content-Length: 0 X-Redirect-By: WordPress Location: https://blog.roblox.com/2021/10/update-on-our-outage/ X-Powered-By: WP Engine X-Cacheable: bot X-Cache-Group: bot Strict-Transport-Se…

Pretty much same here

Re: An Update on Our Outage

#6

Post by dreadlordbone »

mr-karan wrote: post #4

Huh, strangely the link doesn't open for me. Seems like an infinite redirect loop. $ curl -i -L https://blog.roblox.com/2021/10/update-on-our-outage/ HTTP/1.1 301 Moved Permanently Server: nginx Content-Type: text/html; charset=UTF-8 Content-Length: 0 X-Redirect-By: WordPress Location: https://blog.roblox.com/2021/10/update-on-our-outage/ X-Powered-By: WP Engine X-Cacheable: bot X-Cache-Group: bot Strict-Transport-Se…

Works for me, possibly a cached 301?

Re: An Update on Our Outage

#8

Post by reilly3000 »

dannyw wrote: post #3
reilly3000 wrote: post #2

Did they run out of internal ipv4 address space somehow? I’ll be curious for the post-mortem, this is definitely the longest outage of a global web service I can remember in… forever? This incident puts them under 2 9’s.

Not a lot of details here but I'd suspect it's yet another example of autoscaling policies gone wrong.

What does that tend to look like? Do you have any examples?

Re: An Update on Our Outage

#10

Post by aahortwwy »

> We will publish a post-mortem with more details once we’ve completed our analysis, along with the actions we’ll be taking to avoid such issues in the future.

Excellent. That should be a very interesting read.

> In addition, we will implement a policy to make our creator community economically whole as a result of this outage.

Good to hear.

Post reply on HN