Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku was Down

news.ycombinator.com

Post reply on HN
Print view View original

141–149 of 149 posts

Re: Heroku was Down

#141

Post by jrochkind1 »

jermaustin1 wrote: post #79
reubano wrote: post #69

Earlier quoted context omitted.

I still don't get why they use images Instead of unicode: https://www.htmlsymbols.xyz/unicode/U%2b2705

if the image tag is drawn dynamically then it is actually probably less bandwidth than the unicode character since the image can be cached at multiple locations including the browser.

Less bandwidth than the 1-4 bytes of a codepoint in UTF-8? How do you figure?

Re: Heroku was Down

#142

Post by jrochkind1 »

alpb wrote: post #112
dec0dedab0de wrote: post #111

I've only ever used heroku for free tier level personal projects, but as I understand it they use AWS to do the actual hosting. I can understand an outage affecting their deployment process, but what could cause running servers to go down? As I typed that out I remembered that they handle DNS, load balancing, and databases, so I guess any one of them.

All it takes is a load balancer misconfiguration on a core service to take a large scale service down.

it indeed behaved like a routing issue of some kind (my app was still UP, and was still logging, just no traffic could get to it), and a heroku incident status line said "Engineers are recovering affected routing components," so, yup.

Re: Heroku was Down

#143

Post by jrochkind1 »

craigkerstiens wrote: post #98

It appears to only be affecting apps that have multiple dynos. While Heroku is busy fixing the issue if you scale your app to a single dyno (assuming it can handle the traffic) it should restore availability.

My single-dyno app was affected.

However, while I could not connect to my app, nobody I know could connect to my app, and my monitoring service could not connect to my app for a ping... my app logs showed that some traffic continued to connect throughout the outage. So it was not entirely universal. And was clearly a routing problem of some kind.

Re: Heroku was Down

#144

Post by jermaustin1 »

jrochkind1 wrote: post #141
jermaustin1 wrote: post #79

Earlier quoted context omitted.

if the image tag is drawn dynamically then it is actually probably less bandwidth than the unicode character since the image can be cached at multiple locations including the browser.

Less bandwidth than the 1-4 bytes of a codepoint in UTF-8? How do you figure?

Because if it is cached, then it is 0 bytes transferred. First request could be probably a few hundred bytes, but never needed again. And once it is at a CDN, there is never another request to the server.

Re: Heroku was Down

#145

Post by castaway3000 »

chasd00 wrote: post #124
teagee wrote: post #11

HN is truly a market leader in status page technology

A couple of outages that have affected my client i learned about first on HN. We were able to mobilize a team and get on top of it faster than any monitoring team at the client ( a state government ). I feel like HN should invoice us haha

You might benefit from some better monitoring!

Re: Heroku was Down

#147

Post by wuputah »

dspillett wrote: post #117
jlangenauer wrote: post #20

All our apps are down as well. And it seems (and I really have to almost laugh here) that https://status.heroku.com no longer loads.

As always, a note to infrastructure providers: HOST YOUR LIVE STATUS DETAILS ON OTHER INFRASTRUCTURE. Of course, they won't. If they host is on someone else's then that might look bad (tacitly saying that a competitor is reliable and might be up when they are down) and if they hive off an extra copy of some of their infrastructure there will still be single points of failure either accidentally, by human error (someo…

I agree with your lead statement and argued as such, but was overruled. Last I knew and understood, Heroku Status is static pages pushed out to Fastly, with the internal admin site (that does that work) running in a Heroku Private Space. If you look at the DNS, it still appears to be served by Fastly, and Heroku Private Spaces are generally pretty isolated infra, so I would be curious what the failure mode was here. But ultimately this is the fire you play with when you self-host your status site...
Post reply on HN