Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Issues with upstream DNS provider

status.heroku.com

Post reply on HN
Print view View original

71–80 of 159 posts

Re: Issues with upstream DNS provider

#71

Post by samcheng »

rsweeney21 wrote: post #35

We just deployed a workaround that brought our site back up. 1. Find your DNS Target in heroku. It should end with .herokudns.com 2. Lookup the historical DNS record to get the IP addresses. You can find historical DNS records here: https://securitytrails.com/dns-trails 3. Replace your CNAME record in your DNS provider with A records that point to the IP addresses you just found. Your site should come back up shortly…

Heroku - if you're listening. Please do NOT migrate IPs / load balancers here for a little while. You might also do some work to detect those of us who have bypassed your DNS, and give us some nudges individually to move back to your DNS when the time is right.

You just caused a very painful outage for your entire customer base, and a number of us used this hack to get back online. If you don't recognize this, and cause ANOTHER outage (the third in the past few weeks...) we're going to flee even faster to more stable platforms.

Re: Issues with upstream DNS provider

#72

Post by zinekeller »

jeffbee wrote: post #52
zinekeller wrote: post #28

Earlier quoted context omitted.

More likely that Cloudflare is unable to access downed servers.

Maybe. I'm just trying to figure out what's common between Steam, Observable, and Heroku, and Cloudflare itself which are all down according to downdetector.

That's blaming the wrong entity - again. No-one seems to read error pages nowadays, not shocking. Also, Steam never used CF services, and user reports about CF are that - they looked at their favorite Substack (https://substack.com) and just seeing a CF page they assumed it's CF's fault.

Re: Issues with upstream DNS provider

#73

Post by Trasmatta »

ab-dm wrote: post #66
exitnode_ttl0 wrote: post #58

Earlier quoted context omitted.

Anybody know what the risks are for doing this? We have not one but 13 heroku domains running where we might do this. What are the chances that their current work on the DNS issues might replace the underlying IP-addresses and hence we are making it harder on ourselves?

I thought about this, and it can't really get any worse than 'total outage', so I thought it was worth taking a shot. It can't go any more down. It worked for me, so once Heroku has sorted their s*t out and their DNS system is back up it's just a matter of deleting the A records and replacing it with the CName. Worst case scenario is the underlying Amazon IP Addresses change and it goes back down.

Wouldn't the worst case be the underlying IP address changes, and your site now points to some other service? Since Heroku doesn't offer static IPs, I don't think.

Re: Issues with upstream DNS provider

#74

Post by tdehnke »

theunixbeard wrote: post #45
rsweeney21 wrote: post #35

We just deployed a workaround that brought our site back up. 1. Find your DNS Target in heroku. It should end with .herokudns.com 2. Lookup the historical DNS record to get the IP addresses. You can find historical DNS records here: https://securitytrails.com/dns-trails 3. Replace your CNAME record in your DNS provider with A records that point to the IP addresses you just found. Your site should come back up shortly…

Presumably if your site is behind Cloudflare then this strategy won't work, right? Since the IP addresses that Security Trails sees are just of Cloudflare rather than your actual Heroku IPs...? Or is it possible in the Cloudflare dashboard there is somewhere to see your Heroku server's IP address?

We use Cloudflare + Heroku and it worked for us.

Re: Issues with upstream DNS provider

#75

Post by exitnode_ttl0 »

ab-dm wrote: post #66
exitnode_ttl0 wrote: post #58

Earlier quoted context omitted.

Anybody know what the risks are for doing this? We have not one but 13 heroku domains running where we might do this. What are the chances that their current work on the DNS issues might replace the underlying IP-addresses and hence we are making it harder on ourselves?

I thought about this, and it can't really get any worse than 'total outage', so I thought it was worth taking a shot. It can't go any more down. It worked for me, so once Heroku has sorted their s*t out and their DNS system is back up it's just a matter of deleting the A records and replacing it with the CName. Worst case scenario is the underlying Amazon IP Addresses change and it goes back down.

There is one more problem I can think of (but yes you're right, it can't get worse than 'total outage'): Heroku might detect that the CNAME is missing and will deactivate some stuff from their automatic domain management / TLS certificate issuing.

Re: Issues with upstream DNS provider

#76

Post by caffeinated_me »

bobsingor wrote: post #30
rajamaka wrote: post #9

Production completely inaccessible, on the plus side I might finally now get backing to migrate away from the Heroku cesspit.

Yes, i'm also done with Heroku. What alternative do you recommend?

We did some trials and ended up on Crunchy Data for the Postgres part of the equation. Wish we had done so earlier, logical replication was a big win.

Sadly, we're still using Heroku DNS, but this should accelerate finding an alternative.

Re: Issues with upstream DNS provider

#77

Post by craigkerstiens »

samcheng wrote: post #71
rsweeney21 wrote: post #35

We just deployed a workaround that brought our site back up. 1. Find your DNS Target in heroku. It should end with .herokudns.com 2. Lookup the historical DNS record to get the IP addresses. You can find historical DNS records here: https://securitytrails.com/dns-trails 3. Replace your CNAME record in your DNS provider with A records that point to the IP addresses you just found. Your site should come back up shortly…

Heroku - if you're listening. Please do NOT migrate IPs / load balancers here for a little while. You might also do some work to detect those of us who have bypassed your DNS, and give us some nudges individually to move back to your DNS when the time is right. You just caused a very painful outage for your entire customer base, and a number of us used this hack to get back online. If you don't recognize this, and ca…

Those IPs tend to stay pretty stable and are shared across the fleet. While you should change them back once things are stable, it's reasonably likely these won't change out from under you in a few hrs.

Re: Issues with upstream DNS provider

#78

Post by ab-dm »

jarym wrote: post #62
ab-dm wrote: post #20

Complete outage on all of my sites. I have a migration plan in place but it's been put on hold to launch a new Product.

And the new product might have to be put on hold until Heroku is back… catch-22 - you’ll never migrate away

Well, the new product is hosted on AWS, so it's much more stable, and we've made our first lot of pre-sales so we're on a tight deadline. Not sure why you're being so negative for no reason.

Re: Issues with upstream DNS provider

#80

Post by rsweeney21 »

Trasmatta wrote: post #73
ab-dm wrote: post #66

Earlier quoted context omitted.

I thought about this, and it can't really get any worse than 'total outage', so I thought it was worth taking a shot. It can't go any more down. It worked for me, so once Heroku has sorted their s*t out and their DNS system is back up it's just a matter of deleting the A records and replacing it with the CName. Worst case scenario is the underlying Amazon IP Addresses change and it goes back down.

Wouldn't the worst case be the underlying IP address changes, and your site now points to some other service? Since Heroku doesn't offer static IPs, I don't think.

Heroku uses the host name to route requests. So you wouldn't serve another site, it would just give you an error that your site wasn't found or something.
Post reply on HN