Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Issues with upstream DNS provider

status.heroku.com

Post reply on HN
Print view View original

151–159 of 159 posts

Re: Issues with upstream DNS provider

#151

Post by craigkerstiens »

rsavage wrote: post #150
benschwarz wrote: post #136

Earlier quoted context omitted.

My company moved a 600GB postgres db to crunchydata ( https://www.crunchydata.com ), the process was super smooth. In fact, it went so well that I agreed to a quick case study: https://www.crunchydata.com/case-studies/calibre . We're currently using Heroku for app hosting, while evaluating fly.io, render.com and railway.app. All three have had exceptional reviews from other customers and differ slightly on their serv…

I’d be really interested to know how your research goes. We are running a large production scale app on heroku. And that we have no DevOps or Infra staff is awesome. But heroku has been a huge let down lately. Most of the reviews we have seen of the competitors are all hobby level. And last time we check some of this competitors we found their security posture was not the level we would require. So we had to simple r…

Craig here from Crunchy Data. I think you're speaking to the app side of things on hobby level. On the database side of things our security posture for Crunchy Bridge I'd say is stronger than the Heroku one. By default we isolate all databases in a VPC, everything is purely single tenant where as Heroku Postgres at least when I was there had multiple forms of multi-tenancy which when doing multi-tenancy in Postgres can have risks[1]-this applies even to the major 3 cloud providers. Our team is essentially the original Heroku Postgres team so we've built with security but also user experience for Postgres in mind since day one.

Now I assume you were speaking to the 3 mentioned, render, railway, fly in terms of hobby level. All three are fairly young relative to Heroku's age, but Fly did recently get their SOC2 and the team really took it to heart and invested in it so I'd put some stock in that. I can't speak definitively to the others, but do know all three can be solid for production apps. If you've got HIPAA or other specific requirements I'd encourage a conversation with them.

[1] https://www.wiz.io/blog/the-cloud-has-an-isolation-problem-p...

Re: Issues with upstream DNS provider

#152

Post by craigkerstiens »

alexthehurst wrote: post #147
benschwarz wrote: post #136

Earlier quoted context omitted.

My company moved a 600GB postgres db to crunchydata ( https://www.crunchydata.com ), the process was super smooth. In fact, it went so well that I agreed to a quick case study: https://www.crunchydata.com/case-studies/calibre . We're currently using Heroku for app hosting, while evaluating fly.io, render.com and railway.app. All three have had exceptional reviews from other customers and differ slightly on their serv…

For what it's worth: Crunchydata appears to have been using Heroku DNS in some way. We have a Heroku app with Crunchy databases, and our Crunchy dbs became inaccessible shortly before the app did.

Craig here from Crunchy, we do run some small pieces on Heroku, but during the DNS outage saw no interruption of databases and they all seemed to be available and up. Please do feel free to drop us a note and we'll happily investigate if something did occur there.

Re: Issues with upstream DNS provider

#155

Post by nightpool »

bombcar wrote: post #145
nightpool wrote: post #104

Earlier quoted context omitted.

The advantage is that you can do the CNAME swap atomically in the Cloudflare interface, instead of deleting a CNAME record and adding a new A record, which would lead to a few seconds when your DNS is entirely removed. Those few seconds may not sound like a lot, but they could be cached by Cloudflare or other external DNS resolvers, trigger "clean-up" functions within Cloudflare to remove resources associated with th…

Sounds like cloudflare could add an atomic cname -> a record feature.

Honestly, I think this solution worked better than any dedicated feature could have, since it used the exact same existing codepaths that Cloudflare already had for CNAME flattening. Using CNAME records for self-documentation seems like a better practice anyway. Spending the time building a complicated new feature for such a rare case seems like a bad idea.

Re: Issues with upstream DNS provider

#156

Post by johnklos »

Wow. Imagine being so out of touch that you have to have another entity run your DNS services.

Email I get, because there has been a hard push for decades to force everyone on to big providers, but DNS can literally be run by anyone, anywhere.

Did the primary servers push bad data, making the secondary / tertiary ones break, too? If not, why not extend the cache lifetime and run off of them until the primary are fixed?

Sigh. This is rather ridiculous, and is rather embarrassing for Heroku.

Re: Issues with upstream DNS provider

#157

Post by bennylope »

benschwarz wrote: post #136
bobsingor wrote: post #30

Earlier quoted context omitted.

Yes, i'm also done with Heroku. What alternative do you recommend?

My company moved a 600GB postgres db to crunchydata ( https://www.crunchydata.com ), the process was super smooth. In fact, it went so well that I agreed to a quick case study: https://www.crunchydata.com/case-studies/calibre . We're currently using Heroku for app hosting, while evaluating fly.io, render.com and railway.app. All three have had exceptional reviews from other customers and differ slightly on their serv…

This is the same path we're on. Migrated to Crunchy a month ago or so to remove the major migration risk and are using Render to host an auxiliary service while our core application remains on Heroku. Haven't yet done any non-toy deployments on Fly.io or Railway but I very much like Render's Blueprints and environment groups.

Re: Issues with upstream DNS provider

#158

Post by newalexandria »

rsweeney21 wrote: post #35

We just deployed a workaround that brought our site back up. 1. Find your DNS Target in heroku. It should end with .herokudns.com 2. Lookup the historical DNS record to get the IP addresses. You can find historical DNS records here: https://securitytrails.com/dns-trails 3. Replace your CNAME record in your DNS provider with A records that point to the IP addresses you just found. Your site should come back up shortly…

Heroku reports the outage is over - but I wasn't able to resolve our Heroku-based domain name until recently today.

Should be able to go back to CNAME now. Regardless, recheck yours

Re: Issues with upstream DNS provider

#159

Post by rsavage »

craigkerstiens wrote: post #151
rsavage wrote: post #150

Earlier quoted context omitted.

I’d be really interested to know how your research goes. We are running a large production scale app on heroku. And that we have no DevOps or Infra staff is awesome. But heroku has been a huge let down lately. Most of the reviews we have seen of the competitors are all hobby level. And last time we check some of this competitors we found their security posture was not the level we would require. So we had to simple r…

Craig here from Crunchy Data. I think you're speaking to the app side of things on hobby level. On the database side of things our security posture for Crunchy Bridge I'd say is stronger than the Heroku one. By default we isolate all databases in a VPC, everything is purely single tenant where as Heroku Postgres at least when I was there had multiple forms of multi-tenancy which when doing multi-tenancy in Postgres c…

Thanks Craig. I took at look at Crunchydata - however best I can tell unless we are in Enterprise Tier Heroku (or maybe not even then) we have to connect to Crunchydata via internet (with IP whitelist?) rather than through VPC peering or similar. Which is a limitation of Heroku rather than you. I assume with something like fly it could be done via VPC peering?

I just read fly had SOC2 type I recently. But I mean this hosting infra containing all our data and our customers data. People providing infra really need to take security extremely seriously and prove it.

Awesome what they are doing - just don’t feel like they are ready for primetime busines. We are a small startup (5k monthly on Heroku) but there is just no reasonable way we can tell our enterprise customers security teams are hosted on these guys and can vouch and vet their security.

Once fly has type II - we’ll take another look.

Post reply on HN