Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku is down

heroku.com

Post reply on HN
Print view View original

101–107 of 107 posts

Re: Heroku is down

#101

Post by tptacek »

joevandyk wrote: post #82
jsprinkles wrote: post #77

Earlier quoted context omitted.

That's exactly why you don't want to use those features, for what it's worth.

That means that I need to know about (and program chef to do): logging security hardening firewalls backups (both point in time and complete backups, both for database and for any other artifacts like image uploads) testing backup recovery (both the point in time and complete backups) replication HA LB SSL termination postgresql configuration monitoring (security, system stats, application availability, individual pr…

Know about? Program Chef to do? How about: be on call 24/7 to handle random problems in? The best, most competent ops teams can all tell you horror stories about black swan outages traceable to some of the least likely components in the stack you just outlined.

Almost all of the things you outlined are things that have in the past blown up in real world deployments, often at cloud hosting providers where you didn't know about it it because a well-paid, well-trained ops team hid the drama for you.

So yeah, definitely factor that in to the cost of hosting at a cloud provider. You're absolutely right.

Re: Heroku is down

#102

Post by tptacek »

jsprinkles wrote: post #91
chao- wrote: post #83

Earlier quoted context omitted.

I am intrigued. Do you have any quick links on hand, or pointers on how one would go about learning how to set up something like this? Or just learning about how to distribute services in general? That is to say, I know my way around Linux, have set up single/standalone servers for all sorts of services, but have never known remotely where to begin on the distributed side of things, much less understand what is requi…

I learned from trial and error and also on the job. In my experience, how-tos are typically antiquated or not terribly clear. My best advice would be to dig into ServerFault and Google what you don't understand.

I did too; for instance, I learned how to terminate DS1s and PRIs by trial-and-error punching down every possible combination of little colored wires, and to this day remember that it's ESF/B8ZS and not AMI because that setting change, at 10:30PM on a Friday night, was when I got the Livingston box to light up properly.

Fun? Yes. Miss it? Fuck no. Recommend people relive the 2012 version of the experience? Huh? Get back to work writing code. Re-read the Wikipedia page on "comparative advantage" first if you need to motivate yourself.

Re: Heroku is down

#103

Post by rdl »

T-hawk wrote: post #98
rdl wrote: post #96

Earlier quoted context omitted.

Ultimately this is why it would be nice for the code/configuration to be independent of the operations. I agree it is unreasonable to expect a random developer to build AND MAINTAIN the entire stack for every project. PaaS makes a lot of sense, especially as a starting place. The solution is either to have a PaaS provider who ruthlessly eliminates single points of failure (there isn't one, currently), or use some sta…

The solution is either to have a PaaS provider who ruthlessly eliminates single points of failure (there isn't one, currently) This isn't possible by definition, right? The PaaS provider itself becomes the single point of failure. Rings a bit like a "Who created God?" argument. "What single entity can I use to defend against failures by a single entity?"

Theoretically, sure, but it's an engineering and economic thing.

You can mitigate specific risks, and you try to prioritize those based on cost, frequency, and severity. If there were a great redundant provider with good authentication on accounts, a strong balance sheet and business, and sane policies on managing accounts, you would be fairly safe using just that provider. After all, you could always get a court order to cease providing services, yourselves, like if you do something some troll has patented. It kind of depends on your application, too -- if I were doing a wikileaks, a bitcoin exchange or torrent site or some other legally at risk business, I'd want country-level separation across multiple providers, at least as a cold backup. Casual game for facebook or mobile, not really much of a concern.

Re: Heroku is down

#104

Post by ralphleon »

joevandyk wrote: post #84
ralphleon wrote: post #62

It's so scary to think that our entire startup almost died with this glitch. Our enterprise customers use our product in the mornings... this happened at 9-fucking-am right during our peak hours. Obviously it's our fault for not having a redundant system / for trusting heroku. Though I still can't help but be a little pissed as I email 100 people about how sorry I am for their service disruption. Heroku didn't email…

You can subscribe to notifications on http://status.heroku.com .

Thanks so much! Though I can't imagine anyone who wouldn't want these notifications...

Re: Heroku is down

#105

Post by moe »

ricardobeat wrote: post #55
wheels wrote: post #48

Earlier quoted context omitted.

Honestly, that's pious bullshit. It's the thing people keep saying every time Heroku gets mentioned is that it'd do better than you would yourself for availability, but you'd have to be borderline incompetent to have as much downtime with a more traditional VPS / hardware hosting to match the combined Heroku / AWS downtime. We moved all but one of our Rails apps off of Heroku precisely because of the frequent downtim…

They are at 99.97% uptime. It's not that easy to get to even 99.5%, just the flakiness of a standard uplink will put you below that.

It's not that easy to get to even 99.5%

99.9% is very doable. If your sites have less then you should consider hiring better staff.

Also the heroku figure (99.97%) is a lie - just skim their status-page.

Re: Heroku is down

#106

Post by philip1209 »

garindra wrote: post #99
pbreit wrote: post #75

Earlier quoted context omitted.

Also, careful with Cloudflare SSL. It only secures between the browser and Cloudflare's data center. The connection between your server and Cloudflare remains non-SSL.

You actually have the option to enable SSL both on (customer->cloudflare) and (cloudflare->your server).

And, as I understand it you, you can do:

customer=> Cloudflare SSL

then

cloudflare=> Heroku SSL through Heroku Piggyback

Re: Heroku is down

#107

Post by philip1209 »

garindra wrote: post #99
pbreit wrote: post #75

Earlier quoted context omitted.

Also, careful with Cloudflare SSL. It only secures between the browser and Cloudflare's data center. The connection between your server and Cloudflare remains non-SSL.

You actually have the option to enable SSL both on (customer->cloudflare) and (cloudflare->your server).

And, as I understand it you, you can do:

customer=> Cloudflare SSL

then

cloudflare=> Heroku SSL through Heroku Piggyback

Post reply on HN