Anyone remember big iron and servers with uptimes of 5 or 7 nines?
I mean: it used to be a thing. Now we have the cloud.
Ask HN: Cloudflare Workers are down?
71–80 of 87 posts
Re: Ask HN: Cloudflare Workers are down?
#72It's probably bad that I noticed this just due to a large percentage of my regular online-habits suddenly breaking. I liked the old internet where websites just broke one at a time.
That was before ddos became common and cheap to execute.
Re: Ask HN: Cloudflare Workers are down?
#73Earlier quoted context omitted.
Not related. (I am the PM lead for Workers databases & storage)
Is there a postmortem coming ? Would you be able to tell us what happened at a high level ?
See my comment here: https://news.ycombinator.com/item?id=38075877
(We’ll share more when we can)
Re: Ask HN: Cloudflare Workers are down?
#74Earlier quoted context omitted.
It’s only reasonable to be angry but do try to remember that the people fixing this are people like you who showed up at work to build something and are instead dealing with a fire. Ask their bosses about how they got in that situation but be nice to them, they’re having an even worse day than you are.
Fair enough. They resolved it now and I was in a bit of panic considering our revenue depends on the website. As a developer though, I should have been more sympathetic.
I'm curious, I definitely get the panic. How much does a 30 minute outage cost you, vs how much would it cost to build a solution with some kind of standby that you could fail over to in scenarios like this?
It could be worth it, but if you do the math and it seems like it's not worth it, it could perhaps give you some equanimity the next time it happens?
Re: Ask HN: Cloudflare Workers are down?
#75Earlier quoted context omitted.
Omg. What timing. I feel your pain. We recently migrated to Cloudflare Pages and I was happy at the speed and everything and now this :(. Never had a downtime when I self hosted on my DigitalOcean droplet. damn. Re-considering going back to old school nginx static site hosting.
Well then you haven't used DO that long, I get regular emails about X or Y server needing to go down for maint.
I've used Digital Ocean (and many other hosting providers) for as long as most of them have existed. Most of my servers have been running nearly uninterrupted for many years. Yes, there will be a reboot or move every so often but the uptime is incredibly high.
The idea that single server is capable beat the reliability of a massively distributed system is counter-intuitive and yet usually it's the case.
The average distributed system is a house of cards that can come tumbling down if any one of a number of pieces fails. The average static server is a rock of stability, with very few failure modes.
Re: Ask HN: Cloudflare Workers are down?
#76For any terraform users that may be using code like this: data "cloudflare_ip_ranges" "cloudflare_ipv4_list" {} This is coming back with an empty list on some fields and causing havoc in terraform.
It is shocking to me how bad to non-existent error handling is in most terraform providers. It leads to some remarkably arcane and esoteric error messages
it's shocking how much of a desirable skill it is in devops job roles given its clear deficiencies.
Re: Ask HN: Cloudflare Workers are down?
#77Earlier quoted context omitted.
Fair enough. They resolved it now and I was in a bit of panic considering our revenue depends on the website. As a developer though, I should have been more sympathetic.
I'm curious, I definitely get the panic. How much does a 30 minute outage cost you, vs how much would it cost to build a solution with some kind of standby that you could fail over to in scenarios like this? It could be worth it, but if you do the math and it seems like it's not worth it, it could perhaps give you some equanimity the next time it happens?
Honestly, the impact may be less monetary and more reputation but as a B2B SAAS provider, I agree that it may not have been that dramatic as I made it sound for the 25 mins of downtime. It is just that we never had a downtime this long, EVER in 8+ years of business so I hit the panic button fast. 25 mins seemed like 25 hours if you ask me :).
Re: Ask HN: Cloudflare Workers are down?
#78And just 30 minutes ago we were about to flip the switch on a months long migration to Cloudflare Pages for our new website, I guess some things weren't meant to be :')
Omg. What timing. I feel your pain. We recently migrated to Cloudflare Pages and I was happy at the speed and everything and now this :(. Never had a downtime when I self hosted on my DigitalOcean droplet. damn. Re-considering going back to old school nginx static site hosting.
Did you ever reboot for patches or was it load balanced?
Re: Ask HN: Cloudflare Workers are down?
#79Earlier quoted context omitted.
It is shocking to me how bad to non-existent error handling is in most terraform providers. It leads to some remarkably arcane and esoteric error messages
Terraform error handling as a whole is nuts anyway. Like, I recently tried to delete an ACM cert that still was in use in a Cloudfront distribution - didn't work, but it took 20 minutes for Terraform to recognize that, yes, there's an API error. It shouldn't have come so far given that the API call immediately errors out when trying over the CLI or Web Console, but instead of erroring out, Terraform retried for 20 mi…
This is usually down to provider implementation which switching the core won't help. The provider controls HTTP calls and errors against the relevant service API.
Here are the retries in the provider code https://github.com/hashicorp/terraform-provider-aws/blob/mai...
It's hard coded to "certificateCrossServicePropagationTimeout" which is 20 minutes here https://github.com/hashicorp/terraform-provider-aws/blob/mai...
Re: Ask HN: Cloudflare Workers are down?
#80Earlier quoted context omitted.
Terraform error handling as a whole is nuts anyway. Like, I recently tried to delete an ACM cert that still was in use in a Cloudfront distribution - didn't work, but it took 20 minutes for Terraform to recognize that, yes, there's an API error. It shouldn't have come so far given that the API call immediately errors out when trying over the CLI or Web Console, but instead of erroring out, Terraform retried for 20 mi…
This is usually down to provider implementation which switching the core won't help. The provider controls HTTP calls and errors against the relevant service API. Here are the retries in the provider code https://github.com/hashicorp/terraform-provider-aws/blob/mai... It's hard coded to "certificateCrossServicePropagationTimeout" which is 20 minutes here https://github.com/hashicorp/terraform-provider-aws/blob/mai...
Sure, but Terraform Core doesn't provide any way of getting user feedback in case unexpected situations happen, or aborting while saving the current state, both of which would save me serious amounts of time.