Live data from Hacker News

Cloudflare Down Again – and DownDetector Is Also Down

news.ycombinator.com

161–170 of 177 posts

Re: Cloudflare Down Again – and DownDetector Is Also Down

#161

The old guard has left as they we too much of an expense in this cost-cutting age... without mentors, crap creeps in and now we are seeing what happens when people don't know how things work, are in charge...

It may not be true everywhere, but at my company we 100% had more SEVs after two rounds of RIFs. We are talking simple statistics of SEVs per month plotted against RIFs.

Re: Cloudflare Down Again – and DownDetector Is Also Down

#162

Earlier quoted context omitted.

Knowledge + tech skills are not the only factor that lead to subpar outcomes with these scenarios. In my experience the thing that causes the most problems with H1Bs is the weak English and related communication issues.

In my experience, the communication problems stem from the Americans who expect perfect English from all others. English is spoken across the entire business world between people for whom it is not their first language. The accents and broken English is epic in many organizations. Yet they work through it and get things done together. If you work harder at taking the burden upon yourself to understand others, you mig…

That’s just saying the same thing. American companies have engineering quality loss when they try to collaborate with people they can’t communicate well with. Whether it’s dumb Americans or poor ESL, it’s not really relevant to the outcome because it’s the same.

Re: Cloudflare Down Again – and DownDetector Is Also Down

#163

There are too many of that recently. Cloudflare is starting to look amateurish. Can't they test their stuff properly before deploying it to production?

Maybe they test with shopify.com before the deploy it to their important customers ;)

Re: Cloudflare Down Again – and DownDetector Is Also Down

#164
I have a tentative take, and I kind of feel stupid for even claiming this, since I don't work in Cloud-ops, or whatever, but it's fun to try to participate, and I spent some time articulating what i think is a good perspective on Cloudflare now a days, and as psychologist, I am primary interested in the psychology of things.

Basically, my take is: It’s not a technical monoculture; it’s a billing psychology + inertia culture.

I dont think the internet is fragile simply because Cloudflare is so ubiquitous, because that view ignores the economic factor of why people choose them. The situation is really a perfect bi-modal distribution: at the low end, you have hobbyists and personal sites who use Cloudflare because it is the only viable free option, and at the extreme high end, you have massive enterprises that truly need that specific global capacity to scrub terabits of attack traffic.

However, I think the following perspective is important: For the vast middle ground of the internet—most standard businesses and SaaS platforms—Cloudflare could be viewed as redundant. If you are hosting on AWS, Google Cloud, or Azure, you are already sitting behind world-class infrastructure protection that rivals anything Cloudflare offers. The reason this feels like a dangerous monoculture isn't because Google or Amazon can't protect you, but rather because Cloudflare wins on the psychology of billing. They sell a flat-rate insurance policy against attacks, whereas the cloud giants charge for usage, which scares people.

Ultimately, the internet isn't suffering from a lack of technical alternatives to DDoS protection, nor is Cloudflare a NECESSARY single point of failure; it is just suffering from a market preference for predictable invoices over technical redundancy, and inertia, leading to an extremely high usage of Cloudflare. So basically: Even though we are currently relying a lot on Cloudflare, we are far from vendor lock-in, and there is a clear path to live without them, given that there are many alternatives.

Maybe we could view this as a good thing, since basically medium to large-scale enterprises efficiently subsidize small and hobby-level actors? So to summerize: The 2018-era "just use Cloudflare for everything" advice is outdated, and the following is a better philosopy: If you're tiny: Cloudflare free tier is still a no-brainer. If you're huge and actually get attacked: pay for Cloudflare Enterprise or equivalent.

If you're anywhere in between: seriously consider whether you need it at all. The hyperscalers are good enough, and removing Cloudflare can actually improve your availability (fewer moving parts).

I think Cloudflare thinks this way too, which is why they've been pushing Zero Trust, Workers, WARP, Access, and Magic Transit, to become the default network stack for companies, not just the default firewall.

/wall-of-text

Re: Cloudflare Down Again – and DownDetector Is Also Down

#166
post #164

I have a tentative take, and I kind of feel stupid for even claiming this, since I don't work in Cloud-ops, or whatever, but it's fun to try to participate, and I spent some time articulating what i think is a good perspective on Cloudflare now a days, and as psychologist, I am primary interested in the psychology of things. Basically, my take is: It’s not a technical monoculture; it’s a billing psychology + inertia…

Bah, I think I double-posted. Is this visible? :o

Re: Cloudflare Down Again – and DownDetector Is Also Down

#167
post #33

> A change made to how Cloudflare's Web Application Firewall parses requests caused Cloudflare's network to be unavailable for several minutes this morning. This was not an attack; the change was deployed by our team to help mitigate the industry-wide vulnerability disclosed this week in React Server Components. We will share more information as we have it today.

When are they going to figure out that canary deployments are a good idea? Do they just push every change straight to prod globally?

Re: Cloudflare Down Again – and DownDetector Is Also Down

#168

Earlier quoted context omitted.

In the chain of events that led to Cloudflare's largest ever outage, code they'd rewritten from C to Rust was significant factor. There are, of course, other factors that meant the Rust-based problem was not mitigated. They expected a maximum config size but an upstream error meant it was much larger than normal. Their Rust code parsed a fraction of the config, then did ".unwrap()" and panicked, crashing the entire p…

This is generally BS apologetics for C. If that was in C this would have just been overrunning the statically allocated memory amount and would have resulted in a segfault. Rust did its job and forced them to return an error from the lower function. They explicitly called a function to crash if that returned an error. That’s not a rust problem.

We don't know how the C program would have coped. It could equally have ignored the extra config once it reached its maximum, which would cause new problems but not necessarily cause an outage. It could've returned an error and safely shut down the whole program (which would result in the same problem as Rust panicking).

What we do know is Cloudflare wrote a new program in Rust, and never tested their Rust program with too many config items.

You can't say "Rust did its job" and blame the programmer, any more than I can say "C did its job" when a programmer tells it to write to the 257th index of a 256 byte array, or "Java did its job" when some deeply buried function throws a RuntimeException, or "Python did its job" when it crashes a service that has been running for years because for the first time someone created a file whose name wasn't valid UTF-8.

Footguns are universal. Every language has them, including Rust.

You have to own the total solution, no matter which language you pick. Switching languages does not absolve you of this. TANSTAAFL.

Re: Cloudflare Down Again – and DownDetector Is Also Down

#169
post #126

Earlier quoted context omitted.

Cloudflare is widely used because it's the easiest way to run a website for free or expose local services to internet. I think for most cloudflare users, the ddos protection is not the main reason they're using it.

Cloudflare hosts websites for free?

Yup, the free plan is quite generous.

Re: Cloudflare Down Again – and DownDetector Is Also Down

#170
post #45

https://downdetectorsdowndetectorsdowndetectorsdowndetector.... reports that https://downdetectorsdowndetectorsdowndetector.com/ is down, guessing downdetectorsdowndetectorsdowndetector runs via cloudflare!

I always wondered who detects when downdetector.com is down.

    Each downed page diminishes me,
    for I am involved in WANkind.
    Therefore, ping not to know
    for whom the downdetector detects,
    it detects for thee.
Post reply on HN