Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare was down

cloudflare.com

Post reply on HN
Print view View original

531–540 of 560 posts

Re: Cloudflare was down

#531

Post by koakuma-chan »

mike_hearn wrote: post #528
koakuma-chan wrote: post #431

Earlier quoted context omitted.

I am not a C# guy, but I wrote a lot of Java back in the day, and I can authoritatively tell you that it has so-called "checked exceptions" that the compiler forces you to handle. However, it also has "runtime exceptions" that you are not forced to handle, and they can happen any where and any time. Conceptually, it is the same as error versus panic in Rust. One such runtime exception is the notorious `java.lang.Null…

In practice, programs written in managed languages don't crash in the sense of aborting the entire process. Exceptions are usually caught at the top level (both checked and unchecked) and then logged, usually aborting the whole unit of work. For trapping a bad data load it's as simple as: try { data = loadDataFile(); } catch (Exception e) { LOG.error("Failed to load new data file; continuing with old data", e); } Thi…

Here is the Java equivalent of what happened in that Cloudflare Rust code:

  try {
    data = loadDataFile();
  } catch (Exception e) {
    LOG.error("Failed to load new data file", e);
    System.exit(1);
  }
So the "bad data load" was trapped, but the programmer decided that either it would never actually occur, or that it is unrecoverable, so it is fine to .unwrap(). It would not be any less idiomatic if, instead of crashing, the programmer decided to implement some kind of recovery mechanism. It is that programmer's fault, and has nothing to do with Rust.

Also, if you use general try-catch blocks like that, you don't know if that try-catch block actually needs to be there. Maybe it was needed in the past, but something changed, and it is no longer needed, but it will stay there, because there is no way to know unless you specifically look. Also, you don't even know the exact error types. In Rust, the error type is known in advance.

Re: Cloudflare was down

#532

Post by sega_sai »

After reading the post, my personal takeaway (not being expert) is that there are simply so many moving parts/configuration options that the complexity of the whole system is too high. I think without some sort of formal validation/enumeration of all possible states of the system the reliability is simply not possible. Whether this formal verification is achievable, I don't know.

Re: Cloudflare was down

#533

Post by mvdtnz »

karmakurtisaani wrote: post #85
pm90 wrote: post #25

This is not good. One major outage? Something exceptional. Several outages in a short time? As someone thats worked in operations, I have empathy; there are so many “temp havks” that are put in place for incidents. but the rest of the world won’t… they’re gonna suffer a massive reputation loss if this goes on as long as the last one.

Probably fired a lot of their best people in the past few years and replaced it with AI. They have a de-facto monopoly, so we'll just accept it and wait patiently until they fix the problem. You know, business as usual in the grift economy.

This didn't happen at all. You're just completely making shit up.

Re: Cloudflare was down

#534

Post by runeb »

yoctosec wrote: post #382
runeb wrote: post #202

Earlier quoted context omitted.

Checkout tailscale

And what about a website I want to make public? I'm just concerned about my IP being visible, like for my personal website or my searxng instance

Tailscale Funnel, but might need a paid account

Re: Cloudflare was down

#536

Post by hinkley »

uyzstvqs wrote: post #453
luastoned wrote: post #439

From the incident page: A change made to how Cloudflare's Web Application Firewall parses requests caused Cloudflare's network to be unavailable for several minutes this morning. This was not an attack; the change was deployed by our team to help mitigate the industry-wide vulnerability disclosed this week in React Server Components. We will share more information as we have it today. https://www.cloudflarestatus.com…

Ah yes, Cloudflare's worst enemy: The configuration change.

On fridays, yes.

Re: Cloudflare was down

#537

Post by DirkH »

I feel like all the BS we were taught about architecture design principles multi-AZ, failover strategies, graceful degradation etc was gaslighting us all into thinking any of out work on it actually matters.

This isn't true, but it feels like this when the entire engineering world order seems to actually run on single-point-of-failures where one CEO just messages another when some 3rd party is down. And reputational risk here is completely safeguarded because as long as everyone is down you are fine. Use a service everyone uses and it goes down = no reputational risk. Use a more robust architecture and make some mistake = massive reputational risk and everyone asks why you don't use what everyone else uses.

Blind leading the blind and all that.

Re: Cloudflare was down

#538

Post by mike_hearn »

koakuma-chan wrote: post #531
mike_hearn wrote: post #528

Earlier quoted context omitted.

In practice, programs written in managed languages don't crash in the sense of aborting the entire process. Exceptions are usually caught at the top level (both checked and unchecked) and then logged, usually aborting the whole unit of work. For trapping a bad data load it's as simple as: try { data = loadDataFile(); } catch (Exception e) { LOG.error("Failed to load new data file; continuing with old data", e); } Thi…

Here is the Java equivalent of what happened in that Cloudflare Rust code: try { data = loadDataFile(); } catch (Exception e) { LOG.error("Failed to load new data file", e); System.exit(1); } So the "bad data load" was trapped, but the programmer decided that either it would never actually occur, or that it is unrecoverable, so it is fine to .unwrap(). It would not be any less idiomatic if, instead of crashing, the p…

Yes, I know. But nobody writes code like that in Java. I don't think I've ever seen it outside of top level code in CLI tools. Never in servers.

> It is that programmer's fault, and has nothing to do with Rust.

It's Rust's fault. It provides a function in its standard library that's widely used and which aborts the process. There's nothing like that in the stdlibs of Java or .NET

> Also, if you use general try-catch blocks like that, you don't know if that try-catch block actually needs to be there.

I'm not getting the feeling you've worked on many large codebases in managed languages to be honest? I know you said you did but these patterns and problems you're raising just aren't problems such codebases have. Top level exception handlers are meant to be general, they aren't supposed to be specific to certain kinds of error, they're meant to recover from unpredictable or unknown errors in a general way (e.g. return a 500).

Re: Cloudflare was down

#539

Post by venturecruelty »

zwnow wrote: post #227

I love it, and we wont learn from this again :-) Looking forward for the 3rd outage in a few weeks.

We should have a weekly thread for it! It can be a fun meetup. Hell, move it to a cafe or a pub or something, and we can use it as a chance to disconnect from the internet and talk to people face to face. Instead of saying "touch grass", we can say "get Cloudflared".

Re: Cloudflare was down

#540

Post by The_President »

glimshe wrote: post #461

Not only they make my browsing experience a LOT worse (seconds per site for bot detection and additional "are you human" clicks even without VPNs), now they are bringing the entire Internet down. They don't deserve the position they currently have.

The whole “not a bot” prompt every three hours seems like it has potential to get out of the way more often.
Post reply on HN