Cloudflare API Down

It annoys me to no end that the idea of metro-ISPs never took off. If somebody in the same city as me wants to connect to any services I host at home they first have to round-trip to my ISPs core-routers and then back to me.

Now this all happens very quickly within 5-6 ms but it could be 1ms or less in an ideal world. I understand why we cannot have nice things though (cost mainly).

I'm not an expert in network security, so I can only give you an idea. The correct way to do things depends on so many factors, but the TLDR n probably use multiple ways to identify malicious traffic, and then block it as early as possible. Dropping all traffic from abusive IP addresses with netfilter is where you start because it's easy/simple. Then you can move on to grouping by ASN, and dropping closer to a load balancer.

The problem (as far as I know) that cloudflare actually provides, is being able to identify abusive networks, and making them prove they're not bots before they reach your server. If you can't even identify the abusive traffic, you don't have many options other than cloudflare.

The point of the internet is to provide a robust communictions platform. If fundamental infrastructure of that communcations platform can be abused to deny communications, and further that the abuse can continue with the root cause unaddressed for decades, then the platform is broken.

The fact that routing is designed to go around damage is orthoganal to this, and has not bearing on the fact that the communications platform can be used against itself to prevent communications (via spoofed IPs).

For literal decades partial solutions to the spoofing problem have been known - rp filtering would eliminate a lot of problems yet still isn't close to universal.

BGP has been vulnerable to all sorts of simple human mistakes for decades and decade old solutions like IRR are only slowly being adopted because many of the people that run the internet are too busy pretending they are important and good at building systems to actually make the systems good. When those same simple mistakes are intentional, all sorts of IP traffic can be spoofed including full TCP connections.

The fact that there isn't a widely supported way for the consequences of spoofing to be mitigated without paying out the nose for a 3rd party service is pretty broken too. Allowing destinations be overwhelmed without any sort of backpressure or load shedding is a fundamental flaw in "get packets to destination no matter what". An AS should be able to say "I no longer want packets from this subnet", and have it honored along the entire path. This should be a core feature, not an add-on from some providers.

The internet does work as designed, however it's folly to think that the first attempt at building something so different to anything that came before it is the best way to do it and reusing to address design decisions is fundamentally broken.

Well, sort of. I feel that something like Interplanetary File System (IPFS) should have been the original web. Or at least, after BitTorrent arrived, we should have all moved to content-addressable memories because they automatically handle caching and scaling. That would have required solving circles of trust with potential improvements to SSL certificates so that they don't require a central authority, like a distributed version of letsencrypt.org. It also would have needed improvements to DNS for making domain-data pairs instead of domain-IP. And at the very least, IPv6 should have improved UDP so that it "just works" everywhere and doesn't have to deal with NAT hole punching. That stuff represents the real work of rolling out a true P2P internet.

Since we did none of that, we cemented the centralized web we have today, full of walled gardens funding the copyright nanny state. So that people are so aghast at seeing the real internet running on stuff like TikTok that their first instinct is to ban it.

To me, it feels like we're living under a bizarro version of the internet, so far removed from its academic/declarative/data-driven roots that it's almost unrecognizable. Javascript is just distributed async desktop programming now, reminiscent of the C++ days of the late 1980s, complete with all of the nondeterminism that we worked so hard to get away from. Paid services like Cloudflare and 5G are a symptom of that inside-the-box thinking. If we had real distributed P2P through our routers and cell phones, we probably wouldn't need ISPs today, because we'd be accustomed to gigabit download speeds and would only perceive long distance as slow and worth paying for. In which case, maybe the government should pay to maintain the backbone and perhaps satellite internet like Starlink. Which would be a good thing IMHO, because at least there would be oversight and we wouldn't have to worry about billionaire narcissists making the satellites highly reflective and visible to the naked eye, like carving a McDonalds sign on the moon.

Thanks for sharing, that was a very entertaining read and gave me fond memories of the decade or so I worked in a phone switch.

My office was in a space that was shared by a large legacy phone switch, SONET node, and part of our regional data center but I worked in infrastructure doing software development. My proximity[0] meant I ended up being used to support larger infrastructure efforts at times, but it usually just meant I got a lot of good ... stories.

I wonder if there's a collection of Data Center centric "Daily WTF" stories or something similar.

For me, I think my favorite is when we had multiple unexplained power failures very late at night in our test/management DC[0]. It turned out "the big red kill switch" button behind the plexiglass thing designed to make sure someone doesn't accidentally "lean into it and shut everything off" was mistaken for the "master light switch" by the late night cleaning crew. Nobody thought about "the cleaning crew" because none of the other DCs allowed cleaning crew anywhere near them but this was a test switch (someone forgot about the other little detail). If memory serves, it took a few outages before they figured it out. The facilities manager actually hung around one night trying to witness it only to have the problem not happen (because the cleaning lady didn't turn the lights off when people were there, duh!). I'd like to say that it was almost a "maybe bugs/animals/ghosts are doing it" impulse that caused them to check the cameras but it was probably also the pattern being recognized as "days coinciding with times that the late night cleaning crew does their work."

Outside of that, the guy who made off with something like 4 of these legacy switch cards because some fool put a door stop on the door while moving some equipment in was probably really excited when he found out they were valued at "more than a car" but really disappointed when he put them on eBay for something like $20,000 (which was, I wanna say at least a 50% markdown), was quickly noticed by the supplier[2] was arrested and we were awaiting the return of our hardware.

[0] Among many other things due to a diverse 17-year career, there, but mostly just because I was cooperative/generally A-OK with doing things that "were far from my job" when I could help out my broader organization.

[1] When that went down, we couldn't connect to the management interfaces of any of the devices "in the network". It's bad.

[2] Alarms went off somewhere -- these guys know if you are using their crap, you're stuck with their crap and they really want you stuck paying them for their crap. I'm fairly certain the devices we used wouldn't even function outside of our network but I don't remember the specifics. AFAIK, there's no "pre-owned/liquidation-related market" except for stripping for parts/metals. When these things show up in unofficial channels, they're almost certainly hot.