Live data from Hacker News

Cloudflare outage on February 20, 2026

blog.cloudflare.com

101–110 of 135 posts

Re: Cloudflare outage on February 20, 2026

#101
post #18

Earlier quoted context omitted.

The featured blog post where one of their senior engineering PMs presented an allegedly "production grade" Matrix implementation, in which authentication was stubbed out as a TODO, says it all really. I'm glad a quarter of the internet is in such responsible hands.

Matrix doesn't actually define how one should do authentication though... every homeserver software is free to implement it however they want.

the main bit of auth which was left unimplemented on matrix-workers was the critical logic which authorizes traffic over federation: https://spec.matrix.org/latest/server-server-api/#authorizat...

Auth for clients is also specified in the spec - there is some scope for homeservers to freestyle, but nowadays they have to implement OIDC: https://spec.matrix.org/latest/client-server-api/#client-aut...

Re: Cloudflare outage on February 20, 2026

#102

Earlier quoted context omitted.

I’ve had a lot of problems lately. Basic things are failing and it’s like product isn’t involved at all in the dash. What’s worse? The support.. the chat is the buggiest thing I’ve ever seen.

don't worry, if it gets much worse the ceo will just throw all of support under the bus again. it will surely get better.

How about accurate billing info. The ux can’t even figure out we’re annually not monthly. Maybe the AI slop will continue to miscount resources and cost you revenue or piss off a customer when the dashboards they been using don’t match the invoice

Re: Cloudflare outage on February 20, 2026

#103
The code they posted doesn't quite explain the root cause. This is a good study case for resilient API design and testing.

They said their /v1/prefixes endpoint has this snippet:

  if v := req.URL.Query().Get("pending_delete"); v != "" {
      // ignore other behavior and fetch pending objects from the ip_prefixes_deleted table
      prefixes, err := c.RO().IPPrefixes().FetchPrefixesPendingDeletion(ctx)
      
      [..snip..]
  }
What's implied but not shown here is that endpoint normally returns all prefixes. They modified it to return just those pending deletion when passing a pending_delete query string parameter.

The immediate problem of course is this block will never execute if pending_delete has no value:

  /v1/prefixes?pending_delete   
This is because Go defaults query params to empty strings and the if statement skips this case. Which makes you wonder, what is the value supposed to be? This is not explained. If it's supposed to be:

  /v1/prefixes?pending_delete=true   
Then this would work, but the implementation fails to validate this value. From this you can infer that no unit test was written to exercise the value:

  /v1/prefixes?pending_delete=false   
The post explains "initial testing and code review focused on the BYOIP self-service API journey." We can reasonably guess their tests were passing some kind of "true" value for the param, either explicitly or using a client that defaulted param values. What they didn't test was how their new service actually called it.

So, while there's plenty to criticize on the testing front, that's first and foremost a basic failure to clearly define an API contract and implement unit tests for it.

But there's a third problem, in my view the biggest one, at the design level. For a critical delete path they chose to overload an existing endpoint that defaults to returning everything. This was a dangerous move. When high stakes data loss bugs are a potential outcome, it's worth considering more restrictive API that is harder to use incorrectly. If they had implemented a dedicated endpoint for pending deletes they would have likely omitted this default behavior meant for non-destructive read paths.

In my experience, these sorts of decisions can stem from team ownership differences. If you owned the prefixes service and were writing an automated agent that could blow away everything, you might write a dedicated endpoint for it. But if you submitted a request to a separate team to enhance their service to returns a subset of X, without explaining the context or use case very much, they may be more inclined to modify the existing endpoint for getting X. The lack of context and communication can end up missing the risks involved.

Final note: It's a little odd that the implementation uses Go's "if with short statement" syntax when v is only ever used once. This isn't wrong per se but it's strange and makes me wonder to what extent an LLM was involved.

Re: Cloudflare outage on February 20, 2026

#104
post #78

It's something we debated in our team: if there's an API that returns data based on filters, what's the better behavior if no filters are provided - return everything or return nothing? The consensus was that returning everything is rarely what's desired, for two reasons: first, if the system grows, allowing API users to return everything at once can be a problem both for our server (lots of data in RAM when fetching…

But that query had parameter. They just fucked up parsing it

Re: Cloudflare outage on February 20, 2026

#105
post #87
post #57

Earlier quoted context omitted.

Thats actively malicious. I understand not going out of your way to catch the LLMs' bugs so as to show the folly of the initiative, but actively sabotaging it is legitimately dangerous behavior. Its acting in bad faith. And i say this as someone who would mostly oppose such an initiative myself I would go so far as to say that you shouldnt be employed in the industry. Malicious actors like you will contribute to an e…

Forcing developers to use unsafe LLM tools is also malicious. This is completely ethical to me. Not commenting on legality.

I dont like it either but its not malicious. The LLM isnt accessing your homeserver, its accessing corporate information. Your employer can order you to be reckless with their information, thats not malicious, its not your information. You should CYA and not do anything illegal even if your asked. But using LLMs isnt illegal. This is bad faith argument

Re: Cloudflare outage on February 20, 2026

#107

The code they posted doesn't quite explain the root cause. This is a good study case for resilient API design and testing. They said their /v1/prefixes endpoint has this snippet: if v := req.URL.Query().Get("pending_delete"); v != "" { // ignore other behavior and fetch pending objects from the ip_prefixes_deleted table prefixes, err := c.RO().IPPrefixes().FetchPrefixesPendingDeletion(ctx) [..snip..] } What's implied…

> But there's a third problem, in my view the biggest one, at the design level. For a critical delete path they chose to overload an existing endpoint that defaults to returning everything. This was a dangerous move. When high stakes data loss bugs are a potential outcome, it's worth considering more restrictive API that is harder to use incorrectly. If they had implemented a dedicated endpoint for pending deletes they would have likely omitted this default behavior meant for non-destructive read paths.

Or POST endpoint, with client side just sending serialized object as query rather than relying that the developer remembers the magical query string.

Re: Cloudflare outage on February 20, 2026

#108
post #64
post #57

Earlier quoted context omitted.

Thats actively malicious. I understand not going out of your way to catch the LLMs' bugs so as to show the folly of the initiative, but actively sabotaging it is legitimately dangerous behavior. Its acting in bad faith. And i say this as someone who would mostly oppose such an initiative myself I would go so far as to say that you shouldnt be employed in the industry. Malicious actors like you will contribute to an e…

Might be but sometimes you don’t have another choice when employers are enforcing AIs which have no „feeling“ for context of all business processes involved created by human workers in the years before. Those who spent a lot of love and energy for them mostly. And who are now forced to work against an inferior but overpowered workforce. Don’t stop sabotaging AI efforts.

Honestly i kinda like the aesthetic of cyberanarchism, but its not for me. It erodes trust

Re: Cloudflare outage on February 20, 2026

#109
post #78

It's something we debated in our team: if there's an API that returns data based on filters, what's the better behavior if no filters are provided - return everything or return nothing? The consensus was that returning everything is rarely what's desired, for two reasons: first, if the system grows, allowing API users to return everything at once can be a problem both for our server (lots of data in RAM when fetching…

> to have a separate method named accordingly, like ListAllObjects, without any filters

For me it's like `filter1=*`

Re: Cloudflare outage on February 20, 2026

#110
post #12

is this blog post LLM generated? the explanation makes no sense: > Because the client is passing pending_delete with no value, the result of Query().Get(“pending_delete”) here will be an empty string (“”), so the API server interprets this as a request for all BYOIP prefixes instead of just those prefixes that were supposed to be removed. The system interpreted this as all returned prefixes being queued for deletion.…

better explanation here https://news.ycombinator.com/item?id=47106852

but in short they are changing whether string is empty, and query string "pending_delete" is same as "pending_delete=" and will return empty

Or, if they specified `/v1/prefixes?pending_delete=potato` it would return "correct" list of objects to delete

Or in other words "Go have types safety, fuck it, let's use strings like in '90s PHP apps instead"

Post reply on HN