Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A postmortem of three recent issues

anthropic.com

Post reply on HN
Print view View original

111–120 of 120 posts

Re: A postmortem of three recent issues

#111

Post by bobbylarrybobby »

vintagedave wrote: post #94
mike_hearn wrote: post #91

The most interesting thing about this is the apparent absence of unit tests. The test for the XLA compiler bug just prints the outputs, it's more like a repro case than a unit test in the sense that it'd be run by a test harness and have coverage tracked. And the action items are simply to lean more aggressively into evals. Although unit testing an entire LLM is not really feasible right now, all these bugs were in s…

I've had to write some detailed prompts and examples to have AI generate the kind of unit tests I want in Python. I've seen the assertions on types alone too. I want assertions on values and more. Even more than that, AI tends to mock _everything_. Mocking is useful, but the more real code a unit test invokes, the better, because the risk is not only the code itself but its interactions, the interface. Yet AI in Pyth…

When asked to write UI tests (playwright), I've seen Claude Code do essentially the following:

const elem = document.querySelector(".foo"); // actual element that exists elem.innerHTML = ''; const child = elem.locator(".bar"); // child we actually want to test for expect(child).toExist()

Gee thanks Claude, what a great test...

Re: A postmortem of three recent issues

#112

Post by whatevaa »

redman25 wrote: post #102
vintagedave wrote: post #94

Earlier quoted context omitted.

I've had to write some detailed prompts and examples to have AI generate the kind of unit tests I want in Python. I've seen the assertions on types alone too. I want assertions on values and more. Even more than that, AI tends to mock _everything_. Mocking is useful, but the more real code a unit test invokes, the better, because the risk is not only the code itself but its interactions, the interface. Yet AI in Pyth…

I wish I had 100 upvotes to give you. Weak, heavily mocked tests are my biggest pet peave. Test “quality” is important and not something a lot of devs pay attention to. I’ve found myself preferring integration tests or unit tests with a “real” database set up because the tests are much more effective. If you design them right, they don’t even need to be slower.

They will be locally if you have to also run 3 virus scanners :)

Re: A postmortem of three recent issues

#113

Post by data-ottawa »

renewiltord wrote: post #77
data-ottawa wrote: post #43

With all due respect to the Anthropic team, I think the Claude status page[1] warrants an internal code red for quality. There were 50 incidents in July, 40 incidents in August, and 21 so far in September. I have worked in places where we started approaching half these numbers and they always resulted in a hard pivot to focusing on uptime and quality. Despite this I'm still a paying customer because Claude is a fanta…

This is always why you should put as few incidents on status page as possible. People's opinion will drop and then the negative effect will fade over time. But if you have a status page then it's incontrovertible proof. Better to lie. They'll forget. e.g. S3 has many times encountered increased error rate but doesn't report. No one says anything about S3. People will say many things, but their behaviour is to reward…

It’s good they update the status page, but the issues are noticeable without it.

Re: A postmortem of three recent issues

#114

Post by nadis »

itsgrimetime wrote: post #93

Wish they would have included what the actual failure mode was. I’ve been having issues where Claude Code will just hang after running some tool call, was that caused by one of these bugs?

I've noticed the same issue increasingly in the past couple of days; I'm guessing it's unrelated (but still a bug) but hope I'm wrong. The number of times I've had to ask variations of "why did you stop? proceed" has really escalated.

Re: A postmortem of three recent issues

#115

Post by FossQuestion »

cyanf wrote: post #25

> On August 29, a routine load balancing change unintentionally increased the number of short-context requests routed to the 1M context servers. At the worst impacted hour on August 31, 16% of Sonnet 4 requests were affected. Interesting, this implies that the 1M context servers performs worst at low context. Perhaps this is due to some KV cache compression, eviction or sparse attention scheme being applied on these…

The key issue is that their post-mortem never explained what went wrong on two out of three issues.

All I know is that my requests can now travel along three completely different code paths, each on its own stack and tuned differently. Those optimizations can flip overnight, independent of any model-version bump—so whatever worked yesterday may already be broken today.

I really don't get the praise that they are getting for this postmortem, it only made me more annoyed.

Re: A postmortem of three recent issues

#116

Post by FossQuestion »

levocardia wrote: post #16
bravetraveler wrote: post #9

> We don't typically share this level of technical detail about our infrastructure, but the scope and complexity of these issues justified a more comprehensive explanation. Layered in aggrandizing. You host a service, people give you money.

No, what that statement means is "we know that if we just say 'we weren't downgrading performance to save money', you won't believe us, so here is a deep dive on the actual reason it happened"

What deep dive? They explained one of the three issues. and never offered a real solution to the problems. their action items list is just "We will test better next time..."

Re: A postmortem of three recent issues

#117

Post by vintagedave »

bobbylarrybobby wrote: post #111
vintagedave wrote: post #94

Earlier quoted context omitted.

I've had to write some detailed prompts and examples to have AI generate the kind of unit tests I want in Python. I've seen the assertions on types alone too. I want assertions on values and more. Even more than that, AI tends to mock _everything_. Mocking is useful, but the more real code a unit test invokes, the better, because the risk is not only the code itself but its interactions, the interface. Yet AI in Pyth…

When asked to write UI tests (playwright), I've seen Claude Code do essentially the following: const elem = document.querySelector(".foo"); // actual element that exists elem.innerHTML = ' '; const child = elem.locator(".bar"); // child we actually want to test for expect(child).toExist() Gee thanks Claude, what a great test...

Same. Drives me up the wall. I’m writing my own coding agent now and I’m baking into it prompts against all the anti patterns I’ve see.

Re: A postmortem of three recent issues

#118

Post by lanstin »

renewiltord wrote: post #77
data-ottawa wrote: post #43

With all due respect to the Anthropic team, I think the Claude status page[1] warrants an internal code red for quality. There were 50 incidents in July, 40 incidents in August, and 21 so far in September. I have worked in places where we started approaching half these numbers and they always resulted in a hard pivot to focusing on uptime and quality. Despite this I'm still a paying customer because Claude is a fanta…

This is always why you should put as few incidents on status page as possible. People's opinion will drop and then the negative effect will fade over time. But if you have a status page then it's incontrovertible proof. Better to lie. They'll forget. e.g. S3 has many times encountered increased error rate but doesn't report. No one says anything about S3. People will say many things, but their behaviour is to reward…

S3 autoscales, so any time the load increases you can see 5xx and 429 errors, but it flexes up in a few hours. That’s not exactly an incident, sort of Works as Designed.

The first time you write a multithreaded utility to do something in account with S3 you will see this, and have to write the temporary back off code.

Re: A postmortem of three recent issues

#119

Post by l1n »

Razengan wrote: post #110

Anthropic seems to have a "We're special" syndrome: * No Sign in with Apple on the website, so tough luck if you signed up on iOS via that * Can't buy via in-app purchase so have to give them your card * Can't remove your payment method from your account * Hard to get support after buying a subscription * Asking Claude questions about Claude itself such as privacy just gives you links to their website What even is th…

> so tough luck if you signed up on iOS via that

FYI that you can sign in with your private relay email address. It's annoying though.

Re: A postmortem of three recent issues

#120

Post by bangaladore »

ants_everywhere wrote: post #69
vlovich123 wrote: post #39

The value of figuring out how to make their LLM serving deterministic might help them track this down. There was a recent paper about how the received wisdom that kept assigning it to floating point associativity actually overlooked the real reasons for non-determinism [1]. [1] https://thinkingmachines.ai/blog/defeating-nondeterminism-in...

network traffic and machine load aren't deterministic. I think for the near term, getting full determinism (e.g. for auditing) is going to only be feasible for batch jobs that are not cost sensitive. A google search isn't deterministic. Neither is loading upvote count on social media. It's common advice in distributed systems to have a graceful degradation state instead of becoming unavailable. That wouldn't be possi…

Same input -> Model -> Same output

My understanding is non-determinism is usually due to explicitly introducing randomness while sampling. No reason why they couldn't use a static seed.

And if not, that's something that they should solve, unintentional non-determinism is a bug, not a feature.

Post reply on HN