Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A postmortem of three recent issues

anthropic.com

Post reply on HN
Print view View original

81–90 of 120 posts

Re: A postmortem of three recent issues

#81

Post by vlovich123 »

ants_everywhere wrote: post #69
vlovich123 wrote: post #39

The value of figuring out how to make their LLM serving deterministic might help them track this down. There was a recent paper about how the received wisdom that kept assigning it to floating point associativity actually overlooked the real reasons for non-determinism [1]. [1] https://thinkingmachines.ai/blog/defeating-nondeterminism-in...

network traffic and machine load aren't deterministic. I think for the near term, getting full determinism (e.g. for auditing) is going to only be feasible for batch jobs that are not cost sensitive. A google search isn't deterministic. Neither is loading upvote count on social media. It's common advice in distributed systems to have a graceful degradation state instead of becoming unavailable. That wouldn't be possi…

Network traffic and machine load don’t usually impact the output of a pure (in the CS sense of purity) math function (which is what an LLM is) unless you’ve written your system to be sensitive to that.

> to have a graceful degradation state instead of becoming unavailable. That wouldn't be possible in a system that's completely deterministic.

What does this even mean? I see no incompatibility between determinism and your ability to perform the same function more slowly. Determinism just means that the output of the system is solely dependent on the inputs - feed the same inputs get the same outputs. If by degraded state you’re intentionally choosing to change your inputs, that doesn’t change the determinism of your system.

When it is said that LLMs aren’t deterministic, it’s because the output token is dependent on the inputs context and all other contexts processed in the same batch because the kernels are written non-deterministically. If the kernels were written deterministically (so that the output only depended on your input context), then there wouldn’t be a problem and it also wouldn’t change the ability for the system to degrade; it would be deterministic because capturing the input context and random seed would be sufficient. As it stands you’d have to capture the interim states of the other inputs being processed in the same batch and that interim state problem is what makes it non deterministic.

As for Google search, it’s not clear to me it’s non-deterministic. When you Google the exact same thing twice you get exactly the same page of results and selected snippets. That suggests there’s more determinism in the system than you’re giving it credit for.

Re: A postmortem of three recent issues

#82

Post by gabriel666smith »

extr wrote: post #12
deepdarkforest wrote: post #7

Wow. Sneaky. They do not even state the rate of impact for the XLA bug afaik, which affected everyone, not just claude code users, very vague. Interesting. Claude code made almost half a billion so far[1] (>500m in ARR and its like 9 months old) , and 30% of all users have been impacted at least once, just from the first routing bug. Scary stuff. Their post mortem is basically "evaluations are hard, we relied on vibe…

Is your contention that paying for a service entitles you to zero bugs, ever?

We already kind of have a solution for this with SLAs. Humans, being (probably) non-deterministic, also fuck up. An expectation of a level of service is, I think, reasonable. It's not "zero mistakes ever", just as it can't be "zero bugs ever".

We're firmly in the realms of 'this thing is kind of smarter / faster at a task compared to me my employees, so I am contracting it to do that task'.

That doesn't mean 'if it fails, no payment'.

But I think it's too analogous to non-tech-products to hide behind a 'no refunds' policy. It's that good - there are consequences for it, I think.

Re: A postmortem of three recent issues

#84

Post by efskap »

dytyruio wrote: post #44
extr wrote: post #18

Earlier quoted context omitted.

I don't know about you but my feed is filled with people claiming that they are surely quantizating the model, Anthropic is purposefully degrading things to save money, etc etc. 70% of users were not impacted. 30% had at least one message degraded. One message is basically nothing. I would have appreciated if they had released the full distribution of impact though.

> Anthropic is purposefully degrading things to save money Regardless of whether it’s to save money, it’s purposefully inaccurate: “When Claude generates text, it calculates probabilities for each possible next word, then randomly chooses a sample from this probability distribution.” I think the reason for this is that if you were to always choose the highest probable next word, you may actually always end up with th…

>or get stuck in a loop

You are absolutely right! Greedy decoding does exactly that for longer seqs: https://huggingface.co/docs/transformers/generation_strategi...

Interestingly DeepSeek recommends a temperature of 0 for math/coding, effectively greedy.

Re: A postmortem of three recent issues

#86

Post by kashunstva »

flutas wrote: post #14

And yet no offers of credits to make things right for the users, for what was essentially degraded performance of what you paid for. I know I'll probably get push back on this, but it left a sour taste in my mouth when I paid for a $200 sub that felt like it was less useful than ChatGPT Plus ($20) at times. Or to summarize: [south park "we're sorry" gif]

I would not count on any compensation. I was a Pro subscriber until the third week of May when I was no longer able to login due to some auto ban. No idea what triggered it but no one from support has bothered to respond to my appeals form submissions. A lost cause.

Re: A postmortem of three recent issues

#87

Post by kashunstva »

extr wrote: post #29
flutas wrote: post #23

Earlier quoted context omitted.

That 30% is of ALL users, not users who made a request, important to note the weasel wording there. How many users forget they have a sub? How many get a sub through work and don't use it often? I'd bet a large number tbh based on other subscription services.

That's a pretty cynical read. My personal impression is that Anthropic has a high level of integrity as an organization. Believe what you want, I'm inclined to give them the benefit of the doubt here and move on.

> My personal impression is that Anthropic has a high level of integrity as an organization.

Unless you consider service responsiveness as a factor of integrity. Still waiting on a service message reply from third week of May. I’m sure it’s right around the corner though.

Re: A postmortem of three recent issues

#88

Post by am17an »

They must really be having a bad time if Anthropic of all labs is willing to share their infra details. On the actual precision bug, it is quite unfortunate on FMA side, numerical issues are often deeply bewildering and no AI can solve them (yet.) Also goes to show, if you are in a super crunch situation like this one (competitor literally eating your lunch every day), you need humans to understand what went wrong and even then it can take weeks to rectify.
Post reply on HN