Do we know why Google’s Gemini didn’t get affected by the XLA bug? Don't they use approx top-k or mixed precision? Also, TIL that Claude is served in prod at bf16
A postmortem of three recent issues
101–110 of 120 posts
Re: A postmortem of three recent issues
#102The most interesting thing about this is the apparent absence of unit tests. The test for the XLA compiler bug just prints the outputs, it's more like a repro case than a unit test in the sense that it'd be run by a test harness and have coverage tracked. And the action items are simply to lean more aggressively into evals. Although unit testing an entire LLM is not really feasible right now, all these bugs were in s…
I've had to write some detailed prompts and examples to have AI generate the kind of unit tests I want in Python. I've seen the assertions on types alone too. I want assertions on values and more. Even more than that, AI tends to mock _everything_. Mocking is useful, but the more real code a unit test invokes, the better, because the risk is not only the code itself but its interactions, the interface. Yet AI in Pyth…
I wish I had 100 upvotes to give you. Weak, heavily mocked tests are my biggest pet peave. Test “quality” is important and not something a lot of devs pay attention to.
I’ve found myself preferring integration tests or unit tests with a “real” database set up because the tests are much more effective. If you design them right, they don’t even need to be slower.
Re: A postmortem of three recent issues
#103Earlier quoted context omitted.
I've had to write some detailed prompts and examples to have AI generate the kind of unit tests I want in Python. I've seen the assertions on types alone too. I want assertions on values and more. Even more than that, AI tends to mock _everything_. Mocking is useful, but the more real code a unit test invokes, the better, because the risk is not only the code itself but its interactions, the interface. Yet AI in Pyth…
I'm curious how you structure your Python to be well testable. I have to admit, my own use of Python has been limited to scripts and (a long time ago) a game engine, not large codebases. So unit testing for those hardly came up. It seems there's a couple of dependency injection frameworks but they're clones of what's found in Java, right down to the type names. One of them even calls injectable objects beans! (Rhazes…
Most of my Python is web. Individual components, same as always - approach with a set API and not too many dependencies, and allow injection via some route if so. I also test web endpoints. One thing I really like is isolating tests that require data -- rather than mocking the database, for example, I'll create an in-memory SQLite DB used while running tests. That way I can test the full stack: a web API, see its results, and check what was changed in the database at the same time, all isolated from the 'real' stack.
Re: A postmortem of three recent issues
#104With all due respect to the Anthropic team, I think the Claude status page[1] warrants an internal code red for quality. There were 50 incidents in July, 40 incidents in August, and 21 so far in September. I have worked in places where we started approaching half these numbers and they always resulted in a hard pivot to focusing on uptime and quality. Despite this I'm still a paying customer because Claude is a fanta…
This is always why you should put as few incidents on status page as possible. People's opinion will drop and then the negative effect will fade over time. But if you have a status page then it's incontrovertible proof. Better to lie. They'll forget. e.g. S3 has many times encountered increased error rate but doesn't report. No one says anything about S3. People will say many things, but their behaviour is to reward…
Yup, these guys aren't the customers anyway. The investors are the only ones they care about because the customers don't come close to paying the actual costs.
Re: A postmortem of three recent issues
#105Earlier quoted context omitted.
I've had to write some detailed prompts and examples to have AI generate the kind of unit tests I want in Python. I've seen the assertions on types alone too. I want assertions on values and more. Even more than that, AI tends to mock _everything_. Mocking is useful, but the more real code a unit test invokes, the better, because the risk is not only the code itself but its interactions, the interface. Yet AI in Pyth…
I'm curious how you structure your Python to be well testable. I have to admit, my own use of Python has been limited to scripts and (a long time ago) a game engine, not large codebases. So unit testing for those hardly came up. It seems there's a couple of dependency injection frameworks but they're clones of what's found in Java, right down to the type names. One of them even calls injectable objects beans! (Rhazes…
I learned to write well testable code when I learned go. It pushes you to pass interfaces instead of direct implementations. There's also no inheritance, just composition. While there's no 1 to 1 translation to Python the concepts are still useful. It can be easier in Python thanks to duck typing.
Re: A postmortem of three recent issues
#106I don't know if this is related, but I've noticed a massive issue recently within web app design where claude will just create random streams of text that displays in the dom. Think it's something specifically related to attempting to use svelte, but definitely a massive degredation that I didn't notice prior to this.
Re: A postmortem of three recent issues
#107You're absolutely right!
Re: A postmortem of three recent issues
#108The most interesting thing about this is the apparent absence of unit tests. The test for the XLA compiler bug just prints the outputs, it's more like a repro case than a unit test in the sense that it'd be run by a test harness and have coverage tracked. And the action items are simply to lean more aggressively into evals. Although unit testing an entire LLM is not really feasible right now, all these bugs were in s…
I've had to write some detailed prompts and examples to have AI generate the kind of unit tests I want in Python. I've seen the assertions on types alone too. I want assertions on values and more. Even more than that, AI tends to mock _everything_. Mocking is useful, but the more real code a unit test invokes, the better, because the risk is not only the code itself but its interactions, the interface. Yet AI in Pyth…
Mocked tests also make refactoring a pain in the ass.
This is why I heavily prefer integration tests
Re: A postmortem of three recent issues
#109I do wonder what a random dip in quality causes in a long running conversation? Does the conversation recover at a later point, or does the introduction of temporary idiocy permanently affect the rest of the conversation? Statistically, probably likely that the dip occurred at a point that wasn't too important? But what happens if the idiot comes out at a critical point? Kind of reminds me of the two alternate ways t…
Depends how good your competitors are at capitalizing on it.
Guess what Sam Altman is good at.
Re: A postmortem of three recent issues
#110Anthropic seems to have a "We're special" syndrome:
* No Sign in with Apple on the website, so tough luck if you signed up on iOS via that
* Can't buy via in-app purchase so have to give them your card
* Can't remove your payment method from your account
* Hard to get support after buying a subscription
* Asking Claude questions about Claude itself such as privacy just gives you links to their website
What even is the benefit of paying for Claude over ChatGPT or Grok that are better in overall UX?