Earlier quoted context omitted.
check out my pi forks.
Ummmmmm, how?
I searched his HackerNews username on Google.
Claude.ai unavailable and elevated errors on the API
251–260 of 277 posts
Re: Claude.ai unavailable and elevated errors on the API
#252Earlier quoted context omitted.
GitHub, along with MSFT in general, have massive copilot mandates where workers are being shamed into using slop tools to fix serious on-going issues. GitHub seems wholly incapable of resolving their issues: money isn't a problem, talent isn't a problem, but business leadership is definitely a major problem. Look at how other companies are suffering massive outages due to LLMs too like AWS and Cloudflare. Two compani…
None of the three major Cloudflare outages in the past six months had anything to do with LLMs. They were regular old human mistakes. We did, however, determine that at least one of them (and perhaps all) would have been easily caught by AI code reviewers, had AI code reviewers been in use. So now we mandate that. And honestly, I love it, the AI reviewer spots all sorts of things that humans would probably miss. (We…
[deleted]
Re: Claude.ai unavailable and elevated errors on the API
#253Earlier quoted context omitted.
Truly! As someone who's worked with HPC and GPUs in a scientific research context, trying to get a service like this to work reliably is a different ballgame to your usual webapp stack...
I think you have to see this as a bunch of stateless requests, and this makes the problem way easier. LLM requests that do not call tools do not need anything external by definition. No central server, nothing, they can even survive without the context cache. All you need is to load (and only once!) the read-only immutable model weights from a S3-like source on startup. If it takes 4 servers to process a request, the…
Yeah, and totally missed RAI part, billing, model deployment, security patches, rate-limiting, caching, dead GPUs, metrics, multiple regions, gov clouds, gdpr(or data locality issues), monitoring, alerting and god knows what else while at extreme loads.
Re: Claude.ai unavailable and elevated errors on the API
#254Earlier quoted context omitted.
None of the three major Cloudflare outages in the past six months had anything to do with LLMs. They were regular old human mistakes. We did, however, determine that at least one of them (and perhaps all) would have been easily caught by AI code reviewers, had AI code reviewers been in use. So now we mandate that. And honestly, I love it, the AI reviewer spots all sorts of things that humans would probably miss. (We…
> None of the three major Cloudflare outages in the past six months had anything to do with LLMs. They were regular old human mistakes. Is that true? At least one of them seemed to involve LLM-written code from what I saw. (Not to say that human error wasn't _also_ a contributing factor, but I wouldn't say it had _nothing_ to do with LLMs). > We did, however, determine that at least one of them (and perhaps all) woul…
> Not that you would know that the way our genius CTO talks about it...
Honestly I find it bizarre that there are people at Cloudflare who have this attitude. Without Dane, the company wouldn't be half the size it is today.
Re: Claude.ai unavailable and elevated errors on the API
#255Earlier quoted context omitted.
I think you have to see this as a bunch of stateless requests, and this makes the problem way easier. LLM requests that do not call tools do not need anything external by definition. No central server, nothing, they can even survive without the context cache. All you need is to load (and only once!) the read-only immutable model weights from a S3-like source on startup. If it takes 4 servers to process a request, the…
Yeah, and totally missed RAI part, billing, model deployment, security patches, rate-limiting, caching, dead GPUs, metrics, multiple regions, gov clouds, gdpr(or data locality issues), monitoring, alerting and god knows what else while at extreme loads.
GDPR doesn’t affect load, dead GPUs are no different than any software freeze, model is a file update, metrics are already scaling very well and even way way way bigger and they are very linear, security updates are hedged with gradual rollouts, canary, feature flags, etc.
From an ops perspective all of these things are already really well solved issues in a very scalable manner, because plenty of companies had to solve these issues before.
It’s even better here because you can throw millions in salaries to “steal” the insider info on how their production actually.
No doubt it is fast-paced but the complexity to go from 100k GPUs to 1M is much lower than from going from 1k to 10k GPUs.
All 3 big AI companies had the luxury that during the scaling phase they could do everything directly on production servers.
This is because customers were very very tolerant, and are still quite tolerant.
You can even set limits of requests to large users and shape the traffic.
Cloudflare in comparison, high-scale, low-latency, end users not tolerant at all to downtime, customers even less tolerant, clearly hostile actors that actively try to make your systems down, limited budget, a lot of different workloads, etc.
So, for LLM companies where you have to scale a single workload, largely from mostly free users, and where most paid customers can be throttled and nobody is going to complain because nobody knows what are the limits + a lot of tolerance to high-latency and even downtimes then you are very lucky.
Re: Claude.ai unavailable and elevated errors on the API
#256Re: Claude.ai unavailable and elevated errors on the API
#257Earlier quoted context omitted.
If you are paying API rates (not using Max subscriptions) there's no reason to use Anthropic's API directly, the same models are hosted by both AWS and Google with better uptime than Anthropic.
The enterprise tier is API pricing only. https://support.claude.com/en/articles/9797531-what-is-the-e...
[deleted]
Re: Claude.ai unavailable and elevated errors on the API
#258Earlier quoted context omitted.
On what hardware? Like companies would buy up GPUs?
Presumably you'd buy really beefy laptops. The price delta between buying the most basic MacBook Pro possible (14", M5, 16 GB unified memory, 1 TB SSD) and one with the M5 Max with 40 GPU cores, 128 GB unified memory, 2 TB SSD is $3400. How much Claude usage does that get you/in what time does it pay itself back?
That doesn't get you any Claude usage. Claude models obviously aren't open, but equivalent models to Opus take about 400GB of memory to run.
You can run the versions with fewer parameters or quantized weights but depending on how much quality you're sacrificing, now you'd have to compare the price against cheaper Claude models like Sonnet.
Re: Claude.ai unavailable and elevated errors on the API
#259As a long-term 20x user, Claude has recently felt a lot like using AI for coding a year or so ago. It can't reliably handle basic tasks. I ask for something straightforward and get something subtly wrong, incomplete, or just not workable. I always use the best model available and effort levels maxed, but with all their changes I have to relearn how to make the model perform at best every day, and it seems I can't kee…
The inconsistency has always been there you’re just noticing it more over time and the models are not really improving at real work in spite of all the new releases and churn.