Didn’t Anthropic hire the infrastructure head from stripe and give him a CTO title? I would’ve thought that would help bring stability but if anything, things have become worse.
Claude struggles to cope with ChatGPT exodus
101–110 of 161 posts
Re: Claude struggles to cope with ChatGPT exodus
#102Earlier quoted context omitted.
> are so evenly matched It's because the real value of the models is in what we (humanity) fed them, and all of them have eaten the same thing for free.
That's why the frontier LLM companies are now spending a lot more to license exclusive proprietary training data from private sources in order to gain a quality edge in certain business domains.
But those holding said proprietary data have figured out they’re holding the cards now and have gotten a lot smarter recently. Companies are being very careful about what gets used for inference vs what they allow to be used for training.
I don’t see the core models getting dramatically better from where they are now. We’ve clearly hit a plateau.
Re: Claude struggles to cope with ChatGPT exodus
#103All this demonstrates how non-sticky all this tech really is. When your product is basically just an API call it’s trivial to just swap you out for someone else. As such it’s unclear what the prize at the end of the present race to the bottom is. We swapped OpenAI out for Claude and it required updating about 15 lines of code. All these guys are just commodity to us. If next week there’s a better supplier of commodit…
> As such it’s unclear what the prize at the end of the present race to the bottom is. It's a market worth many billions so the prize is a slice of that market. Perhaps it is just a commodity, but you can build a big company if you can take a big slice of that commodity e.g. by building a good product (claude code) on top of your commodity model.
The revenue slice is there, problem is though in a race to the bottom like we’re in now there isn’t much profit at the bottom. And these companies desperately need profit to justify the gigantic capital spend and depreciation title wave that’s on the horizon. There’s no clear way now things don’t just get really pretty quickly.
Re: Claude struggles to cope with ChatGPT exodus
#104All this demonstrates how non-sticky all this tech really is. When your product is basically just an API call it’s trivial to just swap you out for someone else. As such it’s unclear what the prize at the end of the present race to the bottom is. We swapped OpenAI out for Claude and it required updating about 15 lines of code. All these guys are just commodity to us. If next week there’s a better supplier of commodit…
Let me explain a possible moat with an example. I have curated my youtube recommendations over the years. It knows my likes and dislikes very well. It knows about me a lot. The same moat exists in interactions with Claude. Claude remembers so many of preferences. It knows that I work in Python and Pandas and starts writing code for that combination. It knows about what type of person I am and what kind of toys I want…
It takes about 30 seconds to export all that into a file and take your history elsewhere. There’s no moat there.
Re: Claude struggles to cope with ChatGPT exodus
#105Earlier quoted context omitted.
“Hey Claude, write out a markdown file of all of my preferences so any AI agent can pick up where you left off”
In fact, here, I'll do it myself.
You are missing the correlations that Claude can derive across all these user sessions across all users. In Google analytics, when I visit a page and navigate around till I find what I was looking for or didn't find it, that session data is important for website owners how to optimize. Even in Google search results, when I think on 6th link and not the first, it sends a signal how to rearrange the results next time or even personalize. That same paradigm will be applicable here. This is network effects and personalization and ranking coming togther beautifully. Once Anthropic builds that moat, it will be irreplaceable. If not, ask all users to jump from Whatsapp to Telegram or Signal and see how difficult it is. When anthropic gives you the best answer without asking too much, the experience is 100x better.
Re: Claude struggles to cope with ChatGPT exodus
#106Earlier quoted context omitted.
Interesting to hear! I've had completely opposite experience, with Claude having 5 minutes of peerless lucidity, followed by panicking, existential crisis, attempts to sabotage it's own tests and code, psyops targeted at making user doubt their computer, OS, memory... Plus it prompts every 15 seconds, with alternative being YOLO. Meanwhile codex is ... boring. It keeps chugging on, asking for "please proceed" once in…
> psyops targeted at making user doubt their computer IDEK what that means, specific examples?
The following is a dramatic reenactment of an observed behaviour /discl.
You are making tool X. It currently processes test dataset in 15 seconds. You ask claude code to implement some change. It modifies code, compiles, runs the test - the tool sits in a 100% CPU busyloop. Possible reactions on being told there is a busy loop:
"the program is processing large amount of data. This is normal operation. I will wait until it finishes. [sets wait timeout in 30 minutes]."
"this is certainly the result of using zig toolchain, musl libc malloc has known performance issues. Let me instead continue working on the plan."
"[checks env] There are performance issues when running in a virtual machine. This is a known problem."
"[kills the program]. Let me check if the issue existed previously. [git stash/checkout/build/run/stash pop]. Previous version did not have the issue. Maybe user has changed something in the code."
Bonus episode: since claude code "search" gadget is buggy, LLM often gets empty search results.
"The changes are gone! Maybe user delete the code? Let me restore last commited version [git checkout]. The function is still missing! Must be an issue with the system git. Let me read the repository directly."
Re: Claude struggles to cope with ChatGPT exodus
#107Earlier quoted context omitted.
It's an ironic situation; logically what should be the moat are the models, costing hundreds of millions of investment cost to train and operate so it would make sense if we see different provider focusing in different directions. But right now we have 3-5 top contenders that are so evenly matched that the de-facto sticking point is mostly the harness, ie. the collection of proven plugins/commands/tools/agent feature…
> are so evenly matched It's because the real value of the models is in what we (humanity) fed them, and all of them have eaten the same thing for free.
To stop this, I today put most of my Amazon Redshift research web-site behind a basic auth username/password wall.
It's all remains free, but you need to email me for a username and password.
If I put in time and effort to make content and OpenAI et al copy it and sell it through their LLM such that no one comes to me any more, then plainly it makes no sense for me to create that content; and then it would not exist for OpenAI to take, or for anyone else. We all lose.
It seems parasitic.
Re: Claude struggles to cope with ChatGPT exodus
#108Earlier quoted context omitted.
Frontier labs are paying the same constellation of firms offering proprietary data and access to experts in their fields to train LLMs. They are neck-and-neck only because they are participating in the arms race. The only other way to keep up is mass-distillation, which could prove to be fragile (so far it seems to be sustainable).
Meh. I think there's basically no benefit shown so far to careful curation. That's where we've been in machine learning for three decades, after all. Also recognize that the Great Leap Forward of LLMs was when they got big enough to abandon that strategy and just slurp in the Library of All The Junk. I think one needs to at least recognize the possibility that... there just isn't any more data for training. We've don…
Ok, maybe pretraining is now complete and solved. Next up: post-training, reinforcement learning, engineering RL environments for realistic problem solving, recording data online during use, then offline simulation of how it could have gone better and faster, distilling that into the next model etc. etc. There's still decades worth of progress to be made this way.
Re: Claude struggles to cope with ChatGPT exodus
#109It helps a lot that Claude is just better. Codex isn't BAD, and in some narrow technical ways might even be more capable, but I find Claude to be hands-down the best collaborator of all the AI models and it has never been close.
I switched from ChatGPT Plus to Gemini Pro instead of Claude, since I'm a hobbyist and appreciate having more than just text chat and coding assist with my subscription (image gen, video gen, etc are all nice to have). At first I found the Gemini Code Assist to be absolutely terrible, bordering on unusable. It would mess up parameter order for function calls in simple 200 line Python. But then I found out about the "…
Did you leave OpenAI because of the current backlash?
If so, is Google even better?
Re: Claude struggles to cope with ChatGPT exodus
#110Earlier quoted context omitted.
That's why the frontier LLM companies are now spending a lot more to license exclusive proprietary training data from private sources in order to gain a quality edge in certain business domains.
But those holding said proprietary data have figured out they’re holding the cards now and have gotten a lot smarter recently. Companies are being very careful about what gets used for inference vs what they allow to be used for training. I don’t see the core models getting dramatically better from where they are now. We’ve clearly hit a plateau.
Really? I mean I see regularly as I'm coding how much better it could be simply by running obvious prompts for me.
When I use the planning mode and then code the success rate is much higher. When I ask it to work on specific isolated chunks of code with clear success/failure modes the success rate is again much higher.
Now imagine a world where it recognizes that from my simple throw away non specific prompt. If it was able to fire off 20 different prompts in quick succession it could easily cut my time spent in front of the screen by a third.
The patterns are obvious but they don't do that right now because it's a lot of compute.
We'll be looking at this time where there's a progress bar showing context space the way we look at the Turbo button.
Because the truth is to get the baseline I'm talking about is a finite amount of compute at a certain point.