In my intuition it makes sense that there is going to be some significant friction in LLM development going forward. We're talking about models that will cost upwards of $1bn to train. Save for a technological breakthrough, GPT-6/7 will probably have to wait for hardware to catch up.
GPT-5 is behind schedule
81–90 of 1001 posts
Re: GPT-5 is behind schedule
#82Earlier quoted context omitted.
"With o3 now public knowledge, imagine how long it’s been churning out new thinking at expert level across every field." I highly doubt that. o3 is many orders of magnitude more expensive than paying subject matter experts to create new data. It just doesn't make sense to pay six figures in compute to get o3 to make data a human could make for a few hundred dollars.
Someone needs to dress up Mechanical Turk and repackage it as an AI company…..
That’s basically every AI company that existed before GPT3
Re: GPT-5 is behind schedule
#83Earlier quoted context omitted.
Not sure what you are using it for, but it is terrible for me for coding; claude beats it always and hands down. o1 just thinks forever to come up with stuff it already tried the previous time. People say that's just prompting without pointing to real million line+ repositories or realistic apps to show how that can be improved. So I say they are making todo and hello world apps and yes, there it works really well. C…
That so weird, it’s seems like everybody here prefers Claude. I’ve been using Claude and openai in copilot and I find even 4o seems to understand the problem better. O1 definitely seems to get it right more for me.
I try to sprinkle 'for us/me' everywhere as much as I can; we work on LoB/ERP apps mostly. These are small frontends to massive multi million line backends. We carved a niche by providing the frontends on these backends live at the client office by a business consultant of ours: they simply solve UX issues for the client on top of large ERP by using our tool and prompting. Everything looks modern, fresh and nice; unlike basically all the competitors in this space. It's fast and no frontend people are needed for it; backend is another system we built which takes a lot longer of course as they are complex business rules. Both claude and o1 turn up something that looks similar but only the claude version will work and be, after less prompting, correct. I don't have shares in either and I want open source to win; we have all open (more open) solutions doing all the same queries and we evaluate all but claude just wins. We did manage even big wins with openai davinci in 2022 (or so; before chatgpt), but this is a massive boost allowing us to upgrade most people to business consultant and just have them build with clients real time and have the tech guys including me add manually tests and proofs (where needed) to know if we are actually fine. Works so much better than the slog with clients before; people are so bad at explaining at what they need, it was slowly driving me insane after doing it for 30+ years.
Re: GPT-5 is behind schedule
#84Earlier quoted context omitted.
I find that o1 and Sonnet 3.5 are good and bad quite equally on different things. That's why I keep asking both the same coding questions.
We do the same (all requests go to o1, sonnet and gemini and we store the results for later to compare) automatically for our research: Claude always wins. Even with specific prompting on both platforms. Especially frontend it seems o1 really is terrible.
Every time I try Gemini, it's really subpar. I found that qwen2.5-coder-32b-instruct can be better.
Also, for me 50% 50% for Sonnet and o1, but although I'm not 100% sure about it, I think o1 is better with longer and more complicated (C++) code and debugging. At least from my brief testing. Also, OpenAI models seem to be more verbose - sometimes it's better - where I'd like additional explanation on chosen fields in a SQL schema, sometimes it's too much.
EDIT: Just asked both o1 and Sonnet 3.5 the same QML coding question, and Sonnet 3.5 succeeded, o1 failed.
Re: GPT-5 is behind schedule
#85probably because it isn't any better
Re: GPT-5 is behind schedule
#86Earlier quoted context omitted.
That leads to overfitting in ML land, which hurts overall performance. We know that unique data improves performance. These LLM systems are not students… Also, which students graduate and are immediately experts in their fields? Almost none. It takes years of practice in unique, often one-off, situations after graduation for most people to develop the intuition needed for a given field.
It's overfitting when you train too large a model on too many details. Rote memorization isn't rewarding. The more concepts the model manages to grok, the more nonlinear its capabilities will be: we don't have a data problem, we have an educational one. Claude 3.5 was safety trained by Claude 3.0, and it's more coherent for it. https://www.anthropic.com/news/claudes-constitution
Overfitting can be caused by a lot of different things. Having an over abundance of one kind of data in a training set is one of those causes.
It’s why many pre-processing steps for image training pipelines will add copies of images at weird rotations, amounts of blur, and different cropping.
> The more concepts the model manages to grok, the more nonlinear its capabilities will be
These kind of hand wavey statements like “practice,” “grok,” and “nonlinear its capabilities will be” are not very constructive as they don’t have solid meaning wrt language models.
So earlier when I was referring to compounding bias in synthetic data I was referring to a bias that gets trained on over and over and over again.
That leads to overfitting.
Re: GPT-5 is behind schedule
#87Earlier quoted context omitted.
I keep reading this on HN so I believe it has to be true in some ways, but I don't really feel like there is any difference in my limited use (programming questions or explaining some concepts). If anything I feel like it's all been worse compared to the first release of ChatGPT, but I might be wearing rose colored glasses.
It’s the same for me. I genuinely don’t understand how I can be having such a completely different experience from the people who rave about ChatGPT. Every time I’ve tried it’s been useless. How can some people think it’s amazing and has completely changed how they work, while for me it makes mistakes that a static analyser would catch? It’s not like I’m doing anything remarkable, for the past couple of months I’ve b…
first time I tried it, I asked it to find bugs in a piece of very well tested C code.
It introduced an off-by-one error by miscounting the number of arguments in an sprintf call, breaking the program. And then proceeded to fail to find that bug that it introduced.
Re: GPT-5 is behind schedule
#88The world is figuring out how to make this technology fit and work and somehow this is "behind" schedule. It's almost comical.
[flagged]
I use them to:
• Build toys that would otherwise require me to learn new APIs (I can read python, but it's not my day job)
• Learn new things like OpenSCAD
• To improve my German
• Learn about the world by allowing me to take photos of things in this world that I don't understand and ask them a question about the content, e.g. why random trees have bands or rectangles of white paint on them
• Help me shopping, by taking a photo of the supermarket that I happen to be in at the time and ask them where I should look for some item I can't find
• Help with meal prep, by allowing me to get a recipe based on what food and constraints I've got at hand rather than the traditional method of "if you want x, buy y ingredients"
Even if they're just an offline version of Wikipedia or Google, they're already a more useful interface for the same actual content.
Re: GPT-5 is behind schedule
#89Earlier quoted context omitted.
[flagged]
There's someone with this comment in every thread. Meanwhile, no one answers this because they are getting value. Please take the time to learn, it will give you value.
I'm someone who generally was a "doubter", but I've dramatically softened my stance on this topic.
Two things: I was casually watching Andreas Kling's streams on Ladybird development (where he was developing a JIT compiler for JS) and was blown away at the accuracy of completions (and the frequency of those completions)
Prior to this, I'd only ever copypasta'd code from ChatGPT output on occasion.
I started adopting the IDE/Editor extensions and prototyping small projects.
There's now small tools and utilities I've written that I'd not have written otherwise, or would have taken twice the time invested had I'd not used these tools.
With that said, they'd be of no use without oversight, but as a productivity enhancement, the benefits are enormous.
Re: GPT-5 is behind schedule
#90Earlier quoted context omitted.
There's someone with this comment in every thread. Meanwhile, no one answers this because they are getting value. Please take the time to learn, it will give you value.
> Meanwhile, no one answers this because they are getting value. You're literally doing the same thing you're accusing of. Every HN thread is full of AI boosters claiming AI to be the future with no backing evidence. Riddle me this. If all these people are "getting value", why are all these companies losing horrendous amounts of money? Why has nobody figured out how to be profitable? > Please take the time to learn,…
> Why has nobody figured out how to be profitable?
From what I've seen claimed about OpenAI finances, this is easy: It's a Red Queen's race — "it takes all the running you can do, to keep in the same place".
If their financial position was as simple as "we run this API, we charge X, the running cost is Y", then they're already at X > Y.
But if that was all OpenAI were actually doing, they'd have stopped developing new versions or making the existing models more efficient some time back, while the rest of the industry kept improving their models and lowering their prices, and they'd be irrelevant.
> People would be lining up to pay OpenAI.
They are.
Not that this is either sufficient or necessary to actually guarantees anything about real value. For lack of sufficiency: people collectively paid a lot for cryptocurrencies and NFTs, too (and before then and outside tech, homeopathic tinctures and sub-prime mortgages); For lack of necessity: there's plenty of free-to-download models.
I get a huge benefit even just from the free chat models. I could afford to pay for better models, but why bother when free is so good? Every time a new model comes out, the old paid option becomes the new free option.