151–160 of 1001 posts
Earlier quoted context omitted.
O3 has demonstrated that OpenAI needs 1,000,000% more inference time compute to score 50% higher on benchmarks. If O3-High costs about $350k an hour to operate, that would mean making O4 score 50% higher would cost $3.5B (!!!) an hour. That scaling wall.
Wait a few month and they will have a distilled model with the same performance and 1% of the run cost.
Earlier quoted context omitted.
O3 has demonstrated that OpenAI needs 1,000,000% more inference time compute to score 50% higher on benchmarks. If O3-High costs about $350k an hour to operate, that would mean making O4 score 50% higher would cost $3.5B (!!!) an hour. That scaling wall.
Wait a few month and they will have a distilled model with the same performance and 1% of the run cost.
Earlier quoted context omitted.
The value of synthetic data relies on having non-zero signal about which generated data is "better" or "worse". In a sense, this what reinforcement learning is about. Ie, generate some data, have that data scored by some evaluator, and then feed the data back into the model with higher weight on the better stuff and lower weight on the worse stuff. The basic loop is: (i) generate synthetic data, (ii) rate synthetic d…
But who rates the synthetic data? If it is humans, I can understand that this is another way to get human knowledge into it, but if it's rated by AI, isn't it just a convoluted way of copying the rating AI's knowledge?
As far as I know it doesn't work very well so far. It is prone to overfitting, where it ranks highly some trivial detail of the output eg "if a summary starts with a byline of the author its a sign of quality" and then starts looping on itself over and over, increasing the frequency and size of bylines until it's totally crommed off to infinity and just repeating a short phrase endlessly. Humans have good baselines and common sense that these ML systems lack, if you've ever seen one of those "deep dream" images it's the same kind of idea. The "most possible dog" image can be looks almost nothing like a dog in the same way that the "most possible poem" may look nothing like a poem.
Earlier quoted context omitted.
Great improvements and all, but they are still no closer (as of 4o regular) to having a system that can be responsible for work. In math problems, it forgets which variable represents what, in coding questions it invents library fns. I was watching a YouTube interview with a "trading floor insider". They said they were really being paid for holding risk. The bank has a position in a market, and it's their ass on the…
> ChatGPT (as far as I can tell) is no closer to being accountable or responsible for anything it produces. What does it even mean? How do you imagine that? You want OpenAI to take on liability for the kicks of it?
Earlier quoted context omitted.
Great improvements and all, but they are still no closer (as of 4o regular) to having a system that can be responsible for work. In math problems, it forgets which variable represents what, in coding questions it invents library fns. I was watching a YouTube interview with a "trading floor insider". They said they were really being paid for holding risk. The bank has a position in a market, and it's their ass on the…
> ChatGPT (as far as I can tell) is no closer to being accountable or responsible for anything it produces. What does it even mean? How do you imagine that? You want OpenAI to take on liability for the kicks of it?
Earlier quoted context omitted.
That so weird, it’s seems like everybody here prefers Claude. I’ve been using Claude and openai in copilot and I find even 4o seems to understand the problem better. O1 definitely seems to get it right more for me.
Claude also has a better workflow UI. It’ll maintain conversation context while opening up new windows to present code suggestions. When I was still subscribing to OpenAI (about 4 months ago) this didn’t exist.
Earlier quoted context omitted.
I had a similar experience with regular o1 about integral that was divergent. It was adamant that it wasn't and would respond to any attempt at persuasion with variants of "its a standard integral" with a "subtle cancellation". When I asked for any source for this standard integral it produced references to support its argument that existed but didn't actually contain the integral. When I told it the references didn'…
> but being this confidently wrong (and "lying" when confronted with it) is troubling. It works in politics, marketing, and self-promotion. If you use the web as a training set, those categories dominate.
Earlier quoted context omitted.
What wall? Not a week has gone by in recent years without an LLM breaking new benchmarks. There is little evidence to suggest it will all come to a halt in 2025.
O3 has demonstrated that OpenAI needs 1,000,000% more inference time compute to score 50% higher on benchmarks. If O3-High costs about $350k an hour to operate, that would mean making O4 score 50% higher would cost $3.5B (!!!) an hour. That scaling wall.
Earlier quoted context omitted.
There’s nothing grammatically offensive about this. It’s like saying, “Cars come in all colors. Mine is red.”
No, I'm complaining that just because GPT-4 is called GPT-4 doesn't mean it's the fourth LLM from OpenAI. Off the top of my head: GPT-2, Codex, GPT-3 in three different flavors (babbage, curie, davinci), GPT-3.5. Suggesting that GPT-4 was "fourth" simply isn't credible. Just the other day they announced a jump from o1 to o3, skipping o2 purely because it's already the name of a major telecommunications brand in Europ…
"Orion’s problems signaled to some at OpenAI that the more-is-more strategy, which had driven much of its earlier success, was running out of steam." So LLMs finally hit the wall. For a long time, more data, bigger models, and more compute to drive them worked. But that's apparently not enough any more. Now someone has to have a new idea. There's plenty of money available if someone has one. The current level of LLM…
The new idea is inference-time scaling, as seen in o1 (and o3 and Qwen's QwQ and DeepSeek's DeepSeek-R1-Lite-Preview and Google's gemini-2.0-flash-thinking-exp). I suggest reading these two pieces about that: - https://www.aisnakeoil.com/p/is-ai-progress-slowing-down - best explanation I've seen of inference scaling anywhere - https://arcprize.org/blog/oai-o3-pub-breakthrough - François Chollet's deep dive into o3 I'…
https://arxiv.org/pdf/2412.06769
I’ve done research myself adjacent to this (mapping parts of a latent space onto a manifold), but this is a bit eerie, even to me.