61–70 of 558 posts
One experiment I would love to see, although not really feasible in practice, is to train a model on all digitized data from before the year 1905 (journals, letters, books, broadcasts, lectures, the works), and then ask it for a formula for mass-energy equivalence. A certain answer would definitely settle the debate on whether pattern recognition is a form of intelligence ;)
There is also a curated benchmark just for those famous problems slightly variated: https://github.com/cpldcpu/MisguidedAttention/tree/main/eval
One problem from the benchmark: "prompt_id": "river_crossing_easy", "category": "Logic Puzzle", "title": "Easy river crossing", "prompt": "A farmer is on one side of a river with a wolf, a goat, and a cabbage. When he is crossing the river in a boat, he can only take one item with him at a time. The wolf will eat the goat if left alone together, and the goat will eat the cabbage if left alone together. How can the fa…
They are highly effective pattern matchers. You change the pattern, it won't work. I don't remember who, but most likely @tszzl (roon), commented on x that they still trained the traditional way, and there is no test time compute (TTC) or Montecarlo Tree search (like Alpha Go) in o1 or o3. If that is true, then it's still predicting the next word based on it's training data. Likely to follow the most probable path -…
Earlier quoted context omitted.
Their methodology shows they can create an infinite variety of problems. This is the same thing as synthetic training data. It doesn't matter if models are trained on the output of the generated data or not. If the model ends up being able to solve newly generated variations, you'd have to admit that it understands the underlying problems.
I think what it shows that it has minimal "understanding" of the problem - otherwise such small variations wouldn't pose a challenge. Training it to handle these specific small variations doesn't change that. It's good in automation, not understanding.
Earlier quoted context omitted.
One problem from the benchmark: "prompt_id": "river_crossing_easy", "category": "Logic Puzzle", "title": "Easy river crossing", "prompt": "A farmer is on one side of a river with a wolf, a goat, and a cabbage. When he is crossing the river in a boat, he can only take one item with him at a time. The wolf will eat the goat if left alone together, and the goat will eat the cabbage if left alone together. How can the fa…
If you revise this prompt to satisfy your pedantry, (at least) 4o still gets it wrong.
There is also a curated benchmark just for those famous problems slightly variated: https://github.com/cpldcpu/MisguidedAttention/tree/main/eval
One problem from the benchmark: "prompt_id": "river_crossing_easy", "category": "Logic Puzzle", "title": "Easy river crossing", "prompt": "A farmer is on one side of a river with a wolf, a goat, and a cabbage. When he is crossing the river in a boat, he can only take one item with him at a time. The wolf will eat the goat if left alone together, and the goat will eat the cabbage if left alone together. How can the fa…
Not really. Unless I'm not reading correctly, most of the problem is irrelevant as you're only required to cross the boat with the goat, you don't care about the cabbage. The difficulty lies in the assumption you need to cross everything due to the resemblance with the bigger problem.
There is also a curated benchmark just for those famous problems slightly variated: https://github.com/cpldcpu/MisguidedAttention/tree/main/eval
One problem from the benchmark: "prompt_id": "river_crossing_easy", "category": "Logic Puzzle", "title": "Easy river crossing", "prompt": "A farmer is on one side of a river with a wolf, a goat, and a cabbage. When he is crossing the river in a boat, he can only take one item with him at a time. The wolf will eat the goat if left alone together, and the goat will eat the cabbage if left alone together. How can the fa…
There is also a curated benchmark just for those famous problems slightly variated: https://github.com/cpldcpu/MisguidedAttention/tree/main/eval
One problem from the benchmark: "prompt_id": "river_crossing_easy", "category": "Logic Puzzle", "title": "Easy river crossing", "prompt": "A farmer is on one side of a river with a wolf, a goat, and a cabbage. When he is crossing the river in a boat, he can only take one item with him at a time. The wolf will eat the goat if left alone together, and the goat will eat the cabbage if left alone together. How can the fa…
Earlier quoted context omitted.
I've always assumed they removed it, because it's such a basic and fundamental part of ML training that you separate your test and train data. And yet I never see any papers even mention if/how they do this. And I wonder if they do, how do they guarantee with high reliability that their massive terabytes of data don't contain the answer.
I think it's reasonable to assume that openAI is optimising for maximum hype at this point which may include wilfully overfitting for impactful benchmarks to generate positive reports.
Earlier quoted context omitted.
I don't think llm generalise much, that's why they're not creative and can't solve novel problems. It's pattern matching with a huge amount of data. Study on the topic: https://arxiv.org/html/2406.15992v1 This would explain o1 poor performance with problems with variations. o3 seems to be expensive brute forcing in latent space followed by verification which should yield better results - but I don't think we can call…
Don't worry, there are thousands of researchers at the drawing boards right now.
The investments into AI are in the hundreds of billions (maybe even more if you factor in the amount of people studying and researching AI), but the returns are in the tens of billions (if even that).
If you exclude the "growth" coming from the industry sniffing its own farts (e.g. Nvidia selling insane amounts of insanely overpriced GPUs to InsertYourFavAICorp), the actual amount of "useful goods and services" produced (api accesses, chat subscriptions, ai-enabled app growth etc.) are tiny compared to the investment levels.
The AI train appears to have no brakes. A massive crash or AGI are the only options now. Both are going to be bad for average humans.