Do you also think they somehow stole the codeforces problems before they were even written or you are willing to believe the #175 global rank there?
I dont think codeforce claims to contain novel unpublished problems. But i'm not saying it's what they did, just that it's a possibility that should be considered till/if it is debunked.
frankly i’m not sure what standard you would possibly consider a debunking
codeforces constantly adds new problems that’s like the entire point of the contest, no?
I think that lamentations about real world data running out is misplaced. We can multiply data with slight variations which might lead to better resilience and more accurate model's responses for novel problems.
They are highly effective pattern matchers. You change the pattern, it won't work. I don't remember who, but most likely @tszzl (roon), commented on x that they still trained the traditional way, and there is no test time compute (TTC) or Montecarlo Tree search (like Alpha Go) in o1 or o3. If that is true, then it's still predicting the next word based on it's training data. Likely to follow the most probable path -…
I recently watched two interviews with OpenAI researchers where they describe that the breakthrough of o-series (unlike GPT series) is to focus on test time compute as they are designed to “think” more specifically to avoid pattern matching. Noam Brown https://youtu.be/OoL8K_AFqkw?si=ocIS0YDXLvaX9Xb6&t=195 and Mark Chen https://youtu.be/kO192K7_FaQ?si=moWiwYChj65osLGy
Thank you, this is helpful. The post on X was seemingly wrong.
Is it just an open secret that the models are currently just being hardcoded for random benchmarks? Seems weird that people would be asking Putnam problems to a chatbot :/
There are tests they are passing that they can't be hardcoded for by design. They still have all kinds of flaws and inconsistency but getting upset they answer "2+2=4" because someone trained them on what the answer to 2+2 is supposed to be is silly.
One experiment I would love to see, although not really feasible in practice, is to train a model on all digitized data from before the year 1905 (journals, letters, books, broadcasts, lectures, the works), and then ask it for a formula for mass-energy equivalence. A certain answer would definitely settle the debate on whether pattern recognition is a form of intelligence ;)
> ask it for a formula for mass-energy equivalence Way too easy. If you think that mass and energy might be equivalent, then dimensional analysis doesn’t give you too much choice in the formula. Really, the interesting thing about E=mc^2 isn’t the formula but the assertion that mass is a form of energy and all the surrounding observations about the universe. Also, the actual insight in 1905 was more about asking the…
It’s nontrivial why it’s mc^2 and not 1/2 mc^2, since kinetic energy generally is 1/2 mv^2
but e=mc^2 is just an approximation e: nice, downvoted for knowing special relativity
Can you elaborate? How is E=mc^2 an approximation, in special relativity or otherwise? What is it an approximation of?
e=mc^2 is only correct for objects at rest. The full equation takes into account velocity, but for "low" speeds where v<<c, the term is close enough to zero than E=mc^2 is still a good approximation.
Is it just an open secret that the models are currently just being hardcoded for random benchmarks? Seems weird that people would be asking Putnam problems to a chatbot :/
Not hardcoded, I think it's just likely that those problems exist in its training data in some form
Yea, people have a really hard time dealing with data leakage especially on data sets as large as LLMs need.
Basically if something appeared online or was transmitted over the wire should no longer be eligible to evaluate on. D. Sculley had a great talk at NeurIPS 2024 (same conference this paper was in) titled Empirical Rigor at Scale – or, How Not to Fool Yourself
Basically no one knows how to properly evaluate LLMs.
https://s.h4x.club/bLuNed45 - it's more crazy to me that my wife CAN in fact read this stuff easily, vs the fact that an LLM can't. (for anyone who doesn't feel like downloading the zip, here is a single image from the zip: https://s.h4x.club/nOu485qx )
have you been trying to provide it as an image directly? if so, doesn’t surprise me at all. really thanks for sharing!
My wifes particular area of research is using the capitalist system to "re build" broken slave family trees, she flys around the US going to archives and getting contracts and receipts for slaves, figures out how they got traded, and then figures out where they ended up, and then "re links" them to their their family to best of her ability. Although her area of research isn't particularly overflowing with researchers, there are still a lot of people like her who just have this very tacit knowledge among each other, they email around a lot and stuff, knowledge like who was running a region during a period, ofc they publish, but it's a small field and it's all extremely poorly documented. Was watching the Adam Brown interview with Dwarkesh Patel the other day and he said for his work LLMs are better than bothering an expert in an area of his field with a question, I'm not sure people in her field are able to do this as readily. Franky, I've yet to find a novel/or good use for an LLM in her work. I often joke that her and "her people" are going to be the last ones with jobs if they don't transfer their knowledge into LLMs, ha! :)
The paper mentions that on several occasions the LLM will provide a correct answer but will either take big jumps without justifying them or will take illogical steps but end up with the right solution at the end. Did you check for that?
No, I don't know enough math to test the logic, only the check questions against their expected answers in https://anonymous.4open.science/r/putnam-axiom-B57C/data/Put...
Putnam problems need to actually be graded, often the answer itself is trivial.