111–120 of 137 posts
Earlier quoted context omitted.
Try: Knowing that 1^3 + 2^3 + 3^3 + 4^3 + ... + 11^3 + 12^3 = 6084, what is the value of 2^3 + 4^3 + 6^3 + ... + 22^3 + 24^3? DeepSeek R1 (1.58-bit GGUF, running locally) has no trouble with that one.
Would you mind sharing the answer to the math question please? The only way I would try and figure it out on my own is using an LLM…
Earlier quoted context omitted.
We beat O1-preview and even many other 7B models over many math benchmarks, which was TEST set (not in training set at all). If you want to make the model fully generalist, feel free to train it over coding datasets (such as RL with passing unit tests as reward).
o1 is more than just math solver. And you cannot possibly train that much in a small model. However smaller specialized models looks to be the right way to handle world's complexity. Sort of mixture of experts on one level above. Orchestrating them will be another problem. Possible solution is generalists model "to rule them all".
We changed the URL from https://github.com/agentica-project/deepscaler to the article that gives more background information (and also points to the repo). (Submitted title was "Replicating Deepseek-R1 for $4500: RL Boosts 1.5B Model Beyond o1-preview")
Earlier quoted context omitted.
But if that's the case - what do the benchmarks even mean then?
Automated benchmarks are still very useful. Just less so when the LLM is trained in a way to overfit to them, which is why we have to be careful with random people and the claims they make. Human evaluation is the gold standard, but even it has issues.
Imagine you have an exam coming up, and the set of questions leaks - how do you prepare for the exam then?
Memorizing the test problems would be obviously problematic, but maybe practicing the problems that appear on the exam would be less so, or just giving extra attention to the topics that will come up would be even less like cheating.
The more honest approach you choose, the more indicative your training would be of exam results but everybody decides how much cheating they allow for themselves, which makes it a test of the honesty not the skill of the student.
Can CoT models also call external functions? What if they had access to a calculator?
say you have a tool:
def calculator(expr: str) -> float
then the model just needs to say:
{ "function": "calculator", "args": { "expr": "5 + 10" } }
then in your code you can easily pass that to the "calculator" function and get the result, then hand the result back to the model. Making it feel like the model can "call" an external function.
deep seek can also do this: https://www.boundaryml.com/blog/deepseek-r1-function-calling
Earlier quoted context omitted.
Automated benchmarks are still very useful. Just less so when the LLM is trained in a way to overfit to them, which is why we have to be careful with random people and the claims they make. Human evaluation is the gold standard, but even it has issues.
The question is how do you train your LLMs to not 'cheat'? Imagine you have an exam coming up, and the set of questions leaks - how do you prepare for the exam then? Memorizing the test problems would be obviously problematic, but maybe practicing the problems that appear on the exam would be less so, or just giving extra attention to the topics that will come up would be even less like cheating. The more honest appr…
I think it would be interesting to create a dynamic benchmark. For example, a benchmark which uses math and a random value determined at evaluation for the answer. The correct answer would be different for each run. Theoretically, training on it wouldn't help beat the benchmark because the random value would change the answer. Maybe this has already been done.
Earlier quoted context omitted.
Nah, it takes a mastermind to be creative. The tiny models IME are good at following explicit instructions for data transformations (conforming to a JSON schema is especially helpful) but I wouldn't expect one to solve a problem I haven't already solved myself like how I use the .6T full size models.
I thought conforming to a JSON schema was a solved problem for even the most garbage models by restricting the tokens it can generate?
I was pushing back against the parent's notion that tiny models would serve general purpose needs.
We changed the URL from https://github.com/agentica-project/deepscaler to the article that gives more background information (and also points to the repo). (Submitted title was "Replicating Deepseek-R1 for $4500: RL Boosts 1.5B Model Beyond o1-preview")
Out of sheer curiosity, how do vacations work for you? Do you take vacations, surly you must? I've been using HN daily for years now I can't think I've seen you take a vacation.
im very naive here but does anyone trust these benchmarks? do they mean anything to you? they seem far too easy to game and it doesn't feel like its an accurate way to really tell how these models compare to one another. seems like benchmark performance declines quite a bit if you introduce a problem that's similar to those in benchmarks but one that the model hasn't seen before
AIME has significant problems: https://x.com/DimitrisPapail/status/1888325914603516214 > Problems near identical to the test set can be found online.
Newb question. If the big boys are crawling, scraping, stealing the whole of the entire internet to train their models — how are these OS alternatives aiming to outperform them? Is it as simple as... ALSO crawl, scrape, steal the whole of the entire internet?
"Stealing the whole of the entire internet" isn't a particularly accurate mental model these days. That's more or less what people were doing back in 2023 - crawling everything and dumping as much data in as possible. It's not a great strategy to build a best-in-class model though, as a lot of the internet is junk. The SolidGoldMagikarp/davidjl bug is the kind of thing that happens if you crawl all of https://www.red…