Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepScaleR: Surpassing O1-Preview with a 1.5B Model by Scaling RL

pretty-radio-b75.notion.site

Post reply on HN
Print view View original

131–137 of 137 posts

Re: DeepScaleR: Surpassing O1-Preview with a 1.5B Model by Scaling RL

#131

Post by evilduck »

noman-land wrote: post #82
ericd wrote: post #78

Earlier quoted context omitted.

I’m not sure about that, I think the current setup is an artifact of human limits, which an immortal shared ML model doesn’t share. Also, cross-disciplinary knowledge is frequently incredibly helpful (if rare, because currently it’s the intersection of multiple uncommon specialties).

The question that's screaming at me right now is that if humans used specialization + cooperation to achieve global dominance, what happens when you pair unlimited global masterminds + cooperation?

Whatever the human equivalent of beehives and ant colonies will be.

Re: DeepScaleR: Surpassing O1-Preview with a 1.5B Model by Scaling RL

#132

Post by wnissen »

cheeko1234 wrote: post #29
simonw wrote: post #10

Does anyone have any good simple prompts for testing new "reasoning" models like this one? "Count the letter Rs in the word strawberry" is a bit dull! I'm trying this one locally using Ollama and the smallest quantized GGUF version (769MB) I could find - https://huggingface.co/bartowski/agentica-org_DeepScaleR-1.5... - I ran it like this: ollama run hf.co/bartowski/agentica-org_DeepScaleR-1.5B-Preview-GGUF:IQ3_XXS He…

I have two. One is a simple one that only deepseek R1 has passed (in my opinion): I have a 12 liter jug and a 6 liter jug. How do I get exactly 6 liters of water? Answer (Deepseek): Fill the 6-liter jug completely to obtain exactly 6 liters of water. Every other LLM I've tried include o3-mini-high: Fill the 12-liter jug completely. Pour it into the 6 liter jug. Although o3 did get it right in the reasoning: It seems…

Tried it in Llama v3.3 70b and it got it every time out of 5 samples. I guess the others are interpreting it as a trick question or something.

Re: DeepScaleR: Surpassing O1-Preview with a 1.5B Model by Scaling RL

#133

Post by newman314 »

mluo wrote: post #33
simonw wrote: post #10

Does anyone have any good simple prompts for testing new "reasoning" models like this one? "Count the letter Rs in the word strawberry" is a bit dull! I'm trying this one locally using Ollama and the smallest quantized GGUF version (769MB) I could find - https://huggingface.co/bartowski/agentica-org_DeepScaleR-1.5... - I ran it like this: ollama run hf.co/bartowski/agentica-org_DeepScaleR-1.5B-Preview-GGUF:IQ3_XXS He…

Hi, one of the lead authors for this work. We recommend using Bfloat16 (not fp16), quantization for small models can really hurt performance!

Is there a MLX version that can be added to the fullmoon iOS app?

Re: DeepScaleR: Surpassing O1-Preview with a 1.5B Model by Scaling RL

#134

Post by varispeed »

simonw wrote: post #10

Does anyone have any good simple prompts for testing new "reasoning" models like this one? "Count the letter Rs in the word strawberry" is a bit dull! I'm trying this one locally using Ollama and the smallest quantized GGUF version (769MB) I could find - https://huggingface.co/bartowski/agentica-org_DeepScaleR-1.5... - I ran it like this: ollama run hf.co/bartowski/agentica-org_DeepScaleR-1.5B-Preview-GGUF:IQ3_XXS He…

I like when the model starts to ask me how to solve something. Often find it with Sonnet when I am looking to solve a problem. The model starts becoming "curious" and treats as if I was the model and tries to nudge me to find solution...

Re: DeepScaleR: Surpassing O1-Preview with a 1.5B Model by Scaling RL

#135

Post by notjulianjaynes »

buyucu wrote: post #46
simonw wrote: post #10

Does anyone have any good simple prompts for testing new "reasoning" models like this one? "Count the letter Rs in the word strawberry" is a bit dull! I'm trying this one locally using Ollama and the smallest quantized GGUF version (769MB) I could find - https://huggingface.co/bartowski/agentica-org_DeepScaleR-1.5... - I ran it like this: ollama run hf.co/bartowski/agentica-org_DeepScaleR-1.5B-Preview-GGUF:IQ3_XXS He…

'Count the letter Rs in the word strawberry' is probably in all training sets by now.

I sometimes do the strawberry question immediately followed by "How many Rs in "bookkeeper?"

Re: DeepScaleR: Surpassing O1-Preview with a 1.5B Model by Scaling RL

#136

Post by francesco »

It looks to me like the performance reported for rStar-Math (both, in the table and in the graph) is incorrect. With a single rollout (at test time) rStar-Math achieves 50 in AIME 2024 not 26.7 as you reported. In Olympiad Bench it achieves 65.3, not 47.1 as you reported. In AMC 2023 it achieves 87.5, not 47.5 as your reported. It outperforms your model across the board. Am I reading something incorrectly?

Re: DeepScaleR: Surpassing O1-Preview with a 1.5B Model by Scaling RL

#137

Post by ekidd »

pona-a wrote: post #104
gpjanik wrote: post #18

This is finetuned to the benchmarks and nowhere close to O1-Preview in any other tasks. Not worth looking into unless you specifically want to solve these problems - however, still impressive.

I tested it on basic long addition problems. It frequently misplaced the decimal signs, used unnecessary reasoning tokens (like restating previously done steps) and overall seemed only marginally more reliable than the base DeepSeek 1.5B. On my own pet eval, writing a fast Fibonacci algorithm in Scheme, it actually performed much worse. It took a much longer tangent before arriving at fast doubling algorithm, but the…

This model can't code at all.

It does high school math homework, plus maybe some easy physics. And it does them surprisingly well. Outside of that, it fails every test prompt in my set.

It's a pure specialist model.

Post reply on HN