Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mini-R1: Reproduce DeepSeek R1 "Aha Moment"

philschmid.de

Post reply on HN
Print view View original

11–17 of 17 posts

Re: Mini-R1: Reproduce DeepSeek R1 "Aha Moment"

#11

Post by senko »

madars wrote: post #9

One wonders at which point models will be sneaky enough to bypass simple eval sandboxes. The article has: # Evaluate the equation with restricted globals and locals result = eval(equation, {"__builtins__": None}, {}) but that's not enough as you can rebuild access to builtins from objects and then go from there: https://ideone.com/qzNtyu By the way, writing this greatly benefited from DeepThink-r1 while o1 just gave…

> that's not enough as you can rebuild access to builtins from objects

In this specific case, it's safe, as that wouldn't pass the regex just a few line before the eval :

    # Define a regex pattern that only allows numbers,
    # operators, parentheses, and whitespace
    allowed_pattern = r'^[\d+\-*/().\s]+$'
Commenting on the R1 reproduction, the heavy lifting there is done by huggingface's trl[0] library, and the heavy use of compute.

[0] Transformer Reinforcement Learning - https://huggingface.co/docs/trl/en/index

Re: Mini-R1: Reproduce DeepSeek R1 "Aha Moment"

#12

Post by senko »

mxwsn wrote: post #8

What's surprising about this is how sparsely defined the rewards are. Even if the model learns the formatting reward, if it never chances upon a solution, there isn't any feedback/reward to push it to learn to solve the game more often. So what are the chances of randomly guessing a solution? The toy Countdown dataset here has 3 to 4 numbers, which are combined with 4 symbols (+, -, x, ÷). With 3 numbers there are 3!…

> What's surprising about this is how sparsely defined the rewards are

Yeah, I would expect the rewards not to be binary. One could easily devise a scoring function in range [0-1] that would depend on how far the model is from the "correct" answer (for example, normalized Levenshtein distance). Whether that would actually do any good is anyone's guess.

Re: Mini-R1: Reproduce DeepSeek R1 "Aha Moment"

#14

Post by perching_aix »

madars wrote: post #9

One wonders at which point models will be sneaky enough to bypass simple eval sandboxes. The article has: # Evaluate the equation with restricted globals and locals result = eval(equation, {"__builtins__": None}, {}) but that's not enough as you can rebuild access to builtins from objects and then go from there: https://ideone.com/qzNtyu By the way, writing this greatly benefited from DeepThink-r1 while o1 just gave…

> why did we ever put up with this?

Is this a serious question?

Re: Mini-R1: Reproduce DeepSeek R1 "Aha Moment"

#15

Post by indrora »

senko wrote: post #11
madars wrote: post #9

One wonders at which point models will be sneaky enough to bypass simple eval sandboxes. The article has: # Evaluate the equation with restricted globals and locals result = eval(equation, {"__builtins__": None}, {}) but that's not enough as you can rebuild access to builtins from objects and then go from there: https://ideone.com/qzNtyu By the way, writing this greatly benefited from DeepThink-r1 while o1 just gave…

> that's not enough as you can rebuild access to builtins from objects In this specific case, it's safe, as that wouldn't pass the regex just a few line before the eval : # Define a regex pattern that only allows numbers, # operators, parentheses, and whitespace allowed_pattern = r'^[\d+\-*/().\s]+$' Commenting on the R1 reproduction, the heavy lifting there is done by huggingface's trl[0] library, and the heavy use…

The fact that () and . are there miiiight enable a pyjail escape.

See also https://github.com/jailctf/pyjailbreaker

See also https://blog.pepsipu.com/posts/albatross-redpwnctf

Re: Mini-R1: Reproduce DeepSeek R1 "Aha Moment"

#16

Post by sitkack »

yurlungur wrote: post #7

https://github.com/Jiayi-Pan/TinyZero what about this one?

They do mention it here

> Note: This blog is inspired by Jiayi Pan [1] who initially explored the idea and proofed it with a small model.

I might have written it as

> Note: This blog is inspired by Jiayi Pan [1] who also reproduced the "Aha Moment" with their TinyZero [2] model.

[1] https://x.com/jiayi_pirate/status/1882839370505621655 (1.1M views btw)

[2] https://github.com/Jiayi-Pan/TinyZero

A lot of people are busy reproing R1 right now. I think this is the spark.

Re: Mini-R1: Reproduce DeepSeek R1 "Aha Moment"

#17

Post by senko »

indrora wrote: post #15
senko wrote: post #11

Earlier quoted context omitted.

> that's not enough as you can rebuild access to builtins from objects In this specific case, it's safe, as that wouldn't pass the regex just a few line before the eval : # Define a regex pattern that only allows numbers, # operators, parentheses, and whitespace allowed_pattern = r'^[\d+\-*/().\s]+$' Commenting on the R1 reproduction, the heavy lifting there is done by huggingface's trl[0] library, and the heavy use…

The fact that () and . are there miiiight enable a pyjail escape. See also https://github.com/jailctf/pyjailbreaker See also https://blog.pepsipu.com/posts/albatross-redpwnctf

That's a neat trick!

It does still require letters to be able to spell attribute/function names (unless I'm reading it wrong in that blog post).

Post reply on HN