Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I RL-trained an agent that trains models with RL (for ~$1.3k)

github.com

Post reply on HN
Print view View original

11–20 of 54 posts

Re: Show HN: I RL-trained an agent that trains models with RL (for ~$1.3k)

#11

Post by lumost »

saberience wrote: post #8

Can you explain how it works? What problems would it do well on and why? Where would it start to fail/break? What are the limitations of a system like this? When you vibe code a system in a complex area like RL, you basically have zero understanding of what its actually doing, whether its actually any good or not, what you're actually benchmarking, and when the system would fail. It's the blind leading the blind.

I think the counter point for these projects is that you may not need a deep understanding if you can measure the outcome. While this may not be true every time today, it plausibly will be in the future - making the activity worthwhile.

Re: Show HN: I RL-trained an agent that trains models with RL (for ~$1.3k)

#12

Post by saberience »

lumost wrote: post #11
saberience wrote: post #8

Can you explain how it works? What problems would it do well on and why? Where would it start to fail/break? What are the limitations of a system like this? When you vibe code a system in a complex area like RL, you basically have zero understanding of what its actually doing, whether its actually any good or not, what you're actually benchmarking, and when the system would fail. It's the blind leading the blind.

I think the counter point for these projects is that you may not need a deep understanding if you can measure the outcome. While this may not be true every time today, it plausibly will be in the future - making the activity worthwhile.

Well, you say that, but when "measuring" anything in RL, that measurement itself is not always obvious.

That is, creating the scoring system/judge models etc for RL is not easy at all. You can easily create an RL loop which is getting better and improving its scores, but actually the result is totally garbage, because you're measuring the wrong thing.

Re: Show HN: I RL-trained an agent that trains models with RL (for ~$1.3k)

#13

Post by babelfish »

saberience wrote: post #8

Can you explain how it works? What problems would it do well on and why? Where would it start to fail/break? What are the limitations of a system like this? When you vibe code a system in a complex area like RL, you basically have zero understanding of what its actually doing, whether its actually any good or not, what you're actually benchmarking, and when the system would fail. It's the blind leading the blind.

Did you read the README?

Re: Show HN: I RL-trained an agent that trains models with RL (for ~$1.3k)

#14

Post by saberience »

babelfish wrote: post #13
saberience wrote: post #8

Can you explain how it works? What problems would it do well on and why? Where would it start to fail/break? What are the limitations of a system like this? When you vibe code a system in a complex area like RL, you basically have zero understanding of what its actually doing, whether its actually any good or not, what you're actually benchmarking, and when the system would fail. It's the blind leading the blind.

Did you read the README?

The AI generated README?

Re: Show HN: I RL-trained an agent that trains models with RL (for ~$1.3k)

#15

Post by Danau5tin »

saberience wrote: post #8

Can you explain how it works? What problems would it do well on and why? Where would it start to fail/break? What are the limitations of a system like this? When you vibe code a system in a complex area like RL, you basically have zero understanding of what its actually doing, whether its actually any good or not, what you're actually benchmarking, and when the system would fail. It's the blind leading the blind.

I chose the key technical decision and direction (such as the system architecture, the tasks to train on, the stack of Tinker, Prime-RL & Runpod - all of which I know well) etc.

The problems it would do well on are training small agentic (multi-turn, tool use) task based models using the prime-rl stack, which are close to the distribution trained upon. It would likely not transfer to other training frameworks such as SLIME, ART or ROLL, it would also likely not transfer well to RL for complex agents such as coding agents etc.

It is limited due to its scale. As a single person, the resources required to train this on a more diverse dataset, with more complex tasks on a larger variety of models, is outside my abilities! I believe there are many avenues to explore to improve performance for this to be genuinely valuable. For now, this just a proof of concept to show the possible.

I would like to think I have a good understanding of RL, evaluations, and agentic systems after a few years of working on these areas. However, I will always have gaps. I use Fable to help accelerate me, and fill those gaps at the same time, from which I can learn from too.

Re: Show HN: I RL-trained an agent that trains models with RL (for ~$1.3k)

#16

Post by Danau5tin »

lumost wrote: post #11
saberience wrote: post #8

Can you explain how it works? What problems would it do well on and why? Where would it start to fail/break? What are the limitations of a system like this? When you vibe code a system in a complex area like RL, you basically have zero understanding of what its actually doing, whether its actually any good or not, what you're actually benchmarking, and when the system would fail. It's the blind leading the blind.

I think the counter point for these projects is that you may not need a deep understanding if you can measure the outcome. While this may not be true every time today, it plausibly will be in the future - making the activity worthwhile.

Yes I do agree with this. I believe we are shifting from "make the model good" (prompt/context engineering, etc) to "define good for the model" (success criteria/rubrics). Over time I believe this will become increasingly obvious (as long as model capabilities continue to increase).

Re: Show HN: I RL-trained an agent that trains models with RL (for ~$1.3k)

#19

Post by r_lee »

Danau5tin wrote: post #5
bitbasher wrote: post #3

Lots of emoji in that readme. Was it mainly codex?

Mainly Fable, but It was me who wanted to emojis added hah. I also of course edited the README by hand (crazy I know), but the code is entirely fable

why...?

Re: Show HN: I RL-trained an agent that trains models with RL (for ~$1.3k)

#20

Post by snerbles »

r_lee wrote: post #19
Danau5tin wrote: post #5

Earlier quoted context omitted.

Mainly Fable, but It was me who wanted to emojis added hah. I also of course edited the README by hand (crazy I know), but the code is entirely fable

why...?

I guess we found the target audience for models that slather READMEs in emojis.
Post reply on HN