Earlier quoted context omitted.
why...?
I guess we found the target audience for models that slather READMEs in emojis.
The thing we must keep in mind with any of the AI-slop style writing is that it was reinforced behavior because humans wanted it.
Show HN: I RL-trained an agent that trains models with RL (for ~$1.3k)
21–30 of 54 posts
Re: Show HN: I RL-trained an agent that trains models with RL (for ~$1.3k)
#22No idea why you got downvoted into oblivion with the context post. Cool idea!
Because HN detects AI comments and automatically makes them dead. I vouched for it as it's important for this post as context.
Re: Show HN: I RL-trained an agent that trains models with RL (for ~$1.3k)
#23I'm curious to whether the recursively trained models degenerate to troglodytes after a couple of generations.
Re: Show HN: I RL-trained an agent that trains models with RL (for ~$1.3k)
#24[dead]
Re: Show HN: I RL-trained an agent that trains models with RL (for ~$1.3k)
#25I'm curious to whether the recursively trained models degenerate to troglodytes after a couple of generations.
I am also curious! The inner-RL-loop models are only trained once, then disgarded. But the outer-RL-loop model is trained on the same tasks over and over again. I imagine it would overfit after many more steps, but perhaps with a larger set of diverse tasks, the model would simply improve.
Re: Show HN: I RL-trained an agent that trains models with RL (for ~$1.3k)
#26AI trains AI already, agents are happy to spin up real training pipelines for deep learning or regression models or whatever you want right? I guess the advantage to your project is that it provides a framework to allow the agent to access extra compute?
Yes I'd heard the labs (Anthropic mostly) speaking about LLMs training LLMs, so I wanted to make things a little more concrete and test it out myself! Essentially you are correct though, my framework allows the agent access to compute, but also the agent itself is being trained to become better at training models with that compute.
i remeber reading in one of the release blog posts that that version was the "first that codex helped train"
Re: Show HN: I RL-trained an agent that trains models with RL (for ~$1.3k)
#27Earlier quoted context omitted.
Yes I'd heard the labs (Anthropic mostly) speaking about LLMs training LLMs, so I wanted to make things a little more concrete and test it out myself! Essentially you are correct though, my framework allows the agent access to compute, but also the agent itself is being trained to become better at training models with that compute.
i remeber reading in one of the release blog posts that that version was the "first that codex helped train"
Yes exactly, and also there is auto-research by Karpathy, Fable-5 on parameter golf etc which point to the direction of highly-assisted / autonomous AI training as the future. Wild.
Re: Show HN: I RL-trained an agent that trains models with RL (for ~$1.3k)
#28Earlier quoted context omitted.
I think the counter point for these projects is that you may not need a deep understanding if you can measure the outcome. While this may not be true every time today, it plausibly will be in the future - making the activity worthwhile.
Well, you say that, but when "measuring" anything in RL, that measurement itself is not always obvious. That is, creating the scoring system/judge models etc for RL is not easy at all. You can easily create an RL loop which is getting better and improving its scores, but actually the result is totally garbage, because you're measuring the wrong thing.
What do you see wrong with the OPs reward design?
Re: Show HN: I RL-trained an agent that trains models with RL (for ~$1.3k)
#29Lots of emoji in that readme. Was it mainly codex?
Mainly Fable, but It was me who wanted to emojis added hah. I also of course edited the README by hand (crazy I know), but the code is entirely fable
i read around launch that anthropic will fallback to opus if fable is used for frontier LLM development. did you run into anything like that?
Re: Show HN: I RL-trained an agent that trains models with RL (for ~$1.3k)
#30I think that En Dash is supposed to be Tilde
–$1.3k -> ~$1.3k