Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Augento (YC W25) – Fine-tune your agents with reinforcement learning

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 63 posts

Re: Launch HN: Augento (YC W25) – Fine-tune your agents with reinforcement learning

#21

Post by lmeierhoefer »

filipeisho wrote: post #6

I love the idea of the product! I would trust your solution to be the best for very simple use cases but not for multistep or ReAct agents. Any thoughts / insights on that? I think the demo could be more exciting, the voice of the person talking sounds like he's bored haha

Yes, great point. We are currently working on multistep RL. The big problem with the trivial approach (give a single reward to the entire (ReAct) trajectory) is that the model receives a weak learning signal per decision (called credit assignment problem in literature), i.e. the individual decisions are not properly taken into account, which will then make the training unstable. I guess this has been an unsolved problem for a long time; however was not really looked at since generalist “planning” agents were not a big thing in RL until o1/DeepSeek.

IMO, the most promising approach to this is something along the lines of MA-RLHF (https://arxiv.org/abs/2410.02743) but adapted to the real world, i.e., spitting up the reward model to grade individual actions inside the trajectory to reduce the “attention distance” between the reward and the decision.

Re: Launch HN: Augento (YC W25) – Fine-tune your agents with reinforcement learning

#24

Post by hannesfur »

codingwagie wrote: post #10
hannesfur wrote: post #5

Earlier quoted context omitted.

In a sense: You are not wrong! But when we got started we thought it is way easier than it actually was. Procuring powerful GPUs alone is difficult, collecting proper data too. But of course you can still do everything yourself. If you want to give this a try yourself, I would recommend taking a look at torchtune ( https://github.com/pytorch/torchtune ).

Should I just put a ui on top of: https://aws.amazon.com/blogs/machine-learning/customize-deep... And charge for it?

Yes, you could do that. However, you would have created a different platform than Augento. Maybe we should make the distinction clearer though.

The blog article you are referring to uses another method to fine-tune models that many other big platforms like Together AI (and even OpenAI themselves) are already supporting: Supervised Fine Tuning (SFT). We are doing Reinforcement Learning using GRPO instead. SFT has the big caveat that it requires good prompt-completion datasets to work, which are rare/hard to curate for many use cases. For GRPO, you (the programmer) don’t even need to know what the correct answer is as long as you can decide if it’s a good answer (P?NP) at its heart, essentially.

Re: Launch HN: Augento (YC W25) – Fine-tune your agents with reinforcement learning

#25

Post by Zollerboy1 »

zoogeny wrote: post #14

I want to make sure I am understanding this. If I have an application that uses OpenAI models then this service can act as a proxy between the my application and the actual OpenAI service. It logs all of the requests that get sent to the OpenAI api. At some later time, I can go through and choose a subset of the API calls and mark them (I'm guessing as good or bad) and these get converted into a training set. I then…

Wow! Thanks for taking the time to think through it. Yes, you are exactly right! I couldn’t have described Augento better than this myself. We actually want to make writing a reward function completely optional and build some RLHF (Reinforcement Learning from Human Feedback) loop soon. One of our long-term goals is to bring the cost of RL down so the barrier of entry to fine-tuning big models is not as high as it currently is.

Re: Launch HN: Augento (YC W25) – Fine-tune your agents with reinforcement learning

#26

Post by spmurrayzzz »

zoogeny wrote: post #14

I want to make sure I am understanding this. If I have an application that uses OpenAI models then this service can act as a proxy between the my application and the actual OpenAI service. It logs all of the requests that get sent to the OpenAI api. At some later time, I can go through and choose a subset of the API calls and mark them (I'm guessing as good or bad) and these get converted into a training set. I then…

I agree with you that the market exists and, as a result, solutions to this problem also exist in abundance. The most difficult part about a building a product like the one presented here is making something super generic that works for a wide swath of use cases. If you simplify the stack to more bespoke/custom approach, the build burden decreases exponentially.

For the folks who are already technical in this vertical, especially ones that leverage a low cardinality architecture (one or two models, small subset of tasks, etc), this type of thing is quite easy to build yourself first as a working prototype and then only slightly more difficult to productionize & automate.

I have some in-house infra that does similar work: monitors inputs and outputs from models, puts them in a UI for a human to score/rank, preps a DPO dataset for training, kicks off training run. The total amount of calendar time I spent from prototype to production was roughly two person weeks. Changing the human intervention mechanism to an automated reward function would be an hour or two worth of work. If I had to make this work for all types of users, tasks, and models — no shot I'd have the time personally to pull that off with any reasonable velocity.

With that said, having a nice UI with great observability into the whole process is a pretty big value-add to get out of the box as well.

(EDIT: for clarity, not affiliated all with the OP project/org)

Re: Launch HN: Augento (YC W25) – Fine-tune your agents with reinforcement learning

#27

Post by hannesfur »

noosphr wrote: post #13
hannesfur wrote: post #5

Earlier quoted context omitted.

In a sense: You are not wrong! But when we got started we thought it is way easier than it actually was. Procuring powerful GPUs alone is difficult, collecting proper data too. But of course you can still do everything yourself. If you want to give this a try yourself, I would recommend taking a look at torchtune ( https://github.com/pytorch/torchtune ).

People not in the field have no idea just how distorted the market is right now. I was working at a startup doing end to end training for modified BERT architectures and everything from buying a GPU - basically impossible right now, we ended up looking at sourcing franken cards _from_ China. To the power and heat removal - you need a large factories worth of power in the space of a small flat. To pre-training somethi…

You are right. If you want/can pay out of your own pocket, RunPod (https://www.runpod.io) deserves a shoutout here. We rented GPUs from them (they have them and they are cheaper and more available than Lambda Labs) until we convinced AWS to give us capacity blocks. But in general the prices for GPUs as well as their scarcity is really crass and unlike mining you can't really use gaming or franken cards as a fallback. I can count the GPUs we can do this on (even for relatively small models) on one hand.

Re: Launch HN: Augento (YC W25) – Fine-tune your agents with reinforcement learning

#28

Post by lmeierhoefer »

brap wrote: post #16

Congrats on the launch! Noob question - from my understanding, SoTA proprietary models already provide APIs for fine tuning, I'd say it's only a matter of time before they provide RL based APIs, no?

Thanks! Yes, absolutely. OpenAI already has a reinforcement learning fine-tuning API in closed beta. However, historically, they’ve always left significant room for integrations into users systems. E.g. in the current demo of their RL fine-tuning platform, you can only select predefined reward functions and must manually upload the query datasets. I think that's the reason why so many open-source supervised fine-tuning companies exist.

My long-term take is that the agent economy will be around a few labs providing (partially open-source) foundational models where you don’t want to be part of the competition, as this will be the AI equivalent of the high-frequency tradings arms race). And above that will sit an infrastructure layer, specializing these very models to the users domains. OpenAI/Anthropic/… RL finetuning will be a part of that infrastructure layer, but so will open-source-model alternatives like ours.

Re: Launch HN: Augento (YC W25) – Fine-tune your agents with reinforcement learning

#29

Post by lmeierhoefer »

serjester wrote: post #23

Are you worried about OpenAI and every other big lab eventually doing this? It’s going to be hard to get anyone to hand over this kind of data / control without a giant name attached.

No, not really. As I posted in the other thread, there are quite a few historical examples of why the big labs won’t take the entire market. They will push to publish something like this soon. Also, I think reinforcement fine-tuning is more convenient on the data-control side. Our platform allows you to self-host the reward function, so we only need the prompts; everything else can theoretically stay on the user side.
Post reply on HN