Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Augento (YC W25) – Fine-tune your agents with reinforcement learning

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 63 posts

Re: Launch HN: Augento (YC W25) – Fine-tune your agents with reinforcement learning

#51

Post by resiros »

Congrats on the launch! The idea sounds very interesting on paper. The tricky part though is the reward function.

Providing finetuning as a service works because the friction with finetuning is operational (getting the GPUs, preparing the training...), so the vendor can take care of that and give you an API. The work becomes straightforward and doesn't require much preparation - give us some examples and we'll provide you a model that works well with these and hopefully generalizes.

RL as a service is much trickier in my opinion. The friction is not only operational. Getting RL to work (at least from my probably deprecated 10-year-old knowledge) is much harder because the real friction is in building the right reward function. I've skimmed your docs, and you don't say much about reward functions other than the obvious.

I think to get this to work, you need to improve your docs and examples a lot, and maybe focus on some recurrent use cases (e.g., customer support agent) with clear reward functions. Perhaps provide some building block reward functions and some UI/tools to help create them. Basically, find a way to remove the real friction on how to use RL in my agent - the reward function part.

In any case, congrats again on the launch. We're building an LLMOps platform (see my profile), there might be collaboration/integration potential, write me if you think that's interesting.

Re: Launch HN: Augento (YC W25) – Fine-tune your agents with reinforcement learning

#52

Post by lukasego »

georgeck wrote: post #41

Is this solution similar to the Direct Preference Optimization (DPO) [1] provided by another 'fine-tuning as a service' - OpenPipe? [1] https://docs.openpipe.ai/features/dpo/overview

No, DPO avoids a Reinforcement Learning training loop. For the current iteration on verifiable domains, our method is GRPO. Let me elaborate: DPO is for preference learning - each data sample in the dataset contains 2 pieces: preferred and non-preferred responses (what the model should avoid generating). DPO optimizes for the preferred response between the 2. That means, DPO is one effective method for making a model learn sentiment or preference. We call a generalization of this alignment mode - it's on our roadmap. On the current GRPO implementation side, dataset needs on Augento are simpler: Just the prompt, and some captured context if you like - it's then the reward function that scores the model generations. Currently, with GRPO, training is done on verifiable domains. Not preference, but one piece of output will be judged by a deterministic reward function, or by a reward model (which the user decides - you can decide it through defining the reward function).

(EDIT: Would you use DPO? Do you have experience with it or needs?)

Re: Launch HN: Augento (YC W25) – Fine-tune your agents with reinforcement learning

#53

Post by lukasego »

filipeisho wrote: post #48
dang wrote: post #8

Earlier quoted context omitted.

Ha - here's the advice I give to YC startups about making demo videos for HN: "What works well for HN is raw and direct, with zero production values. Skip any introductions and jump straight into showing your product doing what it does best. Voiceover is good, but no marketing slickness—no fancy logos or background music!" I guess there's zero production values and zero production values...

Totally agree. Raw is great, but energy matters too. If the person sounds bored, it's hard to get excited about the product—even if it's amazing. Passion is contagious.

That's true, thanks for the feedback! In the end, it wasn't boredom, but the long work - put too much energy into the platform ;) Taking it to heart for the next one!

Re: Launch HN: Augento (YC W25) – Fine-tune your agents with reinforcement learning

#54

Post by lmeierhoefer »

thebeardisred wrote: post #36

I tell you what I don't like, the game y'all are playing with billing of Slack users: Where do you want to access #ext-customers? The organization you select is where you’ll find this channel in Slack. Admins will get a chance to review everything before you start collaborating. Tip: Add this Slack Connect channel to the organization that’s already connected with P2P Industries, or where you have similar channels.

Yes, we wanted to incentivize, that people who want to use the platform (redeeming the $20 training credits) are also joining a slack channel, so we can give direct support. We should have pointed this out in the post.

Re: Launch HN: Augento (YC W25) – Fine-tune your agents with reinforcement learning

#55

Post by [deleted] »

thebeardisred wrote: post #36

I tell you what I don't like, the game y'all are playing with billing of Slack users: Where do you want to access #ext-customers? The organization you select is where you’ll find this channel in Slack. Admins will get a chance to review everything before you start collaborating. Tip: Add this Slack Connect channel to the organization that’s already connected with P2P Industries, or where you have similar channels.

[deleted]

Re: Launch HN: Augento (YC W25) – Fine-tune your agents with reinforcement learning

#56

Post by lmeierhoefer »

foundzen wrote: post #38

Only 20 training samples improved llm performance, that sounds unrealistic! My experience with RLHF for LLM perf differs. Can you be more specific about the case where you achieved this and share technical details about how do you do that?

We are not doing RLHF but fine-tuning directly on a reward function. Our task was around improving a coding agent, coding in JSONata(https://jsonata.org).

GPT4o is quite bad in this, as there are not too many JSONata snippets on the internet. We collected 20 coding problems; the reward function then just assigned a scalar value based on whether the code output of the model was syntactically correct or not (Most interestingly, we found that by optimizing the syntax, it also got better at getting the semantics correct)

I think the discrepancy between our result with direct RL and your experience with RLHF comes from the fact that RLHF is built around non-verifiable/subjective domains, where intrinsically, the reward signal obtained by the HF-proxy is weak(er), i.e. for the same training scenario/prompt you need more samples to get to the same gradient.

Re: Launch HN: Augento (YC W25) – Fine-tune your agents with reinforcement learning

#57

Post by lukasego »

lukasego wrote: post #52
georgeck wrote: post #41

Is this solution similar to the Direct Preference Optimization (DPO) [1] provided by another 'fine-tuning as a service' - OpenPipe? [1] https://docs.openpipe.ai/features/dpo/overview

No, DPO avoids a Reinforcement Learning training loop. For the current iteration on verifiable domains, our method is GRPO. Let me elaborate: DPO is for preference learning - each data sample in the dataset contains 2 pieces: preferred and non-preferred responses (what the model should avoid generating). DPO optimizes for the preferred response between the 2. That means, DPO is one effective method for making a model…

To add, there is the important distinction to be made between RLHF (Reinforcement Learning with Human Feedback) and RL. DPO is a simpler and more efficient way to do RLHF. In its current iteration, Augento does RL (using the term coined by OpenAI: Reinforcement Fine-tuning) which improves model performance on domains where there exists a verification function for the answer that you can use for scoring, rather than a preferred answer such as DPO needs. But as said, such preference mode is on the roadmap.

Re: Launch HN: Augento (YC W25) – Fine-tune your agents with reinforcement learning

#58

Post by lukasego »

resiros wrote: post #51

Congrats on the launch! The idea sounds very interesting on paper. The tricky part though is the reward function. Providing finetuning as a service works because the friction with finetuning is operational (getting the GPUs, preparing the training...), so the vendor can take care of that and give you an API. The work becomes straightforward and doesn't require much preparation - give us some examples and we'll provid…

Thanks for this very lucid post! For many use cases such as coding, formatting, it's very clear for the users how to define the reward function. Fore more intricate ones, you're right in that it can be tricky. I like your ideas of trying to provide tools to help here, and offering recurring reward functions as templates that will only need slight adaptations. It will be the user defining it, but there's a path to simplification. - The operational friction with getting the GPUs, optimizing compute and preparing the training are hard for RL, hence we got these things out of the way. - Thanks for the very thoughtful suggestions and contacting, great input!

Re: Launch HN: Augento (YC W25) – Fine-tune your agents with reinforcement learning

#59

Post by hannesfur »

abc-1 wrote: post #32
hannesfur wrote: post #31

Earlier quoted context omitted.

you mean fine tuning that feels like SFT but is different (since you can't use that with reasoning models) built around the DeepSeek class of models?

I just want to fine tune deepseek v3 chat but it’s not possible or easy for regular consumers

Fireworks has you covered now: https://fireworks.ai/blog/fine-tuning-deepseek-models :)

Re: Launch HN: Augento (YC W25) – Fine-tune your agents with reinforcement learning

#60

Post by jacobross »

Man, this is awesome. I've been obsessed with this idea since reading up on end-to-end RL used in reasoning models and OpenAI using it with Deep Research.

Seems like the most powerful agents will make use of some form of RL or advanced learning.

I'm not from an ML/DL background but these ideas are fascinating and I've begun self-teaching myself some RL.

I'm curious as to how long this took to build and any advice for someone wanting to learn more about RL in this context?

Thanks!

Post reply on HN