Is the name meant to be a jab at you know who or am I reading too much into it?
I have no idea who you mean. Why not just write it?
GEPA: Reflective prompt evolution can outperform reinforcement learning
21–26 of 26 posts
Re: GEPA: Reflective prompt evolution can outperform reinforcement learning
#22This is my research area. I just finished reviewing six NeurIPS papers (myself, no LLM involved) on LLM Agents for discovery and generation and I'm finding that evaluating LLM agents on raw performance for a task isn't as insightful anymore -- every paper is claiming state of the art 10x performance boost by {insert random acronym that devolves into combinatorial search}. Rather the true test for such algorithms is w…
I can't comment on your detailed knowledge of the state of the art, but your points resonate (particularly because I have tried to generate Julia and Lean code).
So, as with any less informed user reviewing LLM output, what you say definitely sounds plausible and correct.
Re: GEPA: Reflective prompt evolution can outperform reinforcement learning
#23This is my research area. I just finished reviewing six NeurIPS papers (myself, no LLM involved) on LLM Agents for discovery and generation and I'm finding that evaluating LLM agents on raw performance for a task isn't as insightful anymore -- every paper is claiming state of the art 10x performance boost by {insert random acronym that devolves into combinatorial search}. Rather the true test for such algorithms is w…
Do the problems you highlighted still appear with higher quality training data?
Re: GEPA: Reflective prompt evolution can outperform reinforcement learning
#24Re: GEPA: Reflective prompt evolution can outperform reinforcement learning
#25This is my research area. I just finished reviewing six NeurIPS papers (myself, no LLM involved) on LLM Agents for discovery and generation and I'm finding that evaluating LLM agents on raw performance for a task isn't as insightful anymore -- every paper is claiming state of the art 10x performance boost by {insert random acronym that devolves into combinatorial search}. Rather the true test for such algorithms is w…
mostly aligned on this. couple of thoughts:
- raw accuracy is now a "vanity" metric. so the benchmarks need to get more sophisticated, and i think they're going to have to be far more task specific than hotpot or hover. they've become like the mnist of multi hop.
- in my use of MIPROv2 and SIMBA, I see a fair amount of improvements for multi hop tasks (published some of these on hn before). I'm going to try GEPA and see how it performs. so I think we're at the start of what I would call "meta learning".. tuning across a huge search surface rather than tweaking one prompt. hyper param search for higher dim spaces.
- tokens burned should be a reported result
Re: GEPA: Reflective prompt evolution can outperform reinforcement learning
#26Earlier quoted context omitted.
Self-distillation generally refers to training a smaller model, right? I suppose for full metacognition you would use it fine-tune the existing model based on its older self?
No, training a smaller model off a more capable larger model (or an ensemble of models) is the "usual" distillation. "Self-distillation" refers to distilling from a model into a copy of itself. Which is of limited use - unless you can steer the teacher, and want the student to internalize that steering. The reason for doing self-distillation here is that we have both access to a richer representation (logit stream),…
self distillation and mutual distillation are used in MoE models. What you can do is freeze all but one expert and then train the model. If you want to do it again, you have to do self/mutual distillation to spread the training result onto the other experts.