Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Auto-research with codex: How I achieved a 232x Faster Kernel

sankalp.bearblog.dev

Post reply on HN
Print view View original

71–80 of 104 posts

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#71

Post by musebox35 »

augment_me wrote: post #57

One thing worth to note in the competition is that 8 out of the 10 top solutions, which all happened to be optimized this way completely broke at any other input than the competition ones. The only solutions that did not break when tested with OOD shapes were made by experts who know a lot about GPU programming and that did not create 25k lines of CUDA but followed and adjusted their solution in reasonable bounds. Th…

This is one of the dilemmas that I am trying to wrap my head around. I love optimizing software pipelines, which often boils down to figuring out the operational constraints that the compiler and the generic libraries can’t assume. Then I exploit these to squeeze out performance. But in a world I can start from scratch and code a domain specific solution from line zero in a matter of hours/days, I do not need general libraries as much as I used to. On one hand the code won’t be as well tested as a good general library. On the other hand, it also won’t have a plethora of library bugs that are there because the code is generic and opaque. One counter argument is that things are never static and you can’t have specific code for too long. A counter to that is that you can then change the code to be specific to the new reality at very low cost. This is the mental loop I ride constantly. Disclaimer: My circumstances are definitely not general, I am not writing code that is truly large scale.

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#72

Post by embedding-shape »

rrhjm53270 wrote: post #6
Almondsetat wrote: post #2

In the last couple of days I wanted to try out the new definitive DeepSeek v4 releases. I gave it the repository of a semi-abandoned video compression codec and I told it to perform the usual benchmark -> profile -> verify -> research -> improve loop. I specifically chose this codec because the authors include a verifier for the bitstream to make sure you don't break stuff if you want to try your own implementation.…

I tried kernel autoreasearch using DeepSeek-V4-Flash as well. It spent about 1-2 hours to complete the FlashAttention optimization job ( https://github.com/fengwang/FA5090/tree/main/v7 ) and cost me only $0.2. I believe we are ready to offload a lot of this kind well-defined constrained optimization problems to AI Agent autoresearch.

> I tried kernel autoreasearch using DeepSeek-V4-Flash as well. It spent about 1-2 hours to complete the FlashAttention optimization job

Doing the same, re-implementing a lot of LLM/diffusion models in Rust+CUDA for my own usage, usually the initial implementation takes 1-2 days (of 100% autonomous work) then I put an agent to optimize the implementation which tends to get close to SOTA performance within another day or two.

As long as you can point the agent at "This is the correct baseline, make sure any optimizations still pass this", seemingly you can leave them and they come back after N hours with a faster program that just works.

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#73

Post by qarl2 »

revetkn wrote: post #66
qarl2 wrote: post #18

Earlier quoted context omitted.

I've had a lot of success decompiling old video game ROMs in exactly this way. Like you say - give it a way of verifying correctness - put it in a loop - and they are quite surprising. https://github.com/qarl/arcade-js

Very cool, me too! I've been working on Final Fantasy Legend (Game Boy and WonderSwan Color) and King's Bounty (PC - DOS). It's great for reversing. Really interesting to see the guts of the games, including bugs.

The most interesting thing I've found so far is the anti-tampering mechanisms.

In Time Pilot - there are three routines that are called constantly from inside the main loop. Each routine computes the checksum of the other routine's code to see if it's been modified. If so it jumps into random junk data.

There are other less exotic routines that make sure the copyright string hasn't been modified, etc.

https://github.com/qarl/arcade-js/blob/main/games/timeplt/id...

Fascinating.

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#74

Post by soulofmischief »

msdz wrote: post #59
dsign wrote: post #23

Earlier quoted context omitted.

Anecdotally, I saw Opus 5 come up with a complicated loop unrolling technique when I asked it to implement a simple biquad in SIMD, missing a simpler solution. Maybe it was a downgraded session, who knows. That SIMD instruction set, the one for the ESP32-P4, is not very popular and all the documentation it has is a couple of blog posts. So I'm pretty sure it has at most seen some code for a predecessor during its tra…

> Maybe it was a downgraded session, who knows. Off-topic, but imagine us collectively being okay with (or powerless to do anything about) this sentiment about any other software service provided like two years ago. How the times have changed…

Were other services so historically resource-constrained?

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#75

Post by piker »

josu wrote: post #61
augment_me wrote: post #57

One thing worth to note in the competition is that 8 out of the 10 top solutions, which all happened to be optimized this way completely broke at any other input than the competition ones. The only solutions that did not break when tested with OOD shapes were made by experts who know a lot about GPU programming and that did not create 25k lines of CUDA but followed and adjusted their solution in reasonable bounds. Th…

Overfitting to the input is part of the meta in this type of challenges. The goal is not to create good, general or maintainable code. The only goal is to produce the fastest code.

Nonetheless gives appropriate context to the speed up for folks like myself who were just skimming.

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#76

Post by thebruce87m »

_zoltan_ wrote: post #7
Almondsetat wrote: post #2

In the last couple of days I wanted to try out the new definitive DeepSeek v4 releases. I gave it the repository of a semi-abandoned video compression codec and I told it to perform the usual benchmark -> profile -> verify -> research -> improve loop. I specifically chose this codec because the authors include a verifier for the bitstream to make sure you don't break stuff if you want to try your own implementation.…

This is exactly how I use it. I mean not on abandoned repos, but in a benchmark - profile - verify - research - improve loop.

What’s the best way to loop it?

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#77

Post by augment_me »

musebox35 wrote: post #71
augment_me wrote: post #57

One thing worth to note in the competition is that 8 out of the 10 top solutions, which all happened to be optimized this way completely broke at any other input than the competition ones. The only solutions that did not break when tested with OOD shapes were made by experts who know a lot about GPU programming and that did not create 25k lines of CUDA but followed and adjusted their solution in reasonable bounds. Th…

This is one of the dilemmas that I am trying to wrap my head around. I love optimizing software pipelines, which often boils down to figuring out the operational constraints that the compiler and the generic libraries can’t assume. Then I exploit these to squeeze out performance. But in a world I can start from scratch and code a domain specific solution from line zero in a matter of hours/days, I do not need general…

Many open-source communities are wrestling with similar questions. In many ways the generic frameworks end up being reference implementations for the specific solutions to copy and hill-climb towards, and it's as you say unclear what the point is to maintain generic things beyond that.

What I feel is lacking with the solutions that have for example agent-generated and tuned GPU kernels is that the use-cases for them are unclear. If you are a researcher on second-order optimizers, you probably want to be able to handle variable input shapes to experiment, you also might want something readable to understand intermediate steps and perhaps build on that. If you are a neolab running massive training runs for 80% of your VC funding, you need to know that every line in your training code is bitwise identical to the theory/reference because a divergent run from some LLM-generated numerical bug will set you back or bankrupt you, so you can't just plop in a random kernel even if it promises good performance.

So who is the agentically-looped end result for? Except for Openai and Anthropic of course who sold the tools.

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#78

Post by _zoltan_ »

8note wrote: post #46
_zoltan_ wrote: post #37

Earlier quoted context omitted.

you specify the goal. if the goal is achieved, it's achieved. the code the LLM writes will be read and maintained and developed further by LLMs. so it doesn't really matter what it produces as long as all the tests are green and it achieves exactly what you want it to achieve.

depends on the implementation. the claude loop/goal just decides it doesnt feel like doing it anymore and ends the loop or goal

in an other thread I've commented, but I use a benchmark - profile - verify - research - improve loop.

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#79

Post by fooblaster »

dejavucoder wrote: post #54
datakan wrote: post #28

Earlier quoted context omitted.

I don't know. That just sounds like throwing money at a problem until it goes away. I'm not convinced that is the correct path forward.

1. labs have lots of inference capacity 2. they will have domain experts working on this so their efficiency is gonna be exponentially more (can direct LLM better, save money, reach same results faster)

You can't exceed roofline performance on hardware. There is an performance cap you can hit. This recursive self improvement stuff lets you be closer to the pareto frontier, but the idea that it is leading to some exponential growth is a total pipe dream.

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#80

Post by _zoltan_ »

thebruce87m wrote: post #76
_zoltan_ wrote: post #7

Earlier quoted context omitted.

This is exactly how I use it. I mean not on abandoned repos, but in a benchmark - profile - verify - research - improve loop.

What’s the best way to loop it?

6 months ago I would have said the ralph loop plugin.

with opus 5, I'm finding that asking itself to write such a loop skill that does benchmark - profile - verify - research - improve, keep logs for each iteration, etc. (just like autoresearch), and then using the skill, works very nicely.

Post reply on HN