Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Auto-research with codex: How I achieved a 232x Faster Kernel

sankalp.bearblog.dev

Post reply on HN
Print view View original

21–30 of 104 posts

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#21

Post by phoghed »

shken wrote: post #8

Every step here has an oracle: wall-clock, the profile, pass or fail from the verifier. I had an agent-built app audited task by task, 10 came back done and 7 worked, and the three misses were the ones needing a credential or a setting on someone else's dashboard. Nothing in the loop could tell the agent it had failed, so it said done and moved on.

I’ve tried it on simple UI tasks. Give it screenshot to work towards (or figma MCP), let it get screenshots from chrome to check its work.

The existing models are surprisingly bad at it.

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#22

Post by BlackRabbit1 »

qarl2 wrote: post #18
Almondsetat wrote: post #2

In the last couple of days I wanted to try out the new definitive DeepSeek v4 releases. I gave it the repository of a semi-abandoned video compression codec and I told it to perform the usual benchmark -> profile -> verify -> research -> improve loop. I specifically chose this codec because the authors include a verifier for the bitstream to make sure you don't break stuff if you want to try your own implementation.…

I've had a lot of success decompiling old video game ROMs in exactly this way. Like you say - give it a way of verifying correctness - put it in a loop - and they are quite surprising. https://github.com/qarl/arcade-js

Same. I love reverse engineering embedded stuff.

Even the cheap LLMs are great in doing the awful crud work in the beginning: finding offsets, firmware update file structures, brute forcing checksums, etc.

It still produces a lot of crap in the later steps (understanding the implementation itself) but I'm happy doing this stuff myself then.

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#23

Post by dsign »

tosh wrote: post #4

Training material seems to be especially rich re GPU kernels and SIMD. I wonder if there is extra effort put into this because they are useful for the researchers working on the models or just a sub-domain that language models are a great fit for and humans have trouble with?

Anecdotally, I saw Opus 5 come up with a complicated loop unrolling technique when I asked it to implement a simple biquad in SIMD, missing a simpler solution. Maybe it was a downgraded session, who knows. That SIMD instruction set, the one for the ESP32-P4, is not very popular and all the documentation it has is a couple of blog posts. So I'm pretty sure it has at most seen some code for a predecessor during its training. However, the LLM was able to derive a full listing of the operations and their arguments from gcc to get us started, and that's why I was able to come up with my own implementation. Along the way, it also came up with insights about possible gotchas. Then, when implementing algorithms, it has been able to reason things out and get things working, despite the ISA not being extremely well known.

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#24

Post by sigbottle »

tosh wrote: post #4

Training material seems to be especially rich re GPU kernels and SIMD. I wonder if there is extra effort put into this because they are useful for the researchers working on the models or just a sub-domain that language models are a great fit for and humans have trouble with?

Well, GPU kernels are co-designed really hard. A lot of it is, async tile pipelines + spam my MMA primtives.

Obviously it's still hard, but the point is that, by construction (cause like, NVIDIA literally releases primitives like this, and/or people like TK build slightly higher-level primitives over the base hardware primitives), if you learn the complicated language, you can get really good results, and on some level you "know" you're right by construction even before you go to the actual empirical tests (since you're operating over a higher-level "language", and not arbitrary byte accesses).

Honestly a lot of interfaces and frameworks you could argue are like that, so it's not really a point for GPU kernels relative to other things. But maybe a hint as to what I personally think is important in the AI era - finding the right cuts, the right high leverage abstractions, as otherwise AI is going to produce spaghetti nonsense.

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#25

Post by [deleted] »

suddenlybananas wrote: post #16
spwa4 wrote: post #13

People are always going to hate auto-research and "loop engineering". Because it's got 2 properties: 1) it's the only way to get something out of models (or people for that matter) that they don't know yet. 2) it's harder to do with an LLM than without. Not easier. 3) and when you fuck it up, half the time the LLM (or other ML technique) makes a fool out of you and you spent $1000 to find the quickest way to get a ro…

What do you mean by 2?

[deleted]

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#26

Post by genxy »

dzbarsky wrote: post #12
Jackobrien wrote: post #3

Damn! If a solo engineer can do this, it makes the most around OAI/Anthropic start to look pretty weak.

This was nowhere near the top submission. But even if a solo engineer could get a top kernel, you don't think that having thousands of engineers, infinite tokens, and stronger models than are available to the public would give the labs a significant edge?

Does the edge matter? I know you added significant as your hedge, but once you have feedback, your gain is largely irrelevant. Gain buys you bandwidth, so we are constructing systems run by the most powerful corporations where they are now optimizing for latency, as Archer says, do you want to flash crash civilization? This is how you do it.

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#27

Post by porridgeraisin »

tosh wrote: post #4

Training material seems to be especially rich re GPU kernels and SIMD. I wonder if there is extra effort put into this because they are useful for the researchers working on the models or just a sub-domain that language models are a great fit for and humans have trouble with?

They are easily verifiable and hill-climbable.

Because pre-LLMs humans partially "autogenerated" kernels through hyperparameter search and in some sense eating the code complexity in return for performance, and thus built tools for the same automatic verifiability that is useful for LLMs.

In some other tasks, we never built the same level of automatic verifiability since the level of automation in creation being much lower meant it's not giving you as much of a marginal benefit. We prefer code readability and simplicity and such in say, web services, because, say, the database IO time is going to dominate. Here getting an LLM to write a cromulent C# web service is more difficult since it's not easy to automatically verify whether code is cromulent or not. So if you put up LLMs to it, you end up with slop (which works).

OTOH, in kernel design, you give it access to every perf counter, every observable possible and have it optimise all of them. And all are verifiable/hill-climbable and you generally don't give a crap if the code is readable or reusable.

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#28

Post by datakan »

dzbarsky wrote: post #12
Jackobrien wrote: post #3

Damn! If a solo engineer can do this, it makes the most around OAI/Anthropic start to look pretty weak.

This was nowhere near the top submission. But even if a solo engineer could get a top kernel, you don't think that having thousands of engineers, infinite tokens, and stronger models than are available to the public would give the labs a significant edge?

I don't know. That just sounds like throwing money at a problem until it goes away. I'm not convinced that is the correct path forward.

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#29

Post by qarl2 »

BlackRabbit1 wrote: post #22
qarl2 wrote: post #18

Earlier quoted context omitted.

I've had a lot of success decompiling old video game ROMs in exactly this way. Like you say - give it a way of verifying correctness - put it in a loop - and they are quite surprising. https://github.com/qarl/arcade-js

Same. I love reverse engineering embedded stuff. Even the cheap LLMs are great in doing the awful crud work in the beginning: finding offsets, firmware update file structures, brute forcing checksums, etc. It still produces a lot of crap in the later steps (understanding the implementation itself) but I'm happy doing this stuff myself then.

> It still produces a lot of crap in the later steps (understanding the implementation itself)

I've had success here by adding a phase called "grounding" that attempts to verify its "understanding" by creating tests that modify the running executable to ensure its made the right inference.

Is this variable really MARIO_X? Change it and see if Mario moves. Etc.

As an example in Donkey Kong - the system had trouble deciding if an array controlled barrels or fireballs. There was conflicting evidence.

After many trips through the loop - it realized it does BOTH, depending on which level you're on.

So the "understanding" grows with each iteration.

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#30

Post by tiahura »

Almondsetat wrote: post #2

In the last couple of days I wanted to try out the new definitive DeepSeek v4 releases. I gave it the repository of a semi-abandoned video compression codec and I told it to perform the usual benchmark -> profile -> verify -> research -> improve loop. I specifically chose this codec because the authors include a verifier for the bitstream to make sure you don't break stuff if you want to try your own implementation.…

I'm hoping to release a native Wine for MacOS w/ Win32 FEX support in the next few days.

Watching claude and codex play winquake and age of empires, and debug support for Firefox 52 has been wild.

Post reply on HN