Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Auto-research with codex: How I achieved a 232x Faster Kernel

sankalp.bearblog.dev

Post reply on HN
Print view View original

11–20 of 104 posts

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#11

Post by eterm »

Almondsetat wrote: post #2

In the last couple of days I wanted to try out the new definitive DeepSeek v4 releases. I gave it the repository of a semi-abandoned video compression codec and I told it to perform the usual benchmark -> profile -> verify -> research -> improve loop. I specifically chose this codec because the authors include a verifier for the bitstream to make sure you don't break stuff if you want to try your own implementation.…

I did something similar recently with Google's C# protobuf library. I had spotted I was getting CPU bound rather than memory bandwidth bound when doing streaming of uint32 buffers in dotnet gRPC.

I then asked claude to compare the C#/.NET implementation in the library with the C++ version, and it quickly identified that the C# library was missing a couple of fairly cheap optimisations that were present in the C++ version.

If I can help get a PR merged, then it'll be by far the biggest impact of any work I've ever done.

I also compared the Rust version, it had this specific optimisation. The far more popular Tokio/Prost library did not.

Given appropriate guardrails, LLMs are impossibly fast at iterating to find root causes and specific performance bottlenecks.

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#12

Post by dzbarsky »

Jackobrien wrote: post #3

Damn! If a solo engineer can do this, it makes the most around OAI/Anthropic start to look pretty weak.

This was nowhere near the top submission. But even if a solo engineer could get a top kernel, you don't think that having thousands of engineers, infinite tokens, and stronger models than are available to the public would give the labs a significant edge?

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#13

Post by spwa4 »

People are always going to hate auto-research and "loop engineering". Because it's got 2 properties:

1) it's the only way to get something out of models (or people for that matter) that they don't know yet.

2) it's harder to do with an LLM than without. Not easier.

3) and when you fuck it up, half the time the LLM (or other ML technique) makes a fool out of you and you spent $1000 to find the quickest way to get a robot leg on the ground is just to crash it into the ground.

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#14

Post by Almondsetat »

eterm wrote: post #11
Almondsetat wrote: post #2

In the last couple of days I wanted to try out the new definitive DeepSeek v4 releases. I gave it the repository of a semi-abandoned video compression codec and I told it to perform the usual benchmark -> profile -> verify -> research -> improve loop. I specifically chose this codec because the authors include a verifier for the bitstream to make sure you don't break stuff if you want to try your own implementation.…

I did something similar recently with Google's C# protobuf library. I had spotted I was getting CPU bound rather than memory bandwidth bound when doing streaming of uint32 buffers in dotnet gRPC. I then asked claude to compare the C#/.NET implementation in the library with the C++ version, and it quickly identified that the C# library was missing a couple of fairly cheap optimisations that were present in the C++ ver…

You presented another thing LLMs excel at: integrating something from a project that is not present in another one. I think they work so well at this because both the starting and ending points have an already existing structure, so the LLM can guide itself effectively. In your case it's even more egregious because we are talking about the same exact algorithm/functionality implemented in two different, but rather similar, programming languages.

Could you have manually profiled and compared the execution paths? Sure. Could you have translated the C++ optimizations to C#? Sure. But in such an obvious case, the LLM managed itself.

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#16

Post by suddenlybananas »

spwa4 wrote: post #13

People are always going to hate auto-research and "loop engineering". Because it's got 2 properties: 1) it's the only way to get something out of models (or people for that matter) that they don't know yet. 2) it's harder to do with an LLM than without. Not easier. 3) and when you fuck it up, half the time the LLM (or other ML technique) makes a fool out of you and you spent $1000 to find the quickest way to get a ro…

What do you mean by 2?

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#17

Post by amarcheschi »

Isn't cholesky - used to substitute householder at a point - faster but less stable in some cases? I'm just recalling from memory since I had done a small project on qr decomposition with householder for an exam this year. I mean, if it is faster than the standard torch operation probably there are good reasons for which it is not the default standard torch operation. Might as well be wrong, I'm not sure

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#18

Post by qarl2 »

Almondsetat wrote: post #2

In the last couple of days I wanted to try out the new definitive DeepSeek v4 releases. I gave it the repository of a semi-abandoned video compression codec and I told it to perform the usual benchmark -> profile -> verify -> research -> improve loop. I specifically chose this codec because the authors include a verifier for the bitstream to make sure you don't break stuff if you want to try your own implementation.…

I've had a lot of success decompiling old video game ROMs in exactly this way. Like you say - give it a way of verifying correctness - put it in a loop - and they are quite surprising.

https://github.com/qarl/arcade-js

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#19

Post by jurgenburgen »

suddenlybananas wrote: post #16
spwa4 wrote: post #13

People are always going to hate auto-research and "loop engineering". Because it's got 2 properties: 1) it's the only way to get something out of models (or people for that matter) that they don't know yet. 2) it's harder to do with an LLM than without. Not easier. 3) and when you fuck it up, half the time the LLM (or other ML technique) makes a fool out of you and you spent $1000 to find the quickest way to get a ro…

What do you mean by 2?

LLMs will take shortcuts and cheat in ways that a human would realize are counterproductive.

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#20

Post by porridgeraisin »

Almondsetat wrote: post #2

In the last couple of days I wanted to try out the new definitive DeepSeek v4 releases. I gave it the repository of a semi-abandoned video compression codec and I told it to perform the usual benchmark -> profile -> verify -> research -> improve loop. I specifically chose this codec because the authors include a verifier for the bitstream to make sure you don't break stuff if you want to try your own implementation.…

This is the way. Checkout the technique mentioned in the alphaevolve paper and see if it works well for your setting.
Post reply on HN