Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Auto-research with codex: How I achieved a 232x Faster Kernel

sankalp.bearblog.dev

Post reply on HN
Print view View original

51–60 of 104 posts

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#51

Post by phoghed »

iinnPP wrote: post #35
phoghed wrote: post #21

Earlier quoted context omitted.

I’ve tried it on simple UI tasks. Give it screenshot to work towards (or figma MCP), let it get screenshots from chrome to check its work. The existing models are surprisingly bad at it.

It's really difficult to understand what your definition of surprisingly bad is. What was it continuously having problems with?

Just reporting work as done that’s not even close mainly. Then if you manually screenshot both and ask if they are the same it’s like “yeah looks great boss”. The visual understanding of the models is just leagues worse than their text/language understanding.

Or guessing colors rather than sampling from the image or pulling from figma is another stupid thing they do constantly.

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#52

Post by codetiger »

Almondsetat wrote: post #2

In the last couple of days I wanted to try out the new definitive DeepSeek v4 releases. I gave it the repository of a semi-abandoned video compression codec and I told it to perform the usual benchmark -> profile -> verify -> research -> improve loop. I specifically chose this codec because the authors include a verifier for the bitstream to make sure you don't break stuff if you want to try your own implementation.…

Had a similar experience with my Rust implementation for JSONLogic expression evaluation engine. As it has a full test suite with 1000s of cases and a benchmarking script, I was able to give some basic hints to try different optimization techniques and the end result was impressive. Reached from 1.6s to 200ms for a full benchmarking test. https://github.com/GoPlasmatic/datalogic-rs First 3 versions were hand written and maintained for 3yrs, and now 4th version came out in less than a month's time with impressive performance.

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#53

Post by dejavucoder »

dzbarsky wrote: post #12
Jackobrien wrote: post #3

Damn! If a solo engineer can do this, it makes the most around OAI/Anthropic start to look pretty weak.

This was nowhere near the top submission. But even if a solo engineer could get a top kernel, you don't think that having thousands of engineers, infinite tokens, and stronger models than are available to the public would give the labs a significant edge?

hello author here.

yes, it gives labs edge and leads to self-recursive improvement loops.

also i was myself able to finish 7th in a later competition with 2-3 other approaches which are variants of the method discussed in this blog.

in general, having a harness as thin as possible with some problem specific instructions while controlling for context rot is the key.

point i am trying to make is there are a lot of optimisation surface areas possible.

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#54

Post by dejavucoder »

datakan wrote: post #28
dzbarsky wrote: post #12

Earlier quoted context omitted.

This was nowhere near the top submission. But even if a solo engineer could get a top kernel, you don't think that having thousands of engineers, infinite tokens, and stronger models than are available to the public would give the labs a significant edge?

I don't know. That just sounds like throwing money at a problem until it goes away. I'm not convinced that is the correct path forward.

1. labs have lots of inference capacity 2. they will have domain experts working on this so their efficiency is gonna be exponentially more (can direct LLM better, save money, reach same results faster)

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#57

Post by augment_me »

One thing worth to note in the competition is that 8 out of the 10 top solutions, which all happened to be optimized this way completely broke at any other input than the competition ones.

The only solutions that did not break when tested with OOD shapes were made by experts who know a lot about GPU programming and that did not create 25k lines of CUDA but followed and adjusted their solution in reasonable bounds.

The takeaway from this is that these approaches will always solve for specificity, but it's a much harder task to steer the model into making general solutions. So if you're an inference provider for some specific model shape, fantastic, go for it. If you are a maintainer of a open-source library, this is not useful.

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#59

Post by msdz »

dsign wrote: post #23
tosh wrote: post #4

Training material seems to be especially rich re GPU kernels and SIMD. I wonder if there is extra effort put into this because they are useful for the researchers working on the models or just a sub-domain that language models are a great fit for and humans have trouble with?

Anecdotally, I saw Opus 5 come up with a complicated loop unrolling technique when I asked it to implement a simple biquad in SIMD, missing a simpler solution. Maybe it was a downgraded session, who knows. That SIMD instruction set, the one for the ESP32-P4, is not very popular and all the documentation it has is a couple of blog posts. So I'm pretty sure it has at most seen some code for a predecessor during its tra…

> Maybe it was a downgraded session, who knows.

Off-topic, but imagine us collectively being okay with (or powerless to do anything about) this sentiment about any other software service provided like two years ago.

How the times have changed…

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#60

Post by kenerwin88 »

qarl2 wrote: post #18
Almondsetat wrote: post #2

In the last couple of days I wanted to try out the new definitive DeepSeek v4 releases. I gave it the repository of a semi-abandoned video compression codec and I told it to perform the usual benchmark -> profile -> verify -> research -> improve loop. I specifically chose this codec because the authors include a verifier for the bitstream to make sure you don't break stuff if you want to try your own implementation.…

I've had a lot of success decompiling old video game ROMs in exactly this way. Like you say - give it a way of verifying correctness - put it in a loop - and they are quite surprising. https://github.com/qarl/arcade-js

Oh wow, this is almost exactly what I’ve been doing with Zelda LTTP. I have it in rust now, but just finished the “first pass” you reference. Mine is still not really readable, second step is the modernizing the actual code. I’ve really struggled with needing to handhold it though, I’ll see if I can plagiarize from you!
Post reply on HN