Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Auto-research with codex: How I achieved a 232x Faster Kernel

sankalp.bearblog.dev

Post reply on HN
Print view View original

81–90 of 104 posts

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#81

Post by dejavucoder »

fooblaster wrote: post #79
dejavucoder wrote: post #54

Earlier quoted context omitted.

1. labs have lots of inference capacity 2. they will have domain experts working on this so their efficiency is gonna be exponentially more (can direct LLM better, save money, reach same results faster)

You can't exceed roofline performance on hardware. There is an performance cap you can hit. This recursive self improvement stuff lets you be closer to the pareto frontier, but the idea that it is leading to some exponential growth is a total pipe dream.

fair enough

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#82

Post by bonoboTP »

augment_me wrote: post #57

One thing worth to note in the competition is that 8 out of the 10 top solutions, which all happened to be optimized this way completely broke at any other input than the competition ones. The only solutions that did not break when tested with OOD shapes were made by experts who know a lot about GPU programming and that did not create 25k lines of CUDA but followed and adjusted their solution in reasonable bounds. Th…

Test coverage is important. You need to test a variety of inputs.

Also "will always" is way overconfident. A year ago nothing close to this existed at all. Next year it will fill yet a different role.

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#83

Post by KuSpa »

halJordan wrote: post #67
sqquima wrote: post #38

Meta commentary but it felt fresh to read a long wall of text that didn't seem to be AI generated. Thanks.

How is this meta commentary when this or its negative version is present on literally every post.

First: Whataboutism, and second: Those are meta comments as well, they just don't declare themselves as one.

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#84

Post by vickychijwani »

phoghed wrote: post #51
iinnPP wrote: post #35

Earlier quoted context omitted.

It's really difficult to understand what your definition of surprisingly bad is. What was it continuously having problems with?

Just reporting work as done that’s not even close mainly. Then if you manually screenshot both and ask if they are the same it’s like “yeah looks great boss”. The visual understanding of the models is just leagues worse than their text/language understanding. Or guessing colors rather than sampling from the image or pulling from figma is another stupid thing they do constantly.

I imagine Gemini would do better on these types of tasks. Curious to try it out.

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#85

Post by alfiedotwtf »

Almondsetat wrote: post #2

In the last couple of days I wanted to try out the new definitive DeepSeek v4 releases. I gave it the repository of a semi-abandoned video compression codec and I told it to perform the usual benchmark -> profile -> verify -> research -> improve loop. I specifically chose this codec because the authors include a verifier for the bitstream to make sure you don't break stuff if you want to try your own implementation.…

> Personally, I believe that LLMs should be treated like an advanced version of Prolog or linear programming: you give the constraints, you have a way of verifying correctness, and you give it a clear goal. If the LLM can verify itself and course-correct you can basically leave it on autopilot

This!

People keep measuring how good AI is by one-shotting a problem, but I e found that a back and fourth via the Superpowers SKILL.md is how the model builds these constraints… then when it has all the holes, it gets to work

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#86

Post by themeiguoren »

I've had pretty good luck with the following process for performance optimization loops:

- Have an agent generate unit tests until it gets to 100% path (not just statement) coverage, with every numerical test asserting checks against golden values to prevent regressions

- Let it rip on a performance improvement loop, for the widest E2E representative test case you have. Have it generate flamegraphs along the way so you can check in and steer it as necessary.

- Optionally allow for 1 ULP changes in output values so that it doesn't kill itself getting bit-exact results.

- Have it flag correctness errors as it goes, since your code probably isn't bug free.

This is also how I've done language ports from python to rust, and having the ironclad test coverage protects you from drifting.

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#87

Post by mycall »

augment_me wrote: post #57

One thing worth to note in the competition is that 8 out of the 10 top solutions, which all happened to be optimized this way completely broke at any other input than the competition ones. The only solutions that did not break when tested with OOD shapes were made by experts who know a lot about GPU programming and that did not create 25k lines of CUDA but followed and adjusted their solution in reasonable bounds. Th…

Isn't generalization of solutions what JEPA is trying to solve?

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#88

Post by michaelmrose »

Almondsetat wrote: post #2

In the last couple of days I wanted to try out the new definitive DeepSeek v4 releases. I gave it the repository of a semi-abandoned video compression codec and I told it to perform the usual benchmark -> profile -> verify -> research -> improve loop. I specifically chose this codec because the authors include a verifier for the bitstream to make sure you don't break stuff if you want to try your own implementation.…

[dead]

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#89

Post by nullbio »

Jackobrien wrote: post #3

Damn! If a solo engineer can do this, it makes the most around OAI/Anthropic start to look pretty weak.

Yeah, but here's a dirty little secret that very few people are discussing:

You can't use Claude for this sort of thing if the goal is to make better AI systems. Anthropic finetunes Claude to dissuade people and the agent from using research that actually works. Anything that they use internally in their own models is poisoned, to protect their moat.

By proxy, that also means any openweights model that was distilled from Claude is equally useless for this purpose.

Thankfully, I don't believe OpenAI does this - they are far more honest and seem to care about their reputation. Anthropic is evil though.

Re: Auto-research with codex: How I achieved a 232x Faster Kernel

#90

Post by vatsachak »

augment_me wrote: post #57

One thing worth to note in the competition is that 8 out of the 10 top solutions, which all happened to be optimized this way completely broke at any other input than the competition ones. The only solutions that did not break when tested with OOD shapes were made by experts who know a lot about GPU programming and that did not create 25k lines of CUDA but followed and adjusted their solution in reasonable bounds. Th…

There's only three kinds of optimization realizations

1) Realize that your problem isn't as general as it was

2) Realize that your problem has a better memory model

3) Realize that your problem can be parallelized further

Hyperoptimizing usually falsely leans on 1

Post reply on HN