41–50 of 110 posts
Does autoresearch work for projects that are not llm based? Eg in karpathy's example he is optimizing the nanogpt. What if I wanted to improve a Unet for image segmentation?
What about more distant software projects? Give it the CPython source code and say you want it to be faster.
There are better techniques for hyper-parameter optimisation, right? I fear I have missed something important, why has Autoresearch blown up so much? The bottleneck in AI/ML/DL is always data (volume & quality) or compute. Does/can Autoresearch help improve large-scale datasets? Is it more compute efficien than humans?
Yes, for example "swarm optimization".
The difference with "autoresearch" (restricting just to the HPO angle) is that the LLM may (at least we hope) beat conventional algorithmic optimization by making better guesses for each trial.
For example, perhaps the problem has an optimization manifold that has been studied in the past and the LLM either has that study in its training set or finds it from a search and learns the relative importance of all the HP axes. Given that, it "knows" not to vary the unimportant axes much and focus on varying the important ones. Someone else did the hard work to understand the problem in the past and the LLM exploits that (again, we may hope).
I often use LLMs to explore prior art and maybe find some alternative ways of thinking of problems. About 90% of what it tells me is useless or inapplicable to my domain due to a technicality it could not have known, but the other 10% is nice and has helped me learn some great new things. I can’t imagine letting an agent try everything that the LLM chatbot had recommended ($$$). Often coming up in recommendations are…
I think the main value lies in allowing the agent to try many things while you aren't working (when you are sleeping or doing other activities), so even if many tests are not useful, with many trials it can find something nice without any effort on your part. This is, of course, only applicable if doing a single test is relatively fast. In my work a single test can take half a day, so I'd rather not let an agent spen…
I often use LLMs to explore prior art and maybe find some alternative ways of thinking of problems. About 90% of what it tells me is useless or inapplicable to my domain due to a technicality it could not have known, but the other 10% is nice and has helped me learn some great new things. I can’t imagine letting an agent try everything that the LLM chatbot had recommended ($$$). Often coming up in recommendations are…
I find LLMs useful in regurgitating one-liners that I can’t be bothered to remember or things where even being flat out wrong is okay and you just do it yourself. For all the folks spending a lot of time and energy in setting up MCP servers, AGENTS.md, etc. I think this represents more that the LLM cannot do what it is being sold as by AI boosters and needs extreme amounts of guidance to reach a desired goal, if it e…
This was the case for me a year ago. Now Claude or Codex are routinely delivering finished & tested complete features in my projects. I move much, much faster than before and I don’t have an elaborate setup - just a single CLAUDE.md file with some basic information about the project and that’s it.
Take some working code. Ask an LLM to fix bugs. Measure performance and test coverage. Feed the results back into the LLM. Repeat. This has been the standard approach for more complex LLM deployments for a while now in our shop. Using different models across iterations is also something I've found useful in my own experiments. It's like getting a fresh pair of eyes.
Can we modify this approach to get LLMs that are good at specific programming languages or frameworks? That seems to be where local LLMs could really shine.
Alternatively, a modular model with multiple “experts” that I could mix and match for my specific stack
I don’t need the model to know all of the Internet plus 20 different human languages. I just want it to be really good with the stack of the project
I often use LLMs to explore prior art and maybe find some alternative ways of thinking of problems. About 90% of what it tells me is useless or inapplicable to my domain due to a technicality it could not have known, but the other 10% is nice and has helped me learn some great new things. I can’t imagine letting an agent try everything that the LLM chatbot had recommended ($$$). Often coming up in recommendations are…
I think the main value lies in allowing the agent to try many things while you aren't working (when you are sleeping or doing other activities), so even if many tests are not useful, with many trials it can find something nice without any effort on your part. This is, of course, only applicable if doing a single test is relatively fast. In my work a single test can take half a day, so I'd rather not let an agent spen…
Definitely not in the budget for non-VC-backed companies who aren’t in the AI bubble.
I often use LLMs to explore prior art and maybe find some alternative ways of thinking of problems. About 90% of what it tells me is useless or inapplicable to my domain due to a technicality it could not have known, but the other 10% is nice and has helped me learn some great new things. I can’t imagine letting an agent try everything that the LLM chatbot had recommended ($$$). Often coming up in recommendations are…
> agent try everything that the LLM chatbot had recommended ($$$) A lot depends on whether it is expensive to you. I use Claude Code for the smallest of whims and rarely run out of tokens on my Max plan.
Earlier quoted context omitted.
It's an LLM-powered evolutionary algorithm.
I'd like see a system like this take more inspiration from the ES literature, similar to AlphaEvolve. Let's see an archive of solutions, novelty scoring and some crossover rather than purely mutating the same file in a linear fashion.
There are lots of old ideas from evolutionary search worth revisiting given that LLMs can make smarter proposals.