61–70 of 110 posts
> Then I lock down Claude Code’s permissions to only edit these two files and run run.sh. No direct Python execution, no pip installs, no network access, no git push, etc. How does one run Claude Code without network access?
in both cases you'd limit it so CC can only talk to the required Anthropic APIs.
So not zero access, but as close to it as you can get.
> Then I lock down Claude Code’s permissions to only edit these two files and run run.sh. No direct Python execution, no pip installs, no network access, no git push, etc. How does one run Claude Code without network access?
There are better techniques for hyper-parameter optimisation, right? I fear I have missed something important, why has Autoresearch blown up so much? The bottleneck in AI/ML/DL is always data (volume & quality) or compute. Does/can Autoresearch help improve large-scale datasets? Is it more compute efficien than humans?
Not true at all. The whole point of ML is to find better mappings from X to Y, even for the same X.
Many benchmarks can’t be solved by just throwing more compute at the problem. They need to learn better functions which traditionally requires humans.
And sometimes an algorithm lets you tap into more data. For example transformers had better parallelism than LSTMs -> better compute efficiency.
Ok, so looking at the commit log[1], I was mostly interested in seeing what the "moonshot ideas" implementations looked like, but basically everything is just hyperparameter tuning. Which is nice, but likely not worth the $$$ spent on the tokens. Am I missing something here? [1] https://github.com/ykumards/eCLIP/commits/main/autoresearch
Optuna or skopt are open source and won't take any GPU time at all to do it.
For a fixed search space it will almost certainly be better though.
This feels less like automated research and more like structured trial and error with a decent feedback loop. Still useful, but I think the real bottleneck is how good your eval metric is. If that’s weak, the whole loop just optimizes for the wrong thing faster.
thanks for sharing your results and the road to them!
pretty cool experiment, i thought about someone maybe doing this and am happy you did it in this way. nice writeup too. this made me giggle a bit: "At one point it got tired of waiting for training to finish and just ended the conversation. I wouldn’t give it full autonomy just yet :)" thanks for sharing your results and the road to them!
Earlier quoted context omitted.
> agent try everything that the LLM chatbot had recommended ($$$) A lot depends on whether it is expensive to you. I use Claude Code for the smallest of whims and rarely run out of tokens on my Max plan.
Our experiments aren’t free. We use cloud infrastructure. An experiment costs on the order of tens of dollars, so massively parallelizing “spaghetti at wall” simulators is costly before we even talk about LLMs.
> The original paper used several medical X-ray datasets which I don’t have access to anymore, so I needed a new dataset with spatial annotations to test the expert attention mechanism. I picked the Ukiyo-eVG dataset: ~11K Japanese woodblock prints That's such a weird switch. There's lots of free medical imaging online. Example: https://www.cancerimagingarchive.net/