Research-Driven Agents: When an agent reads before it codes
blog.skypilot.co
Research-Driven Agents: When an agent reads before it codes
1–10 of 81 posts
Re: Research-Driven Agents: When an agent reads before it codes
#2Coding agents that read papers before writing code find optimizations that code-only agents miss.
We added a literature review phase to Karpathy’s autoresearch loop and pointed it at llama.cpp. The agent autonomously read arxiv papers, studied competing forks and spun up VMs to run parallel experiments.
Re: Research-Driven Agents: When an agent reads before it codes
#3Sorry to spam, I'm working on this also from a different angle. Hopefully sharing adds to the conversation.
First, about the loop, Claude's (coding agent) context and attention is big enough to self-reflect. Agent Tuning shows a technique that not only demonstrates this but a way quantify it. [0] The difference is autoresearch's val_bpb measures what the agent built; Agent Tuning's p̂ measures the agent itself.
> Claude's attention doesn't distinguish between "instructions I'm writing" and "instructions I'm following" -- they're both just tokens in context.
Second, doing research, finding academic research to add to context helps. Here is an example of an implementation that creates trading strategies by reading research and recreating them in creative new ways. [1]
The biggest problem is the coding agents don't "Fail fast and loud". They fail deceivingly.
Re: Research-Driven Agents: When an agent reads before it codes
#4This is obvious, right? If you want to build a Facebook clone, you wouldn't tell the agent "build Facebook". You would provide it with a description of every page on Facebook, behaviors, interactions, UI, etc.
Re: Research-Driven Agents: When an agent reads before it codes
#5I use #PPPCDC for prompting: plan,plan,plan then verify with: Compare the plan to the existing Code. Reread and compare the plan to the Docs. Fix the areas you're not Confident about.
Re: Research-Driven Agents: When an agent reads before it codes
#6I think anyone who uses Claude knows that it works smarter when you have it make a plan first, and ask it to research the existing code as much as possible first ... so the results in this article doesn't surprise me at all.
However, I'd be curious to hear back from others who have tried adding the shell script (at the end of the article) to their flow: does it (really) improve Claude?
Re: Research-Driven Agents: When an agent reads before it codes
#7The skypilot devs need to focus on decoupling their offering, so that their very valuable "find the cheapest cloud" functionality isn't married to a glitchy reinvention of Kubernetes JobSet and MLflow
Re: Research-Driven Agents: When an agent reads before it codes
#8This is obvious, right? If you want to build a Facebook clone, you wouldn't tell the agent "build Facebook". You would provide it with a description of every page on Facebook, behaviors, interactions, UI, etc.
Have you even read the TL;DR in the linked article??
Re: Research-Driven Agents: When an agent reads before it codes
#9This is obvious, right? If you want to build a Facebook clone, you wouldn't tell the agent "build Facebook". You would provide it with a description of every page on Facebook, behaviors, interactions, UI, etc.
Have you even read the TL;DR in the linked article??
You mean this part?
> TL;DR: Coding agents generate better optimizations when they read papers and study competing projects before touching code
What made you think I hadn't read the article, let alone that TL;DR? I'm really curious. Jumping to an insulting "have you read the article" is a big step, so it'll be really interesting to see where your mind went.
Re: Research-Driven Agents: When an agent reads before it codes
#10[dead]