Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Research-Driven Agents: When an agent reads before it codes

blog.skypilot.co

Post reply on HN
Print view View original

31–40 of 81 posts

Re: Research-Driven Agents: When an agent reads before it codes

#31

Post by ctoth »

alex000kim wrote: post #26
ctoth wrote: post #22

I've been very interested in this recently. I'm pretty sure that every project should have a ./papers directory of annotated papers in it like I do in Qlatt[0]. Literally every project. If it's something that's been done a million times then that means it has good literature on it? If not, then even more important to find related stuff! And not just crunchy CS stuff like databases or compilers or whatever. Are you cr…

That directory is huge already! I guess the index.md helps the agent find what it needs, but even the markdown file is very long - this would consume a ton of tokens. Also I wonder who/what decides what papers go in there. In the blog post, the agent is allowed to do its own search.

Check out the Researcher and Process Leads skill in ctoth/research-papers-plugin. I have basically completely automated the literature review.

Re: Research-Driven Agents: When an agent reads before it codes

#32

Post by mkagenius »

dataviz1000 wrote: post #3

Sorry to spam, I'm working on this also from a different angle. Hopefully sharing adds to the conversation. First, about the loop, Claude's (coding agent) context and attention is big enough to self-reflect. Agent Tuning shows a technique that not only demonstrates this but a way quantify it. [0] The difference is autoresearch's val_bpb measures what the agent built; Agent Tuning's p̂ measures the agent itself. > Cla…

> The biggest problem is the coding agents don't "Fail fast and loud". They fail deceivingly.

GPT 2 and 3 used to fail fast (and loud coz we could easily see it lying)

Re: Research-Driven Agents: When an agent reads before it codes

#33

Post by satvikpendem »

simlevesque wrote: post #11

I've been making skills from arxiv papers for a while. I have a one for multi-object tracking for example. It has a SKILL.md describing all important papers (over 30) on the subject and a folder with each paper's full content as reStructuredText. To feed Arxiv papers to LLMs I found that RST gives the best token count/fidelity ratio. Markdown lacks precision. LateX is too verbose. I have a script with the paper's url…

Does that even fit in the context? It seems like 30 papers worth of content would just overflow it.

Re: Research-Driven Agents: When an agent reads before it codes

#35

Post by ctoth »

satvikpendem wrote: post #33
simlevesque wrote: post #11

I've been making skills from arxiv papers for a while. I have a one for multi-object tracking for example. It has a SKILL.md describing all important papers (over 30) on the subject and a folder with each paper's full content as reStructuredText. To feed Arxiv papers to LLMs I found that RST gives the best token count/fidelity ratio. Markdown lacks precision. LateX is too verbose. I have a script with the paper's url…

Does that even fit in the context? It seems like 30 papers worth of content would just overflow it.

For each paper, have your agent extract a three sentence description, create a description.md, then concat those with the paper names into an INDEX.md which it should consult to find appropriate papers. Also: have your agent tag papers, then autogenerate your tagged collection on the filesystem. Then you get nice things like https://github.com/ctoth/Qlatt/tree/master/papers/tagged

Then something in your {CLAUDE,AGENTS}.md that says: when working on something with relevant context supplied by papers, read the papers before doing the work. You can find all papers plus their descriptions in ./papers/INDEX.md and papers by tag in ./papers/tagged

Re: Research-Driven Agents: When an agent reads before it codes

#36

Post by pstuart »

alex000kim wrote: post #26
ctoth wrote: post #22

I've been very interested in this recently. I'm pretty sure that every project should have a ./papers directory of annotated papers in it like I do in Qlatt[0]. Literally every project. If it's something that's been done a million times then that means it has good literature on it? If not, then even more important to find related stuff! And not just crunchy CS stuff like databases or compilers or whatever. Are you cr…

That directory is huge already! I guess the index.md helps the agent find what it needs, but even the markdown file is very long - this would consume a ton of tokens. Also I wonder who/what decides what papers go in there. In the blog post, the agent is allowed to do its own search.

Having a "indexed global data collection" of the markdown would be a kumbaya moment for AI. There's so much data out there but finite disk space. Maybe torrents or IPFS could work for this?

Re: Research-Driven Agents: When an agent reads before it codes

#37

Post by ctoth »

pstuart wrote: post #36
alex000kim wrote: post #26

Earlier quoted context omitted.

That directory is huge already! I guess the index.md helps the agent find what it needs, but even the markdown file is very long - this would consume a ton of tokens. Also I wonder who/what decides what papers go in there. In the blog post, the agent is allowed to do its own search.

Having a "indexed global data collection" of the markdown would be a kumbaya moment for AI. There's so much data out there but finite disk space. Maybe torrents or IPFS could work for this?

I'm actually sort of working on this! https://github.com/ctoth/propstore -- it's like Cyc, but there is no one answer. Plus knowledge bases are literally git repos that you can fork/merge. Research-papers-plugin is the frontend, we extract the knowledge, then we need somewhere to put it :)

Re: Research-Driven Agents: When an agent reads before it codes

#38

Post by dataviz1000 »

mkagenius wrote: post #32
dataviz1000 wrote: post #3

Sorry to spam, I'm working on this also from a different angle. Hopefully sharing adds to the conversation. First, about the loop, Claude's (coding agent) context and attention is big enough to self-reflect. Agent Tuning shows a technique that not only demonstrates this but a way quantify it. [0] The difference is autoresearch's val_bpb measures what the agent built; Agent Tuning's p̂ measures the agent itself. > Cla…

> The biggest problem is the coding agents don't "Fail fast and loud". They fail deceivingly. GPT 2 and 3 used to fail fast (and loud coz we could easily see it lying)

My next exploration will be "Coding Agents: fail slow, silent, and deceivingly".

After one month working on using Claude to create trading strategies, the one thing I learned; if the strategy looks like it can profit, it is a lie. The trading strategy agent doesn't find trading strategies that work, it is really a bug hunting agent.

Post reply on HN