I feel bad for the engineers at Cursor who have to use Grok in these sorts of experiments.
Agent swarms and the new model economics
41–50 of 159 posts
Re: Agent swarms and the new model economics
#42I would have loved to see more of the harness engineering shared as code. Instead we are left with only the outcome. I guess that makes sense since the harness is the product in the case of Cursor.
I don't think there is a lot test harness, given there is a golden implementation (sqlite3), with clear interface (SQL, and sqlite3 db files). Once you slop crappy implementation, then you can create one group of agents finding gap where slop doing wrong vs golden, and another group of agents filling the gap meeting the conformance.
I assume they did similar, as long as test case(or spec) can be parallelized. Having parallel execution / testing environment is crucial if we need 1000 commits/second throughput.
I would not surprised they have 1+ millions of specs.
I would bet they won't release generated specs - actually this will be a valuable asset like https://sqlite.org/th3.html
Re: Agent swarms and the new model economics
#43Earlier quoted context omitted.
Wait, why isn't context cached with /loop?
Because it's typically cached for 5min (1hr is a setting you have to explicitly opt into), and very few people run loops at a cadence of < 5 mins.
Oh! I thought Anthropic cached for one hour (by default), am I wrong about that? Or is this an OpenAI thing?
Re: Agent swarms and the new model economics
#44How do we know if these models weren’t trained on Turso’s rewrite of SQLite in Rust? It seems both likely that they were and impossible to remove that code from pretraining. Doesn’t that make this just about LLM memorization of the training set? What am I missing?
> What am I missing?
They’re testing the same models on the same task but different ways of organising the swarms and the new approach works better.
Re: Agent swarms and the new model economics
#45Earlier quoted context omitted.
not in Rust. might be different enough ...
Even if no Rust code for it was seen during training, an LLM can trivially transpile SQLite's C codebase to Rust on the fly. For example, I just asked ChatGPT to write John Carmack's famous Fast Inverse Square Root algorithm in Erlang, without searching online or thinking, and it transpiled it immediately (while also extracting the knowledge in the same step). SQLite's semantics/code are stored in the middle layers o…
Then you would expect the implementation to be structured the same as SQLite, and having glanced at the result, it looks like at least some things aren't. For example, it seems to use an operator-tree executor rather than SQLite's bytecode interpreter.
Re: Agent swarms and the new model economics
#46Earlier quoted context omitted.
It's hard not read such quotes and immediately think of the Infinite Monkey Theorem. https://en.wikipedia.org/wiki/Infinite_monkey_theorem
If at the end Shakespeare-level literature is produced, does it matter whether we arrived there by random keystrokes?
The problem is the infinite other literature produced along the way. It doesn't work to throw more monkeys at the problem of verification, and checking for matches against existing Shakespeare plays is cheating.
Re: Agent swarms and the new model economics
#47Earlier quoted context omitted.
Because it's typically cached for 5min (1hr is a setting you have to explicitly opt into), and very few people run loops at a cadence of < 5 mins.
Oh! I thought Anthropic cached for one hour (by default), am I wrong about that? Or is this an OpenAI thing?
Anthropic changed their cache duration for some reason a while back.
Re: Agent swarms and the new model economics
#48Can a swarm of agents write the 835 page manual?
Re: Agent swarms and the new model economics
#49Earlier quoted context omitted.
It's hard not read such quotes and immediately think of the Infinite Monkey Theorem. https://en.wikipedia.org/wiki/Infinite_monkey_theorem
If at the end Shakespeare-level literature is produced, does it matter whether we arrived there by random keystrokes?
Yes because it is akin to producing a universe of slop and trying to find the single atom of gold.
Re: Agent swarms and the new model economics
#50>The browser swarm from earlier this year peaked at roughly 1,000 commits per hour on Git. The new system peaks at around 1,000 commits per second. >To facilitate this rate of activity, we built a new version control system (VCS) from scratch. Throughput was not the only reason to own this layer. Every change in the system passes through the VCS, so it is where collisions first become visible, and several of the coor…
I mean why even use VCS at that point?