Building an Advanced Agentic Harness

A model is non-deterministic. People prove things, LLM string together a bunch of words and do symbol shunting.

Ensure you understand what symbol shunting is before you make claims. https://ell.stackexchange.com/questions/76400/what-does-one-...

Real break throughs come from integral mathematics and not just a few reorderings. I've no doubt these are talented people recognizing output as useful; however, every time I see these links presented it's never from the "Prominent mathematician verifies AI proof"

Don't put the cart before the horse if you want people to think LLMs are cracking math problems in real terms.

But pulling orchestration off is very very tricky.

Even if it is just a small, simple orchestrator.

Ideas like planner, memory, log, subagents, graphs (each on their own) sound great and very promising.

So promising that one would think they must work, how could they not?

I've been there as well!

The challenge is that all these parts of the orchestrator are intertwined with each other

and they are all causing overhead in the main context window in some form or at least overall complexity that is difficult to grasp and predict/engineer for

(even though the idea is to help exactly with the fact that the context window is limited)

To save context window there is also more communication that has 'stille post' ('chinese whispers') like dynamics

Turns out it is very difficult to find out the right context to bubble up and down.

It's very similar to human org communication challenges (think large org stucture vs small teams vs one person that can keep it all in their head)

Yeah, what do you do if one person can't keep it all in their head?

But how great is it when it's possible?

Companies must have figured out how that works right? Maybe we can adopt and implement these ideas?

And yet … easy it is not, especially when you're not dealing with run-of-the-mill well-defined tasks.

But more like with open-ended software development?

I'm not saying it's not possible or that it should not be tried.

On the contrary, I think this is worth pursuing and a bit like the search for the holy grail.

But I also think the other direction of the search space is under-explored.

The holy grail is glamorous.

With 'smol' I'm spelunking on this other extreme (non-orchestration?)

(welcome, join us, we have cookies, and context windows with a lot of room for work items!)

smol is a minimalist agent harness that protects the context window

  - no system prompt
  - no tool spamming (just 1 tool: sh)
  - no agents.md
  - no mcp
  - no planning, todos, graphs, beads, …

it is a worthwhile thread to pull I think

at least from the dozens of benches I'm looking at I see that less stuff in the context window does help a lot

  - cheaper per task
  - finishing faster
  - better tool composition (sh and pipes are great!)

the context window is precious

https://x.com/__tosh/status/2084985580144722369

https://github.com/smol-env/smol

So maybe that's a big chunk of what you need for an 'AI Company': an agent that manages the goals and hierarchies. Although of course the DAG and agent hierarchy is not quite the same thing. But maybe the workflows and subworkflows are what matter.