Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google open-sources experimental agent orchestration testbed Scion

infoq.com

Post reply on HN
Print view View original

61–70 of 78 posts

Re: Google open-sources experimental agent orchestration testbed Scion

#61

Post by jaggederest »

bormaj wrote: post #58
sowbug wrote: post #19

I'm looking forward to trying this. I've had a positive but high-variance experience with Gastown[1], which is in the same genre. I hope that Scion does better. My main complaints with Gastown are that (1) it's expensive, partly because (2) it refuses to use anything but Claude models, in spite of my configuration attempts, (3) I can't figure out how to back up or add a remote to its beads/dolt bug database, which ma…

As someone who hasn't yet jumped into working with multiple agents simultaneously, where does a tool like gastown help you the most?

I'm trialing it on very silly things, like a economic simulator game in Rust/Bevy. I put in an entire road map document with inline specs and goals, wild milestones, with tasks like "working bid/ask spread when factories buy or sell on the market to make pricing dynamic and realistic", "political entities can set work conditions", "international trade has pricing dynamics that take into account currency interchange and tariff rates", "infrastructure for trade improves as trade volumes increase across given tiles".

Out the other end over about 3-4 five-hour-sessions comes about 85% functional code for every single listed thing. I'd guess you'd be looking at a team for months, give or take, without the automation. Total cost was around $50 in VM time (not counting claude since I would be subscribed anyway) I'm not letting that thing anywhere near a computer I care about and rust compiles are resource intensive, so I paid for a nice VM that I could smash the abort button on if it started looking at me funny.

So I liken it to buying an enormous bulldozer. If you're a skilled operator you can move mountains, but there'll still be a lot of manual work and planning involved. It's very clearly directionally where the industry will go once the models are improved and the harnesses and orchestration are more mature than "30% of the development effort is fixing the harness and orchestration itself", plus an additional "20% of your personal time will be knocking two robots heads together and getting them to actually do work"

Edit: some more details of other knock on work - I asked for a complexity metadata field to automatically dispatch work to cheaper/faster models, set up harnesses to make opencode and codex work similarly to how claude works, troubleshot some bugs in the underlying gastown system. Gastown fork is public if you'd like to have a look.

Re: Google open-sources experimental agent orchestration testbed Scion

#62

Post by visarga »

sowbug wrote: post #19

I'm looking forward to trying this. I've had a positive but high-variance experience with Gastown[1], which is in the same genre. I hope that Scion does better. My main complaints with Gastown are that (1) it's expensive, partly because (2) it refuses to use anything but Claude models, in spite of my configuration attempts, (3) I can't figure out how to back up or add a remote to its beads/dolt bug database, which ma…

I made one similar harness, mine does lightweight sandboxing with Seatbelt on Mac and Bubblewrap on Linux. I initially used Docker too, but abandoned it. I like how these 2 sandboxes allow me to make all the file system r/o except the project folder which is r/w (and a few other config folders). This means my code runs inside the sandbox like outside, same paths hold, same file system. The .git folder is also r/o inside sandbox, only outside agent can commit. Sandboxing was intended to enable --yolo mode, I wanted to maximize autonomous time.

Work is divided into individual tasks. I could have used Plan Mode or TodoWriter tool to implement tasks - all agents have them nowadays. But instead I chose to plan in task.md files because they can be edited iteratively, start as a user request, develop into a plan with checkbox-able steps, the plan is reviewed by judge agent (in yolo mode, and fresh context), then worker agent solves gates. The gates enforce a workflow of testing soon, testing extensively. There is another implementation judge again in yolo mode. And at the end we update the memory/bootstrap document.

Task files go into the git repo. I also log all user messages and implement intent validation with the judge agents. The judges validate intent along the chain "chat -> task -> plan -> code -> tests". Nothing is lost, the project remembers and understands its history. In fact I like to run retrospective tasks where a task.md 'eats' previous tasks and produces a general project perspective not visible locally.

In my system everything is a md file, logged and versioned on git. You have no issue extracting your memories, in fact I made reflection on past work a primitive operation of this harness. I am using it for coding primarily, but it is just as good for deep research, literature reviews, organizing subject matter and tutoring me on topics, investment planning and orchestrating agent experiment loops like autoresearch. That is because the task.md is just a generic programming pipeline, gates are instructions in natural language, you can use it for any cognitive work. Longest task.md I ran was 700 steps, took hours to complete, but worked reliably.

https://github.com/horiacristescu/claude-playbook-plugin

Re: Google open-sources experimental agent orchestration testbed Scion

#63

Post by jhavera »

The "isolation over constraints" framing is interesting. Scion enforces safety at the infrastructure layer, letting agents operate freely inside containers while controlling what they can reach on the outside. That is a runtime approach.

We have been exploring a different layer for the same problem. ARIA (aria-ir.org) is an intermediate representation designed for AI-authored code. Instead of constraining the agent at runtime, it constrains what the agent produces at the representation level. Functions must declare effects, intent annotations are mandatory and verifiable, and the compiler enforces memory safety at compile time before anything executes.

The two approaches are not mutually exclusive. Scion handles what the agent can reach. ARIA handles what the agent generates. A system that uses both would have safety at the output layer and safety at the execution layer. Curious whether the Scion team has thought about what properties the code an agent produces should have, independent of how that agent is isolated.

Re: Google open-sources experimental agent orchestration testbed Scion

#64

Post by zavec »

This looks really promising, I am curious about the choice to use containers as the isolation layer though. If the goal is to treat agents as untrusted and isolate them fully I feel like microVMs would be a better option.

If it supports OCI runtimes though then maybe kata containers can be plugged in, I'll have to dig in after work and see.

Re: Google open-sources experimental agent orchestration testbed Scion

#66

Post by harlequinetcie »

Oh Google, I love you guys, but it seems they alway launch these half baked things without the support they deserve behind it.

ADK was (and is) exceptional, but nobody is actually making noise and pushing for it as they should. It feels like Microsoft .net back in the day.

Let's see how it goes. I'm rooting for y'all

Re: Google open-sources experimental agent orchestration testbed Scion

#67

Post by harlequinetcie »

sowbug wrote: post #19

I'm looking forward to trying this. I've had a positive but high-variance experience with Gastown[1], which is in the same genre. I hope that Scion does better. My main complaints with Gastown are that (1) it's expensive, partly because (2) it refuses to use anything but Claude models, in spite of my configuration attempts, (3) I can't figure out how to back up or add a remote to its beads/dolt bug database, which ma…

We ended up adding workflows with deterministic paths, that can use RAW API calls, CLIs, and agents. I think that was a big differential.

We also added pi-mono, and started using more and more other models for different tasks (Gemini, K2.5, GLM-5, you name it).

I think the problem is that most are building solutions that rely in one provider, instead of focusing self learning capabilities on improving the cost-quality-speed ratio.

For reference: https://github.com/desplega-ai/agent-swarm

Re: Google open-sources experimental agent orchestration testbed Scion

#68

Post by realharo »

jaggederest wrote: post #61
bormaj wrote: post #58

Earlier quoted context omitted.

As someone who hasn't yet jumped into working with multiple agents simultaneously, where does a tool like gastown help you the most?

I'm trialing it on very silly things, like a economic simulator game in Rust/Bevy. I put in an entire road map document with inline specs and goals, wild milestones, with tasks like "working bid/ask spread when factories buy or sell on the market to make pricing dynamic and realistic", "political entities can set work conditions", "international trade has pricing dynamics that take into account currency interchange a…

>working bid/ask spread when factories buy or sell on the market to make pricing dynamic and realistic

Does it deliver on the "realistic" part? My experience with most models is they make something that technically fulfills the ask, but often in a way that doesn't really capture my intent (this is with regular Claude Code though).

Post reply on HN