Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Freestyle – Sandboxes for Coding Agents

freestyle.sh

Post reply on HN
Print view View original

121–130 of 182 posts

Re: Launch HN: Freestyle – Sandboxes for Coding Agents

#121

Post by tomComb »

benswerd wrote: post #116
tomComb wrote: post #77

Earlier quoted context omitted.

But your pricing page suggests that that is not available without a subscription: in the on-demand pricing section "persistent Snapshots" and "Persistent VM's" have an 'x'.

We do not allow long term persistence for the free tier. This is purely a defense mechanism, I don't want to guarantee storing the data of an entire VM forever for non paying users. We have persistence options for them like Sticky persistence but it doesn't come with the reliability of long term persistence storage.

But it wouldn’t be non paying customers. That was from the on demand section. I just want to pay for what I use without getting into a subscription.

Re: Launch HN: Freestyle – Sandboxes for Coding Agents

#122

Post by benswerd »

tomComb wrote: post #121
benswerd wrote: post #116

Earlier quoted context omitted.

We do not allow long term persistence for the free tier. This is purely a defense mechanism, I don't want to guarantee storing the data of an entire VM forever for non paying users. We have persistence options for them like Sticky persistence but it doesn't come with the reliability of long term persistence storage.

But it wouldn’t be non paying customers. That was from the on demand section. I just want to pay for what I use without getting into a subscription.

Ah I see. This is very interesting but not what we're focused on right now. I will keep this in mind for future prioritization.

Re: Launch HN: Freestyle – Sandboxes for Coding Agents

#123

Post by lll-o-lll »

benswerd wrote: post #50
croes wrote: post #44

Earlier quoted context omitted.

The problem is the agent, which should be treated untrusted. The computer isn’t the problem

Kind of. The chat logs of the agent are trustworthly, as should any telemetry you have on it or coming out of the VM. Its behavior should be treated as probabilistic and therefore untrustworthly.

It’s untrustworthy because its context can be poisoned and then the agent is capable of harm to the extent of whatever the “computer” you give it is capable of.

The mitigation is to keep what it can do to “just the things I want it to do” (e.g. branch protection and the like, whitelisted domains/paths). And to keep all the credentials off its box and inject them inline as needed via a proxy/gateway.

I mean, that’s already something you can do for humans also.

Re: Launch HN: Freestyle – Sandboxes for Coding Agents

#124

Post by ghm2199 »

benswerd wrote: post #96
chrisweekly wrote: post #93

Earlier quoted context omitted.

Have you considered git worktree?

Great for simple things, but git worktrees don't work when you have to fork processes like postgres/complex apps.

For postgres there are pg containers, we use them in pytest fixtures for 1000's of unit-tests running concurrently. I imagine you could run them for integration test purposes too. What kind of testing would you run with these that can't be run with pg containers or not covered by conventional testing?

I'll say this is still quite useful win for browser control usecases and also for debugging their crashes.

Re: Launch HN: Freestyle – Sandboxes for Coding Agents

#125

Post by Bnjoroge »

Looks cool - would be great to see a PR with some benchmarks on this repo if you can: https://github.com/computesdk/benchmarks

edit: just saw the pr for freestyle. something seems to be blocking, but curious how it compares: https://github.com/computesdk/benchmarks/pull/41

Re: Launch HN: Freestyle – Sandboxes for Coding Agents

#126

Post by ghm2199 »

benswerd wrote: post #86
ianberdin wrote: post #85

Congrats guys! Would share some technical details, I bet you have great stories to tell. Let’s, what is forking? You completely copy disk, make ram snapshot and run it? If CoW, but ram? You mentioned 8GB ram vms. Sounds like impossible to copy 8Gb under 500ms, also disk?

So fork time is actually O(1) with VM size, its 500ms even for 64gb + disk. We're using some pretty weird COW techniques to pull it off.

O(1) What! What might bring it down to say 10's of ms? Looks like its some kind of optimizable wall that its 500 for everything.

Like with 10ms then online replication/backup — analogus to litestream for sqlite — but for in memory processes becomes feasible, no?

Re: Launch HN: Freestyle – Sandboxes for Coding Agents

#129

Post by benswerd »

ghm2199 wrote: post #126
benswerd wrote: post #86

Earlier quoted context omitted.

So fork time is actually O(1) with VM size, its 500ms even for 64gb + disk. We're using some pretty weird COW techniques to pull it off.

O(1) What! What might bring it down to say 10's of ms? Looks like its some kind of optimizable wall that its 500 for everything. Like with 10ms then online replication/backup — analogus to litestream for sqlite — but for in memory processes becomes feasible, no?

We're actually median under 500ms — ~320ms median — I just didn't want to piss of hacker news with over estimatation.

We have another set of optimizations that we believe can take us to ~200ms in the next few months but beyond that we're pretty much completely stuck.

Realistically other sandboxes will be able to get there before us because we've chosen to support so much of Linux/if you don't run an operating system or don't support custom snapshots that is much easier.

Re: Launch HN: Freestyle – Sandboxes for Coding Agents

#130

Post by esseph »

> In order to make this possible, we’ve moved to our own bare metal racks. Early in our testing we realized that moving VMs across cloud nodes would not have acceptable performance properties. We asked Google Cloud and AWS for a quote on their bare metal nodes and found that the monthly cost was equivalent to the total cost of the hardware so we did that.

Yes! And good on you, well-tuned bare metal performance is hard to beat.

Post reply on HN