Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Notes on the new Claude analysis JavaScript code execution tool

simonwillison.net

Post reply on HN
Print view View original

31–40 of 60 posts

Re: Notes on the new Claude analysis JavaScript code execution tool

#31

Post by bhl »

TimTheTinker wrote: post #10
simonw wrote: post #3

I've been trying to figure out the right pattern for running untrusted JavaScript code in a browser sandbox that's controlled by a page for a while now, looks like Anthropic have figured that out. Hoping someone can reverse engineer exactly how they are doing this - their JavaScript code is too obfuscated for me to dig out the tricks, sadly.

You should check out how Figma plugins work. They have blog posts on all the tradeoffs they considered. What I believe they settled on was a JS interpreter compiled to WASM -- it can run arbitrary JS but with very well-defined and restricted interfaces to the outside world (the browser's JS runtime environment).

> We now use QuickJS, a JavaScript VM written in C and cross-compiled to WebAssembly.

https://www.figma.com/blog/an-update-on-plugin-security/

Re: Notes on the new Claude analysis JavaScript code execution tool

#32

Post by bhl »

mlejva wrote: post #7
koolala wrote: post #6

JavaScript is the perfect language for this. I can't wait for a sandboxed coding environment to totally set AI loose.

Shameless plug here. We're building exactly this at E2B [0] (I'm the CEO). Sandboxed cloud environments for running AI-generated code. We're fully open-source [1] as well. [0] https://e2b.dev [1] https://github.com/e2b-dev

Is sandboxed browser environments on your roadmap? Would much prefer to use the client's runtime for non-computational expensive things like web dev.

Re: Notes on the new Claude analysis JavaScript code execution tool

#33

Post by rekttrader »

bhl wrote: post #31
TimTheTinker wrote: post #10

Earlier quoted context omitted.

You should check out how Figma plugins work. They have blog posts on all the tradeoffs they considered. What I believe they settled on was a JS interpreter compiled to WASM -- it can run arbitrary JS but with very well-defined and restricted interfaces to the outside world (the browser's JS runtime environment).

> We now use QuickJS, a JavaScript VM written in C and cross-compiled to WebAssembly. https://www.figma.com/blog/an-update-on-plugin-security/

Yo dog, we put a JavaScript VM inside your JavaScript VM

Re: Notes on the new Claude analysis JavaScript code execution tool

#34

Post by l1n »

advaith08 wrote: post #22

The custom instructions to the model say: "Please note that this is similar but not identical to the antArtifact syntax which is used for Artifacts; sorry for the ambiguity." They seem to be apologizing to the model in the system prompt?? This is so intriguing

Multiple system prompt segments can be composed depending on needs, so it's useful for this sort of thing to be there to resolve inconsistencies.

Re: Notes on the new Claude analysis JavaScript code execution tool

#35

Post by pawelduda »

andai wrote: post #23
advaith08 wrote: post #22

The custom instructions to the model say: "Please note that this is similar but not identical to the antArtifact syntax which is used for Artifacts; sorry for the ambiguity." They seem to be apologizing to the model in the system prompt?? This is so intriguing

Has anyone looked into the effect of politeness on performance?

If you assume asking someone nicely is more likely for them to try help you, and this tendency shows in the training set, wouldn't you be more likely to "retrieve" a better answer from the model trained on it? Take this with a grain of salt, it's just my guess not backed by anything

Re: Notes on the new Claude analysis JavaScript code execution tool

#36

Post by purple-leafy »

spankalee wrote: post #11
simonw wrote: post #3

I've been trying to figure out the right pattern for running untrusted JavaScript code in a browser sandbox that's controlled by a page for a while now, looks like Anthropic have figured that out. Hoping someone can reverse engineer exactly how they are doing this - their JavaScript code is too obfuscated for me to dig out the tricks, sadly.

The key is running the untrusted code in a cross-origin iframe so you can rely on the same-origin policies and `sandbox`[1]. You can control the code in a number of ways - loading a trusted shim that sets up a postMessage handler is pretty common. You can be careful and do that in a way that untructed code can't forge messages to look like their from the trusted code. Another way is to use two iframes to the untruste…

The cross origin iframe method is the same I’ve employed in A few browser extensions I’ve built

Re: Notes on the new Claude analysis JavaScript code execution tool

#37

Post by cruffle_duffle »

Me1000 wrote: post #29
stanleydrew wrote: post #19

Earlier quoted context omitted.

Also means you're not having to do a bunch of isolation work to make the server-side execution environment safe.

This is the real value here. Keeping a secure environment to run untrusted code along side user data is a real liability for them. It's not their core competency either, so they can just lean on browser sandboxing and not worry about it.

How is doing it server side a different challenge than something like google collab or any of those Jupyter notebook type services?

Re: Notes on the new Claude analysis JavaScript code execution tool

#38

Post by dartos »

TheRealPomax wrote: post #14
dartos wrote: post #4

Earlier quoted context omitted.

Isn’t that how all JavaScript code runs in a browser?

Isn't what how all JS runs in the browser? There are different restrictions based on where JS comes from, and what context it gets loaded into.

All browser js runs in a browser sandbox and, by default, none of it needs to be explicitly trusted in most browsers.

I don’t think there are very many restrictions on what js can do on a given page. At least none come to mind.

Not really sure you mean by “context” either. Maybe service workers? Unless you’re talking about loading js within iframes… but that’s a different can of worms.

Re: Notes on the new Claude analysis JavaScript code execution tool

#39

Post by tkgally »

andai wrote: post #23
advaith08 wrote: post #22

The custom instructions to the model say: "Please note that this is similar but not identical to the antArtifact syntax which is used for Artifacts; sorry for the ambiguity." They seem to be apologizing to the model in the system prompt?? This is so intriguing

Has anyone looked into the effect of politeness on performance?

I've wondered the same thing. I tend to sprinkle my LLM prompts with "please"s, especially with longer prompts, as I feel that "please" might make clearer where the main request to the LLM is. I have no evidence that they actually yield better results, though, and people I share my prompts with might think I'm anthropomorphizing the models.

Re: Notes on the new Claude analysis JavaScript code execution tool

#40

Post by bravura »

thenaturalist wrote: post #8

Funnily enough, I test code generation both on unpaid Claude and ChatGPT. When working with Python, I've found Sonnet (pre 3.5) to be quite superior to ChatGPT (mostly 4, sometimes 3.5) with regards to verbosity, structure and prompt / instruct comprehension. I've switched to a JavaScript project two weeks ago and the tables have turned. Sonnet 3.5 is much more verbose and I need to make corrections a few times, wher…

Don't call me crazy (I am actually), but sometimes I will keep both ChatGPT and Claude open side-by-side and use them to audit each other.

I'll give them the same prompt.

When they respond, re-prompt with: "What are your thoughts on this approach? Pros and cons. Integrate the best ideas from both: [answer from the other model]"

Repeat until total satisfaction or frustration is achieved.

Post reply on HN