Notes on the new Claude analysis JavaScript code execution tool

#1

Post by bstsb » Fri, Oct 25, 2024, 9:40 AM UTC

Notes on the new Claude analysis JavaScript code execution tool
simonwillison.net

Re: Notes on the new Claude analysis JavaScript code execution tool

#2

Post by animal_spirits » Fri, Oct 25, 2024, 3:25 PM UTC

That's an interesting idea to generate javascript and execute it client side rather than server side. I'm sure that saves a ton of money for Anthropic not by not having to spin up a server for each execution.

Re: Notes on the new Claude analysis JavaScript code execution tool

#3

Post by simonw » Fri, Oct 25, 2024, 3:27 PM UTC

I've been trying to figure out the right pattern for running untrusted JavaScript code in a browser sandbox that's controlled by a page for a while now, looks like Anthropic have figured that out. Hoping someone can reverse engineer exactly how they are doing this - their JavaScript code is too obfuscated for me to dig out the tricks, sadly.

Re: Notes on the new Claude analysis JavaScript code execution tool

#4

Post by dartos » Fri, Oct 25, 2024, 3:41 PM UTC

simonw wrote: post #3

I've been trying to figure out the right pattern for running untrusted JavaScript code in a browser sandbox that's controlled by a page for a while now, looks like Anthropic have figured that out. Hoping someone can reverse engineer exactly how they are doing this - their JavaScript code is too obfuscated for me to dig out the tricks, sadly.

Isn’t that how all JavaScript code runs in a browser?

Re: Notes on the new Claude analysis JavaScript code execution tool

#5

Post by aabhay » Fri, Oct 25, 2024, 3:46 PM UTC

simonw wrote: post #3

I've been trying to figure out the right pattern for running untrusted JavaScript code in a browser sandbox that's controlled by a page for a while now, looks like Anthropic have figured that out. Hoping someone can reverse engineer exactly how they are doing this - their JavaScript code is too obfuscated for me to dig out the tricks, sadly.

What are the attack vectors for a web browser js environment to do malicious things? All browser code is sandboxed via origin controls, and process isolation. It can’t even open an iframe and read the contents of that iframe.

Re: Notes on the new Claude analysis JavaScript code execution tool

#6

Post by koolala » Fri, Oct 25, 2024, 3:49 PM UTC

JavaScript is the perfect language for this. I can't wait for a sandboxed coding environment to totally set AI loose.

Re: Notes on the new Claude analysis JavaScript code execution tool

#7

Post by mlejva » Fri, Oct 25, 2024, 3:51 PM UTC

koolala wrote: post #6

JavaScript is the perfect language for this. I can't wait for a sandboxed coding environment to totally set AI loose.

Shameless plug here. We're building exactly this at E2B [0] (I'm the CEO). Sandboxed cloud environments for running AI-generated code. We're fully open-source [1] as well.

[0] https://e2b.dev

[1] https://github.com/e2b-dev

Re: Notes on the new Claude analysis JavaScript code execution tool

#8

Post by thenaturalist » Fri, Oct 25, 2024, 4:05 PM UTC

Funnily enough, I test code generation both on unpaid Claude and ChatGPT.

When working with Python, I've found Sonnet (pre 3.5) to be quite superior to ChatGPT (mostly 4, sometimes 3.5) with regards to verbosity, structure and prompt / instruct comprehension.

I've switched to a JavaScript project two weeks ago and the tables have turned.

Sonnet 3.5 is much more verbose and I need to make corrections a few times, whereas ChatGPTs output is shorter and on point.

I'll closely follow if this improves if Claude are focussing on JS themselves.

Re: Notes on the new Claude analysis JavaScript code execution tool

#9

Post by willsmith72 » Fri, Oct 25, 2024, 4:09 PM UTC

This is a great step, but to me not very useful until the move out of context. Still I'm high on anthropic and happy gen ai didn't turn into a winner-take-all market like everyone predicted in 2021.

Re: Notes on the new Claude analysis JavaScript code execution tool

#10

Post by TimTheTinker » Fri, Oct 25, 2024, 4:14 PM UTC

simonw wrote: post #3

I've been trying to figure out the right pattern for running untrusted JavaScript code in a browser sandbox that's controlled by a page for a while now, looks like Anthropic have figured that out. Hoping someone can reverse engineer exactly how they are doing this - their JavaScript code is too obfuscated for me to dig out the tricks, sadly.

You should check out how Figma plugins work. They have blog posts on all the tradeoffs they considered.

What I believe they settled on was a JS interpreter compiled to WASM -- it can run arbitrary JS but with very well-defined and restricted interfaces to the outside world (the browser's JS runtime environment).