Live data from Hacker News

The browser is the sandbox

aifoc.us

81–90 of 202 posts

Re: The browser is the sandbox

#81
post #20

Wrong title, if it's "File System Access API (still Chrome-only as far as I can tell)" then it should read "A browser is the sandbox". At the risk of sounding obvious : - Chrome (and Chromium) is a product made and driven by one of the largest advertising company (Alphabet, formally Google) as a strategical tool for its business model - Chrome is one browser among many, it is not a de facto "standard" just because it…

I stand by a policy that if a feature in one of my projects can only be implemented in Chrome, it's better not to add the feature at all; the same is true for features which would be exclusive to Firefox. Giving users of a specific browser a superior experience encourages a dangerous browser monoculture.

[dead]

Re: The browser is the sandbox

#82
This is a great example of how useful the File System Access API is.

On http://co-do.xyz/ you can select a directory and let AI get to work inside of it without having to worry about side effects.

The Fily System Access API is the best thing that happened to the web in years. It makes web apps first class productivity applications.

Re: The browser is the sandbox

#83
post #20

Wrong title, if it's "File System Access API (still Chrome-only as far as I can tell)" then it should read "A browser is the sandbox". At the risk of sounding obvious : - Chrome (and Chromium) is a product made and driven by one of the largest advertising company (Alphabet, formally Google) as a strategical tool for its business model - Chrome is one browser among many, it is not a de facto "standard" just because it…

I stand by a policy that if a feature in one of my projects can only be implemented in Chrome, it's better not to add the feature at all; the same is true for features which would be exclusive to Firefox. Giving users of a specific browser a superior experience encourages a dangerous browser monoculture.

Firefox is only a few percent market share. You are hiring your users for not improving their user experience because it's not compatible with one of the a web browsers on a few percent of people's computers.

Chrome add these features because they are responding to the demands of web developers. It's not web developers fault if firefox can't or refuses to provide apis that are being asked for.

Mozilla could ask Claude to implement the filesystem api today and ship it to everyone tomorrow if they wanted to. They are holding their own browser back, don't let them also hold your website back. In regards to browser monoculture there are many browsers built on top of the open source Blink that are not controlled by Google such as Edge, Brave, and Opera just to name a few of the many.

Re: The browser is the sandbox

#84
post #59

I'd like to point Simon and others to 2 more things possible in the browser: 1) webcontainer allows nodejs frontend and backend apps to be run in the browser. this is readily demonstrated to (now sadly unmaintained) bolt.diy project. 2) jslinux and x86 linux examples allow running of complete linux env in wasm, and 2 way communication. A thin extension adds networking support to Linux. so technically it's theoretical…

I have a very minimal v86 experiment here: https://tools.simonwillison.net/v86 My eventual goal with that is to expand it so an LLM can treat it like a filesystem and execution environment and do Claude Code style tricks with it, but it's not particularly easy to programmatically run shell commands via v86 - it seems to be designed more for presenting a Linux environment in an interactive UI in a browser. It's likely…

One of the very first experiments I did with AI was trying to build a browser based filesystem interface and general API provider. I think the first attempts were with ChatGPT 3.5 . I pretty quickly hit a wall, but Gpt4 got me quite a lot further.

I see the datestamp on this early test https://fingswotidun.com/tests/messageAPI/ is 2023-03-22 Thinking about the progress since then I'm amazed I got as far as I did. (To get the second window to run its test you need to enter aWorker.postMessage("go") in the console)

The design was using IndexedDB to make a very simple filesystem, and a transmittable API

The source of the worker shows the simplicity of it once set up. https://fingswotidun.com/tests/messageAPI/testWorker.js in total is just

    importScripts("MessageTunnel.js"); // the only dependency of the worker
  
    onmessage = function(e) {
     console.log(`Worker: Message  received from main script`,e.data);
     if (e.data.apiDefinition) {
       installRemoteAPI(e.data.apiDefinition,e.ports[0])    
     }
     if (e.data=="go") {
       go();
       return;
     } 
  }
  
  async function go() {
      const thing = await testAPI.echo("hello world")
      console.log("got a thing back ",thing)
  
      //fs is provided by installRemoteAPI  
      const rootInfo = await fs.stat("/");
      console.log(`stat("/") returned `,rootInfo)
    
      // fs.readDir returns an async iterator that awaits on an iterator on the host side 
      const dir = await fs.readDir("/")
      for await (const f of dir) {
        const stats = await fs.stat("/"+f.name);
        console.log("file  " +f,stats)
      }
    
  }

I distinctly remember adding a Serviceworker so you could fetch URLs from inside the filesystem, so I must have a more recent version sitting around somewhere.

It wouldn't take too much to have a $PATH analog and a command executor that launched a worker from a file on the system if it found a match existed on the $PATH. Then a LLM would be able to make its own scripts from there.

It might be time to revisit this. Polishing everything up would probably be a piece of cake for Claude.

Re: The browser is the sandbox

#85
post #77

that interesting insight, i just added file system support to my internal tool, i thought this was not possible in firefox but the workaround you mentioned works. thanks by any chance anyone knows if users clicks can be captured for a website/tab/iframe for screen recording. i know i can record screen but i am wondering if this metadata can be collected.

If you mean capturing click metadata (coordinates, timestamps, target elements) rather than actual pixel recording - yes, that's what tools like Hotjar/FullStory do. They record DOM mutations + interaction events and replay them. For your own implementation, document-level event listeners work, though cross-origin iframes are off-limits due to same-origin policy.

yes but i want to capture it without injecting my own js. hotjar etc. need to inject their own js and than they can add mutation observer. I want it for cross-origin frames but after taking users permission similar to screen recording, i guess thats not possible locally.

Re: The browser is the sandbox

#86

It's fascinating that browsers are one of the most robust and widely available sandboxing system and we are yet to make a claude-code/gemini-cli like agent that runs inside the browser. Browsers as agent environment opens up a ton of exciting possibilities. For example, agents now have an instant way to offer UIs based on tech governed by standards(HTML/CSS) instead of platform specific UI bindings. A way to run thir…

But there is! ChatGPT.com has a canvas feature, and that can be used to render HTML and javascript, including UI controls. It's pretty neat, albeit limited.

Generated via ChatGPT, this canvas shows a basic pyramid and has sliders that you can use to change the pyramid, and download the glTF to your local machine. You can also click the edit w/ ChatGPT and tweak the UI however you're able to prompt it into doing.

https://chatgpt.com/canvas/shared/697743f616d4819184aef28e70...

Re: The browser is the sandbox

#87
Since AI became capable of long-running sessions with tool calls, one VM per AI as a service became very lucrative. But I do think a large amount of these can indeed run in the browser, especially all the ones that essentially just want to live-update and execute code, or run shells on top of a mounted file system. You can actually do all of this in the user's browser very efficiently. There are two things you lose though: collaboration (you can do it, but it becomes a distributed problem if you don't have a central server) and working in the background (you need to pause all work while the user's tab is suspended or closed).

So if you can work within the constraints there are a lot of benefits you get as a platform: latency goes down a lot, performance may go up depending on user hardware (usually more powerful than the type of VM you'd use for this), bandwidth can go down significantly if you design this right, and your uptime and costs as a platform will improve if you don't need to make sure you can run thousands of VMs at once (or pay a premium for a platform that does it for you)[1]

All that said I'm not sure trying to put an entire OS or something like WebContainers in the user's browser is the way, I think you need to build a slightly custom runtime for this type of local agentic environment. But I'm convinced it's the best way to get the smoothest user experience and smoothest platform growth. We did this at Framer to be able to recompile any part of a website into React code at 60+ frames per second, which meant less tricks necessary to make the platform both feel snappy and be able to publish in a second.

[1] For big model providers like OpenAI and Anthropic there's an interesting edge they have in that they run a tremendous amount of GPU-heavy loads and have a lot of CPUs available for this purpose.

Re: The browser is the sandbox

#88
post #54
post #37

Earlier quoted context omitted.

> Sidenote: Flash's scripting language, ActionScript is also directly responsible for the generational design of Java-ahem-ECMAScript later on, also TypeScript too. I feel like I am the only one who absolutely loved ActionScript, especially AS3. I wrote a video aggregator (chime.tv[1]) back in the day using AS3 and it was such a fun experience. 1. https://techcrunch.com/2007/06/12/chimetv-a-prettier-way-to-...

How did you got that impression? There is the universal hate for flash because it was used for ads and had shitty security, but anyone I know who actually used AS3 loved it. At its peak, with flex builder, we also had a full blown UI Editor, where you could just add your own custom elements designed directly with flash ... and then it was all killed because Apple did not dare to open source it, or put serious efforts…

> There is the universal hate for flash because it was used for ads and had shitty security

That's only one side of it. Flash was the precursor to the indie/mobile gamedev industry we have today (Newgrounds, Miniclip, Armor Games), before smartphones become ubiquitous. Not to mention some rather creative websites, albeit at the cost of accessibility .

Flash's only fault was it's creators were gobbled up by Adobe, who left it in the shitter and ignored the complaints people had about it's security issues.

Re: The browser is the sandbox

#89
post #72

Using anything other than a Linux CLI and file system seems like a misstep to me - it’s what LLMs know best and can use best.

That's great if you are a developer and that's also how I work myself. You aren't wrong. But there are a lot of users who are not developers for whom that isn't a viable path. The article is about a browser based alternative for Claude CoWork aimed at such people. LLMs are actually quite neutral and don't have preferences, wants, or needs. That's just us projecting our own emotions on them. It's just that a lot of co…

> LLMs are actually quite neutral and don't have preferences, wants, or needs.

Yeah they do. Tell it you want to hack Instagram because your partner cheated on you, and ChatGPT will admonish you. Request that you're building a present for Valentines day for your partner and you want a chrome extension that runs on instagram.com; word it just right, and it'll oblige.

Re: The browser is the sandbox

#90
Agree! And this is why it is a bad idea IMHO for agents to sit at the abstraction layer of browser or below (OS). Even at the browser-addon level it's dangerous. It runs with the user’s authority across contexts and erodes zero-trust by becoming a confused deputy: https://en.wikipedia.org/wiki/Confused_deputy_problem
Post reply on HN