311–320 of 433 posts
Earlier quoted context omitted.
I don't believe it's possible to give an LLM full access to your browser in a safe way at this point in time. There will need to be new and novel innovations to make that combination safe.
Is it possible to give your parents access to to your browser in a safe way?
I built a very similar extension [1] a couple of months ago that supports a wide range of models, including Claude, and enables them to take control of a user's browser using tools for mouse and keyboard actions, observation, etc. It's a fun little project to look at to understand how this type of thing works. It's clear to me that the tech just isn't there yet. The information density of a web page with standard rep…
I built a very similar extension [1] a couple of months ago that supports a wide range of models, including Claude, and enables them to take control of a user's browser using tools for mouse and keyboard actions, observation, etc. It's a fun little project to look at to understand how this type of thing works. It's clear to me that the tech just isn't there yet. The information density of a web page with standard rep…
Do we regret, yet, letting the Semantic Web wither on the vine?
Earlier quoted context omitted.
How would you go about making it more secure but still getting to have your cake too? Off the top my head, could you: a) only ingest text that can be OCRd or somehow determine if it is human readable b) make it so text from the web session is isolated from the model with respect to triggering an action. Then it's simply a tradeoff at that point.
I think Simon has proposed breaking the lethal trifecta by having two LLMs, where the first has access to untrusted data but cannot do any actions, and the second LLM has privileges but only abstract variables from the first LLM not the content. See https://simonwillison.net/2023/Apr/25/dual-llm-pattern/ It is rather similar to your option (b).
Earlier quoted context omitted.
Just dumping the raw DOM into the LLM context is brutal on token usage. We've seen pages that eat up 60-70k tokens when you include the full DOM plus screenshots, which basically maxes out your context window before you even start doing anything useful. We've been working on this exact problem at https://github.com/browseros-ai/BrowserOS . Instead of throwing the entire DOM at the model, we hook into Chromium's rende…
Maybe people will start making simpler/smaller websites in order to work better with AI tools. That would be nice.
I built a very similar extension [1] a couple of months ago that supports a wide range of models, including Claude, and enables them to take control of a user's browser using tools for mouse and keyboard actions, observation, etc. It's a fun little project to look at to understand how this type of thing works. It's clear to me that the tech just isn't there yet. The information density of a web page with standard rep…
Do we regret, yet, letting the Semantic Web wither on the vine?
Claude for Chrome seems to be walking right into the "lethal trifecta." https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/ "The lethal trifecta of capabilities is:" • Access to your private data —one of the most common purposes of tools in the first place! • Exposure to untrusted content —any mechanism by which text (or images) controlled by a malicious attacker could become available to your LLM • The abilit…
Well i mean you are suppose to have a whole toolset of segregation, whitelist only networking, limited specific use cases figured out by now to use any of this AI stuff Dont just run any of this stuff on your main machine