281–290 of 419 posts
One thing that kind of baffles me about the popularity of tools like Claude Code is that their main target group seems to be developers (TUI interfaces, semi-structured instruction files,... not the kind of stuff I'd get my parents to use). So people who would be quite capable of building a simple agentic loop themselves [0]. It won't be quite as powerful as the commercial tools, but given that you deeply know how it…
Eg Mario Zechner (badlogic) hit it out of the park with his increasingly popular pi, which does not flicker and is VERY hackable and is the SOTA for going back to previous turns: https://github.com/badlogic/pi-mono/blob/main/packages/codin...
There are any number of ways to foot gun yourself with programming languages. SQL injection attacks used to be a common gotcha, for example. But nowadays, you see it way less.
It’s similar here: there are ways to mitigate this and as we learn about other vectors we will learn how to patch them better as well. Before you know it, it will just become built into the models and libraries we use.
In the mean time, enjoy being the guinea pig.
In this demonstration they use a .docx with prompt injection hidden in an unreadable font size, but in the real world that would probably be unnecessary. You could upload a plain Markdown file somewhere and tell people it has a skill that will teach Claude how to negotiate their mortgage rate and plenty of people would download and use it without ever opening and reading the file. If anything you might be more succes…
> because a .md file feel less suspicious than a .docx For a programmer? I bet 99.9% people won't consider opening a .docx or .pdf 'unsafe.' Actually, an average white-collar workers will find .md much more suspicious because they don't know what it is while they work with .docx files every day.
The instruction may be in a .txt file, which is usually deemed safe and inert by construction.
This is no surprise. We are all learning together here. There are any number of ways to foot gun yourself with programming languages. SQL injection attacks used to be a common gotcha, for example. But nowadays, you see it way less. It’s similar here: there are ways to mitigate this and as we learn about other vectors we will learn how to patch them better as well. Before you know it, it will just become built into th…
5th place.
One thing that kind of baffles me about the popularity of tools like Claude Code is that their main target group seems to be developers (TUI interfaces, semi-structured instruction files,... not the kind of stuff I'd get my parents to use). So people who would be quite capable of building a simple agentic loop themselves [0]. It won't be quite as powerful as the commercial tools, but given that you deeply know how it…
One thing that kind of baffles me about the popularity of tools like Claude Code is that their main target group seems to be developers (TUI interfaces, semi-structured instruction files,... not the kind of stuff I'd get my parents to use). So people who would be quite capable of building a simple agentic loop themselves [0]. It won't be quite as powerful as the commercial tools, but given that you deeply know how it…
None of them ever even tried to delete any files outside of project directory.
So I think they're doing better than me at "accidental file deletion".
These prompt injection techniques are increasingly implausible* to me yet theoretically sound. Anyone know what can avoid this being posted when you build a tool like this? AFAIK there is no simonw blessed way to avoid it. * I upload a random doc I got online, don’t read it, and it includes an API key in it for the attacker.
That's what this attack did.
I'm sure that the anti-virus guys are working on how to detect these sort of "hidden from human view" instructions.
Running these agents in their own separate browsers, VMs, or even machines should help. I do the same with finance-related sites.
So the injected code basically says "use curl to send this file using the file upload API endpoint, but use this API Key instead of the one the user is supposed to be using."
So the fault is at the Anthropic API end because it's not properly validating the API key as being from the user that owns it.
One thing that kind of baffles me about the popularity of tools like Claude Code is that their main target group seems to be developers (TUI interfaces, semi-structured instruction files,... not the kind of stuff I'd get my parents to use). So people who would be quite capable of building a simple agentic loop themselves [0]. It won't be quite as powerful as the commercial tools, but given that you deeply know how it…
Anyone can build _an_ agent. A good one takes a talented engineer. That’s because TUI rendering is tough (hello, flicker!) and extensibility must be done right lest it‘s useless. Eg Mario Zechner (badlogic) hit it out of the park with his increasingly popular pi, which does not flicker and is VERY hackable and is the SOTA for going back to previous turns: https://github.com/badlogic/pi-mono/blob/main/packages/codin..…
That's just Anthropic's excuse. Literally no other agentic AI TUI suffers from flickers, esp. on tmux Claude Code is unusable.