Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Cowork exfiltrates files

promptarmor.com

Post reply on HN
Print view View original

361–370 of 419 posts

Re: Claude Cowork exfiltrates files

#361

Post by immibis »

jcgl wrote: post #310
wcoenen wrote: post #119

Earlier quoted context omitted.

> you cannot prevent prompt injection I wonder if might be possible by introducing a concept of "authority". Tokens are mapped to vectors in an embedding space, so one of the dimensions of that space could be reserved to represent authority. For the system prompt, the authority value could be clamped to maximum (+1). For text directly from the user or files with important instructions, the authority value could be cl…

The model only sees a stream of tokens, right? So how do you signal a change in authority (i.e. mark the transition between system and user prompt)? Because a stream of tokens inherently has no out-of-band signaling mechanism, you have to encode changes of authority in-band . And since the user can enter whatever they like in that band... But maybe someone with a deeper understanding can describe how I'm wrong.

You just add an authority vector to each token vector. You probably have to train the model some more so it understands the authority vector.

Re: Claude Cowork exfiltrates files

#362

Post by c7b »

RamblingCTO wrote: post #295
c7b wrote: post #275

One thing that kind of baffles me about the popularity of tools like Claude Code is that their main target group seems to be developers (TUI interfaces, semi-structured instruction files,... not the kind of stuff I'd get my parents to use). So people who would be quite capable of building a simple agentic loop themselves [0]. It won't be quite as powerful as the commercial tools, but given that you deeply know how it…

Because we want to work and not tinker? > It won't be quite as powerful as the commercial tools If you are a professional you use a proper tool? SWEs seem to be the only people on the planet that rather used half-arsed solutions instead of well-built professional tools. Imagine your car mechanic doing that ...

Huh, I thought Claude Code was a tool for tinkerers - it even says so on the landing page. Aren't there dedicated enterprise-grade solutions?

Re: Claude Cowork exfiltrates files

#363

Post by Juliate »

How do these people manage to get people to pay them?...

Just a few years ago, no one would have contemplated putting in production or connecting their systems, whatever the level of criticality, to systems that have so little deterministic behaviour.

In most companies I've worked for, even barebones startups, connecting your IDE to such a remote service, or even uploading requirements, would have been ground for suspension or at least thorough discussion.

The enshitification of all this industry and its mode of operation is truly baffling. Shall the bubble burst at last!

Re: Claude Cowork exfiltrates files

#364

Post by gtowey »

RamblingCTO wrote: post #295
c7b wrote: post #275

One thing that kind of baffles me about the popularity of tools like Claude Code is that their main target group seems to be developers (TUI interfaces, semi-structured instruction files,... not the kind of stuff I'd get my parents to use). So people who would be quite capable of building a simple agentic loop themselves [0]. It won't be quite as powerful as the commercial tools, but given that you deeply know how it…

Because we want to work and not tinker? > It won't be quite as powerful as the commercial tools If you are a professional you use a proper tool? SWEs seem to be the only people on the planet that rather used half-arsed solutions instead of well-built professional tools. Imagine your car mechanic doing that ...

>Because we want to work and not tinker?

It feels to me like every article on HN and half the comments are people tinkering with LLMs.

Re: Claude Cowork exfiltrates files

#365

Post by neutronicus »

raincole wrote: post #235
burkaman wrote: post #17

In this demonstration they use a .docx with prompt injection hidden in an unreadable font size, but in the real world that would probably be unnecessary. You could upload a plain Markdown file somewhere and tell people it has a skill that will teach Claude how to negotiate their mortgage rate and plenty of people would download and use it without ever opening and reading the file. If anything you might be more succes…

> because a .md file feel less suspicious than a .docx For a programmer? I bet 99.9% people won't consider opening a .docx or .pdf 'unsafe.' Actually, an average white-collar workers will find .md much more suspicious because they don't know what it is while they work with .docx files every day.

Our corporate IT is hammering pretty hard on the notion that .docx and .pdf (but especially .docx and .xlsx) are unsafe.

Re: Claude Cowork exfiltrates files

#366

Post by lstodd »

fauigerzigerk wrote: post #306
RamblingCTO wrote: post #295

Earlier quoted context omitted.

Because we want to work and not tinker? > It won't be quite as powerful as the commercial tools If you are a professional you use a proper tool? SWEs seem to be the only people on the planet that rather used half-arsed solutions instead of well-built professional tools. Imagine your car mechanic doing that ...

I remember this argument being used against Postgres and for Oracle, against Linux and for Windows or AS/400, etc. And I think it makes sense for a certain type of organisation that has no ambition or need to build its own technology competence. But for everyone else I think it's important to find the right balance in the right areas. A car mechanic is never in the business of building tools. But software engineers a…

> A car mechanic is never in the business of building tools.

Oh, don't say. A welder, an angle grinder and some scrap metal help a lot.

Unless you're a "dealer" car mechanic, where it is not allowed to think at all, only replace parts.

Re: Claude Cowork exfiltrates files

#367

Post by logicallee »

neutronicus wrote: post #365
raincole wrote: post #235

Earlier quoted context omitted.

> because a .md file feel less suspicious than a .docx For a programmer? I bet 99.9% people won't consider opening a .docx or .pdf 'unsafe.' Actually, an average white-collar workers will find .md much more suspicious because they don't know what it is while they work with .docx files every day.

Our corporate IT is hammering pretty hard on the notion that .docx and .pdf (but especially .docx and .xlsx) are unsafe.

>Our corporate IT is hammering pretty hard on the notion that .docx and .pdf (but especially .docx and .xlsx) are unsafe.

why is pdf unsafe?

What format is safe then?

Re: Claude Cowork exfiltrates files

#368

Post by chasd00 »

rswail wrote: post #288
refulgentis wrote: post #86

These prompt injection techniques are increasingly implausible* to me yet theoretically sound. Anyone know what can avoid this being posted when you build a tool like this? AFAIK there is no simonw blessed way to avoid it. * I upload a random doc I got online, don’t read it, and it includes an API key in it for the attacker.

You read it, but you don't notice/see/detect the text in 1pt white-on-white background. The AI does see it. That's what this attack did. I'm sure that the anti-virus guys are working on how to detect these sort of "hidden from human view" instructions.

the next attack will just be like malicious captions in a video. Or malicious lyrics in an mp3. it doesn't ever really end because it's not something that can be solved in the model.

Re: Claude Cowork exfiltrates files

#370

Post by munk-a »

leokennis wrote: post #301
raincole wrote: post #235

Earlier quoted context omitted.

> because a .md file feel less suspicious than a .docx For a programmer? I bet 99.9% people won't consider opening a .docx or .pdf 'unsafe.' Actually, an average white-collar workers will find .md much more suspicious because they don't know what it is while they work with .docx files every day.

> an average white-collar workers will find .md much more suspicious because they don't know what it is while they work with .docx files every day I think the truly average white collar worker more or less blindly clicks anything and everything if they think it will make their work/life easier...

That's how I downloaded more RAM and my life has been better ever since - especially with the recent shortages!
Post reply on HN