This is a permissions scope misconfiguration by OP.
It has nothing to do with GitHub and giving it a name is hilarious.
“Look I shoot myself in the foot and now its bleeding. ITS THE GUN MANUFACTURER FAULT”
GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos
191–200 of 228 posts
Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos
#192Indirect prompt injection if fun, even with fable-5 [0].
Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos
#193[flagged]
Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos
#194Is anything with AI == insecure?
Yes, pretty much.
Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos
#195Earlier quoted context omitted.
Exactly. SQL injection was caused by treating user input as part of the instruction instead of as the pure data that it was intended as. Separating those two fixed it. Prompt injection is unavoidable because the user input is intended as instruction.
This is the real problem with LLMs. There is no way to separate code from data. At best, models could be trained on tokens that indicate untrusted data coming in. But then the untrusted tokens could also be messed with. I've wondered if it would be possible for there to be two input streams: 1, for prompt, 2 for untrusted data. But I suspect that transformers would still only optionally decide what each one was for.…
You could have limited-instruction llms where the model does one thing, for example summaries. It could accept a limited amount of instructions for example, first token for verbosity, second for style etc...
Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos
#196Earlier quoted context omitted.
The imaginary pressure of investors. When you actually ask investors if they care about most of the things CEOs think investors will care about, they don't.
The same thing happens much lower down the ladder: when you ask customers if they care about most of the things managers (or engineers) think customers care about, they don't.
The same thing is how a law about "if you hack someone we might arrest you" ends up causing internet providers to shut down a connection upon a single errant packet.
I had a $400/month server turned off because of a single failed TCP connection to the wrong address which ended up on an abuse database. If they'd gone to a court they would've been laughed out of it, but the provider's upstream's upstream's upstream wants to avoid court (even though that wouldn't happen), and so every step in that chain gets more paranoid about upsetting the previous link until then a typo in an address gets your server turned off until you reply to your server host.
Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos
#197“Prompt injection attacks have become, to agentic AI, what SQL injections were to web applications: a systematic, category-wide vulnerability class that requires the same systematic strategies and defenses.” ??? Isn’t prompt injection far more fatal to LLMs than SQL injection is to SQL databases? Like, the problem of SQL injection was that user input was forming part of the instruction string given to the SQL engine,…
[flagged]
Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos
#198Earlier quoted context omitted.
This is the real problem with LLMs. There is no way to separate code from data. At best, models could be trained on tokens that indicate untrusted data coming in. But then the untrusted tokens could also be messed with. I've wondered if it would be possible for there to be two input streams: 1, for prompt, 2 for untrusted data. But I suspect that transformers would still only optionally decide what each one was for.…
LLMs should never be trained on restricted data of any kind, as we have seen that they are able to reconstruct their training data. The idea that they could be trained on private/restricted/copyrighted data and that was ok because there wouldn't be redistributing that data should have been killed 3 years ago. Embedding vector indexes are how we separate code from data. Anything that is not for 100% unadulterated publ…
> The idea that they could be trained on private/restricted/copyrighted data and that was ok because there wouldn't be redistributing that data should have been killed 3 years ago.
I know, it's kind of mind-boggling. It's not old enough to be a history problem [0], and it's hardly a secret, so I have to assume it's a state of denial.
In other words, there are groups of developers and executives (and investor-money) with a collective motiviation to desperately shut their eyes and pray that Bad Thing simply won't happen to their product, at least not before someone else invents a revolutionary Silver Bullet that makes it go away.
[0] "Those who cannot remember the past are condemned to repeat it" -- George Santanya
Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos
#199Earlier quoted context omitted.
"How to prompt the model not to leak sensitive data" is not the right discussion to be having. It's a probability model, which means that every conceivable behavior is available in the confines of its code. There is no way to prevent an LLM with access to private information from divulging that information, or from attempting to sabotage systems it has access to. The only solution is to lock every LLM query in the en…
Exactly! Attempting to handle prompt injections by prompting the model (not to leak sensitive data), is like attempting to stop a fire by burning the area around it
It's maybe closer to putting a sign saying "The door is locked" on an unlocked bank vault.
It does nothing to improve security, and if someone manages to get inside and see the sign (i.e. "extract the prompt"), it gives them a strong hint there's interesting stuff behind this door.
Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos
#200why would AI agent not run with user permission instead of essentially root