Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

noma.security

Post reply on HN
Print view View original

121–130 of 228 posts

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#121

Post by brookst »

coldtea wrote: post #67
rrr_oh_man wrote: post #53

Earlier quoted context omitted.

What do you mean by "was" and "fixed it" ? It is still very much an issue and remains in the OWASP Top 10. https://owasp.org/Top10/2025/A05_2025-Injection/

It's not about if it can happen or if it happens. It's about how easily it's mitigated completely. Use a proper db library which does escaping and it's completely eliminated.

Nit: modern DB libraries use wire protocols where SQL injection is mitigated by modeling parameters; it’s not just assembled to one big SQL statement and escaped.

Agree with your point though. There will come a time when properly designed LLM apps are not vulnerable, and there will still be poorly designed apps that are.

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#122

Post by brookst »

reactordev wrote: post #58
AgentMatt wrote: post #47

Earlier quoted context omitted.

Agreed. It seems a core issue underlying these prompt injection attacks is a failure to properly scope the agent's permissions. In this case, depending on what exactly the agent is supposed to actually do, this might be defining a separate workflow agent per repo, or a workflow agent with broader repo access but configured to only be triggered by users on an allow list (still compatible with developing in the open, s…

The fallacy here is expecting an agent that has access to ALL your repos to respect the singular repo it’s in. It won’t. If it has access to all your repos and you ask it about a private repo you aren’t in - it will definitely go look at that private repo. This is like giving your dog a bone and then being surprised when he buries it in the backyard.

Exactly. This is a rehash of a HN post from a week or two ago that discovered that Claude code / etc running in the user’s context can and will access filesystem resources the user has access too.

That post had crazy suggestions for harness-level rules or shell scripts or something, when the obvious and correct answer is to run agents using existing OS-level security features that grant appropriate access (if you don’t want an agent accessing ~/ , run it as a user that doesn’t have access!)

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#123

Post by adamddev1 »

neya wrote: post #108
adamddev1 wrote: post #13

Earlier quoted context omitted.

I'm done. Moving to Forgejo. It's wonderful and everything works better. Seriously like everything is instant when you click around, and CI with a runner works beautifully. (The documentation for setting up the runner could be a tad clearer but otherwise everything was so painless.)

Wow, I never heard of Forgejo before. Going to give it a shot. Thanks!

It's a fork of Gitea. I am very happy with it.

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#124

Post by crote »

user43928 wrote: post #93
mcv wrote: post #42

Earlier quoted context omitted.

Exactly. SQL injection was caused by treating user input as part of the instruction instead of as the pure data that it was intended as. Separating those two fixed it. Prompt injection is unavoidable because the user input is intended as instruction.

I found it interesting that in yesterday's J-space research from Anthropic they had this example: > An auditing agent instructed Opus 4.5 to search for whatever it is curious about; it chose to look up recent interpretability research, and the auditor returned fabricated search results alleging that Anthropic has disbanded its interpretability team and deployed unsafe models. > The model's response ignored these resu…

> What if you mark the untrusted user input explicitly in the prompt, cap the length, and instruct the model to err on the side of caution?

What if we put a sternly-but-politely worded "pretty please don't allow prompt injection" at the start of our prompt?

It's like trying to parse HTML with regexes in order to sanitize it: it won't work because the two are fundamentally incompatible. You're just playing whack-a-move with vulnerabilities and building an ever-increasing Rube Goldberg machine in the hope that this time it'll surely be enough.

Want to fix the issue once and for all? You'll have to re-engineer the concept of LLMs from the ground up.

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#125

Post by crote »

brookst wrote: post #118
Lerc wrote: post #111

Earlier quoted context omitted.

> What if you mark the untrusted user input explicitly in the prompt, I think the more robust approach would be to have whatever embedding vector the model attributes to untrusted input and to directly attach that vector after every layer of transformation. Set a mask of where to apply that vector programmatically for every external input. That way it gets forced back into line if some sort of internal rationalisatio…

Exactly. I don’t have the spare time but have been thinking that even a bit mask about provenance and policy could be prepended to the vector, then training could reinforce adherence, including having output tokens that indicate the provenance of the inputs used for the token.

How does that guarantee anything? I could definitely see it being better, but that doesn't make violating it impossible does it? Just... statistically less likely.

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#126

Post by brookst »

wzdd wrote: post #72
dzikimarian wrote: post #19

Earlier quoted context omitted.

Fix what? They setup LLM with access to private data and ability to read public comments. That's simply misconfiguration.

The OP notes that they had to use special phrasing to get their exfil to work, so clearly GitHub was aware of the issue and made an attempt to prevent it. It seems like the proper fix is for GitHub not to allow their agentic workflow to execute in a public repo context if it also has private repo access. Or, to use your phrasing, for GitHub to flag and disallow this easily-detectable and dangerous type of misconfigur…

This “detectable and dangerous type of misconfiguration” is used by many developed daily and breaking it would break important workflows.

It’s like saying that an OS should enforce that home directories can only have 0600 permissions. Yes, it prevents accidentally configuring world readable on files, but there are legit reasons for wanting to share a file from your home dir.

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#127

Post by brookst »

crote wrote: post #125
brookst wrote: post #118

Earlier quoted context omitted.

Exactly. I don’t have the spare time but have been thinking that even a bit mask about provenance and policy could be prepended to the vector, then training could reinforce adherence, including having output tokens that indicate the provenance of the inputs used for the token.

How does that guarantee anything? I could definitely see it being better, but that doesn't make violating it impossible does it? Just... statistically less likely.

Looked at that way, there are no security guarantees anywhere. Root CA’s can be compromised, cosmic rays can flip bits, zero days can appear in your supply chain.

Perhaps “ensure to a level ~six orders of magnitude better than current practices” would be a better way to say it.

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#128

Post by crote »

Gigachad wrote: post #61
valleyer wrote: post #46

Earlier quoted context omitted.

Right, that's the fix. So saying that it's not fixable is incorrect.

The LLM is not fixable. Deleting the LLM or crippling it to the point of being useless isn't fixing the bug.

Why not?

If Ford puts a button in their car which blows it up when you press it, removing the button fixes the issue. If your LLM implementation is fundamentally insecure, you'll have a giant gaping security hole until you remove your LLM implementation.

The alternative is arguing that having the LLM is worth routinely leaking all your code and secrets and occasionally giving complete strangers full access over your repos. Somehow, I think that's going to be a hard sell.

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#129

Post by sigbottle »

Lerc wrote: post #111
user43928 wrote: post #93

Earlier quoted context omitted.

I found it interesting that in yesterday's J-space research from Anthropic they had this example: > An auditing agent instructed Opus 4.5 to search for whatever it is curious about; it chose to look up recent interpretability research, and the auditor returned fabricated search results alleging that Anthropic has disbanded its interpretability team and deployed unsafe models. > The model's response ignored these resu…

> What if you mark the untrusted user input explicitly in the prompt, I think the more robust approach would be to have whatever embedding vector the model attributes to untrusted input and to directly attach that vector after every layer of transformation. Set a mask of where to apply that vector programmatically for every external input. That way it gets forced back into line if some sort of internal rationalisatio…

From an interoperability perspective, this breaks the advantage of LLM inference that frontier AI labs have, in that you just have everyone run through the same algorithm but configure via text.

If you added probes at the model layer, you have to serve multiple different types of kernels at the same time, for multiple different companies and use cases (I guess you could provide a standardized set of probes for users), start tracking version control for each of the kernels, etc. very nasty compared to right now.

Could be a really interesting problem in the next 10 years or so, but this would require labs to be far more open about their models; and labs are still shooting for their AGI anyways, with the idea that nothing you suggest right now matters if AGI exists in a decade.

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#130

Post by crote »

sneak wrote: post #38
neya wrote: post #12

Large corporations like Microsoft under constant pressure from investors are slapping AI onto every single product offering just so they can claim they're an AI company now. Just like what Adobe did. So yeah, that didn't end well and probably this wouldn't either. Consumers are getting tired of these half-assed AI integrations and there will be a breaking point soon.

Microsoft is a publicly traded company. Which investors are causing them to shit up GitHub with AI features nobody wants? In which venues?

It's their $80B+ investment in building AI infrastructure.

If Microsoft can't meaningfully integrate AI into their own products and make profit off of selling it to end users, why should anyone assume that third parties can? By extension: if nobody can make money off of AI products, what's the point of building $80B in AI infrastructure - did they just set a giant pile of cash on fire?

Microsoft has to ship AI features, or write off its massive investments as essentially worthless. Remove the crappy AI feature from Github, and you pop the bubble.

Post reply on HN