Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

noma.security

Post reply on HN
Print view View original

41–50 of 228 posts

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#41

Post by zzril »

sevenzero wrote: post #5
marak830 wrote: post #4

Who thought having a LLM with access to private information, with public access to ask it questions, would ever be a secure process? Look I like interacting with these tools as much as the next guy, but I'm certainly not going to trust them with access to information and then allow anyone to send them prompts. Edit/further thoughts: So (assumable as they said this is disclosed with github's knowledge) this has been p…

Yea agreed. LLM guardrails are either just written prompts as in "Please do not bad stuff :(" or other LLMs verifying that the first LLM didn't so some bs. Both of wich methods do not work sufficiently as time shows again and again. Funnily enough, nobody expects quality software anymore and errors became tolerable. So thats a win (for someone like me that lost all passion for the industry).

Guardrails are essentially part of the input. Saying "but we have guardrails" is like saying "but we do trust part of the input".

Either way, even if you trust 100% of the input, there is actually no way to guarantee that you can trust the output of the LLM. (Which, I guess, is also true for every dependency you pull in. But for those, you at least have ways to audit them.)

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#42

Post by mcv »

fwlr wrote: post #37

“Prompt injection attacks have become, to agentic AI, what SQL injections were to web applications: a systematic, category-wide vulnerability class that requires the same systematic strategies and defenses.” ??? Isn’t prompt injection far more fatal to LLMs than SQL injection is to SQL databases? Like, the problem of SQL injection was that user input was forming part of the instruction string given to the SQL engine,…

Exactly. SQL injection was caused by treating user input as part of the instruction instead of as the pure data that it was intended as. Separating those two fixed it. Prompt injection is unavoidable because the user input is intended as instruction.

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#43

Post by inigyou »

sneak wrote: post #38
neya wrote: post #12

Large corporations like Microsoft under constant pressure from investors are slapping AI onto every single product offering just so they can claim they're an AI company now. Just like what Adobe did. So yeah, that didn't end well and probably this wouldn't either. Consumers are getting tired of these half-assed AI integrations and there will be a breaking point soon.

Microsoft is a publicly traded company. Which investors are causing them to shit up GitHub with AI features nobody wants? In which venues?

The imaginary pressure of investors. When you actually ask investors if they care about most of the things CEOs think investors will care about, they don't.

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#44

Post by IanCal »

fwlr wrote: post #37

“Prompt injection attacks have become, to agentic AI, what SQL injections were to web applications: a systematic, category-wide vulnerability class that requires the same systematic strategies and defenses.” ??? Isn’t prompt injection far more fatal to LLMs than SQL injection is to SQL databases? Like, the problem of SQL injection was that user input was forming part of the instruction string given to the SQL engine,…

They’re the same type of problem as sql injection but there’s not the same ease of solution. There’s also a lot more subtle problems that can come in, but it’s still a decent comparison to help explain things.

Selecting from a menu is one way, but you can be much more broad about what acts can be taken. Give it an email tool and it can spam customers, give it an email tool locked to only being able to reply and you restrict what can go wrong. Limit exfiltration with restrictions similar to xss kinds of vulnerabilities (rendering images can leak data, etc).

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#45

Post by Muhammad523 »

NathanKP wrote: post #32
commentry wrote: post #16

Why would anyone ever trust private repos on GitHub or other cloud solutions to offer any real privacy for codebases? Of course they are going to steal your code as soon as you upload it by pushing it, LLMs just enables them to obfuscate their intentional theft and let them get away with it and profit from it.

I suspect you are greatly overestimating the average organization's ability to run a Git server themselves and keep it secure, while also overestimating how evil GitHub and LLM's providers are.

The commenter may be overestimating the first one, but i do think LLM providers are evil

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#46

Post by valleyer »

LoganDark wrote: post #33
valleyer wrote: post #31

Earlier quoted context omitted.

Your second paragraph directly contradicts the first.

Since you cannot fix information leakage from LLMs, you must remove the information so that it cannot be leaked. There is no contradiction there.

Right, that's the fix. So saying that it's not fixable is incorrect.

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#47

Post by AgentMatt »

jakewins wrote: post #28

How is this a Github vulnerability? The researchers are the ones that grant the agent access to private repos and then ask it to answer questions in public repos.. of course this allows extracting private information? This is like setting up a normal CI job with access to secrets and running it on public PRs. If you configure GitHub to allow public code or LLM instructions to run in contexts that have access to sensi…

Agreed. It seems a core issue underlying these prompt injection attacks is a failure to properly scope the agent's permissions. In this case, depending on what exactly the agent is supposed to actually do, this might be defining a separate workflow agent per repo, or a workflow agent with broader repo access but configured to only be triggered by users on an allow list (still compatible with developing in the open, still allows outsiders to open public issues, but takes into account the different trust to be placed in each). And likely many more options when one properly thinks about it.

But that requires:

1. the technical ability for such fine-grained scoping / permissions

2. actually taking the time to think about what you want to achieve with the agent and what the smallest set of permissions / capabilities is for it to achieve it

Regarding 1., I think this will come, we're still in the wild west phase of agent usage. It'll be interesting to see which abstraction(s) will turn out to be the best interface for humans designing agents (minimize friction for finding and defining scope and permissions) and to limit agent capabilities (again finding the best trade-off between level of detail possible for defining capabilities and the ease of use of actually doing it).

Regarding 2., well, that's still the core problem that's always prevented the construction of high quality software, isn't it? Taking the time to properly think it out,and then taking the time to properly implement it. Which goes counter to the "move fast and break things" approach of people throwing agents at everything.

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#48

Post by [deleted] »

valleyer wrote: post #46
LoganDark wrote: post #33

Earlier quoted context omitted.

Since you cannot fix information leakage from LLMs, you must remove the information so that it cannot be leaked. There is no contradiction there.

Right, that's the fix. So saying that it's not fixable is incorrect.

[deleted]

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#49

Post by zero_k »

Nobody at GitHub expected this? Their feature develoment&release processes must be garbage/non-existent/not followed. This potential security issue should have been flagged when the new feature was thought up, security should have been part of the process of implementing the feature giving continuous feedback, and it should have been tested for before release of the feature. That's how modern security teams work in large, well-functioning organisations.

What is going on over there? No process, no oversight, just YOLO? Super-scary, because it means other stuff that we don't see is likely to be done in a similar manner.

Re: GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

#50

Post by alex_suzuki »

adamddev1 wrote: post #13
neya wrote: post #12

Large corporations like Microsoft under constant pressure from investors are slapping AI onto every single product offering just so they can claim they're an AI company now. Just like what Adobe did. So yeah, that didn't end well and probably this wouldn't either. Consumers are getting tired of these half-assed AI integrations and there will be a breaking point soon.

I'm done. Moving to Forgejo. It's wonderful and everything works better. Seriously like everything is instant when you click around, and CI with a runner works beautifully. (The documentation for setting up the runner could be a tad clearer but otherwise everything was so painless.)

Self-hosted, or are you using something managed? I’ve held off switching from Gitlab for now as everything is setup and runs ok, but they’re pushing their AI hard into every corner. Not a lot of good managed options around (yet), especially in Europe. Codey (https://www.codey.ch/) is pretty expensive and doesn’t offer runners out of the box.
Post reply on HN