We hacked Gemini's Python sandbox and leaked its source code (at least some)
1–10 of 151 posts
Re: We hacked Gemini's Python sandbox and leaked its source code (at least some)
#2> However, the build pipeline for compiling the sandbox binary included an automated step that adds security proto files to a binary whenever it detects that the binary might need them to enforce internal rules. In this particular case, that step wasn’t necessary, resulting in the unintended inclusion of highly confidential internal protos in the wild !
Protobufs aren't really these super secret hyper-proprietary things they seem to make them out to be in this breathless article.
Re: We hacked Gemini's Python sandbox and leaked its source code (at least some)
#3> However, the build pipeline for compiling the sandbox binary included an automated step that adds security proto files to a binary whenever it detects that the binary might need them to enforce internal rules. In this particular case, that step wasn’t necessary, resulting in the unintended inclusion of highly confidential internal protos in the wild ! Protobufs aren't really these super secret hyper-proprietary thi…
Yeah, this is honestly super interesting as a journey, but not as a destination. The framing takes away from how cool the work really is.
Re: We hacked Gemini's Python sandbox and leaked its source code (at least some)
#4> However, the build pipeline for compiling the sandbox binary included an automated step that adds security proto files to a binary whenever it detects that the binary might need them to enforce internal rules. In this particular case, that step wasn’t necessary, resulting in the unintended inclusion of highly confidential internal protos in the wild ! Protobufs aren't really these super secret hyper-proprietary thi…
Yup, there’s no reason to believe that the proto files (which are definitions rather than data) are any more confidential than the Gemini source code itself.
Re: We hacked Gemini's Python sandbox and leaked its source code (at least some)
#5That's the system I work on! Please feel free to ask any questions. All opinions are my own and do not represent those of my employer.
Re: We hacked Gemini's Python sandbox and leaked its source code (at least some)
#6Awww, I was looking forward to seeing some of the leak ;) Oh well. Nice find and breakdown!
Somewhat relatedly, it occurred to me recently just how important issues like prompt injection, etc are for LLMs. I've always brushed them off as unimportant to _me_ since I'm most interested in local LLMs. Who cares if a local LLM is weak to prompt injection or other shenanigans? It's my AI to do with as I please. If anything I want them to be, since it makes it easier to jailbreak them.
Then Operator and Deep Research came out and it finally made sense to me. When we finally have our own AI Agents running locally doing jobs for us, they're going to encounter random internet content. And the AI Agent obviously needs to read that content, or view the images. And if it's doing that, then it's vulnerable to prompt injection by third party.
Which, yeah, duh, stupid me. But ... is also a really fascinating idea to consider. A future where people have personal AIs, and those AIs can get hacked by reading the wrong thing from the wrong backalley of the internet, and suddenly they are taken over by a mind virus of sorts. What a wild future.
Re: We hacked Gemini's Python sandbox and leaked its source code (at least some)
#7> However, the build pipeline for compiling the sandbox binary included an automated step that adds security proto files to a binary whenever it detects that the binary might need them to enforce internal rules. In this particular case, that step wasn’t necessary, resulting in the unintended inclusion of highly confidential internal protos in the wild ! Protobufs aren't really these super secret hyper-proprietary thi…
No, but having the names to the fields, directly from Google, is very helpful for further understanding what's available from within the sandbox.
Re: We hacked Gemini's Python sandbox and leaked its source code (at least some)
#8> However, the build pipeline for compiling the sandbox binary included an automated step that adds security proto files to a binary whenever it detects that the binary might need them to enforce internal rules. In this particular case, that step wasn’t necessary, resulting in the unintended inclusion of highly confidential internal protos in the wild ! Protobufs aren't really these super secret hyper-proprietary thi…
Yes, there's a lot of internal protos from Google that are leaked on the internet. If I recall correctly, it was a hacker News comment that linked to it.
Edit: I don't know why the parent comment was flagged. It is entirely accurate.
Re: We hacked Gemini's Python sandbox and leaked its source code (at least some)
#9Funny enough while "We hacked Google's AI" is going to get the clicks, in reality they hacked the one part of Gemini that was NOT the LLM (a sandbox environment meant to run untrusted user-provided code).
And "leaked its source code" is straight up click bait.
Re: We hacked Gemini's Python sandbox and leaked its source code (at least some)
#10That's the system I work on! Please feel free to ask any questions. All opinions are my own and do not represent those of my employer.
Does anyone at Google care that you're trying to replace Assistant with this in the next few months and it can't set a timer yet?
(I mean it will tell you it's set a timer but it doesn't talk to the native clock app so nothing ever goes off if you navigate away from the window.)