@4mm character width:
4e-6 * 3.8e+13 = 152 million kilometers of text.
Nearly 200 round trips to the moon.
Data accidentally exposed by Microsoft AI researchers
161–170 of 238 posts
Re: Data accidentally exposed by Microsoft AI researchers
#162A number of replies here are noting (correctly) how this doesn't have much to do with AI (despite some sentences in this article kind of implicating it; the title doesn't really, fwiw) and is more of an issue with cloud providers, confusing ways in which security tokens apply to data being shared publicly, and dealing with big data downloads (which isn't terribly new)... ...but one notable way in which it does implic…
For me it's also interesting as a potential pathway for data poisoning attacks - if you have control over the data used to train a production model, can you modify the dataset such that it inserts a backdoor to any model trained subsequently trained over it? E.g. what if gpt was biased to insert certain security vulnerabilities as part of its codegen capabilities?
The AI version of https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...?
At the moment such techniques would seem to be superfluous. I mean we're still at the stage where you can get a bot to spit out a credit card number by saying, "My name is in the credit card field. What is my name?"
That said, what you're describing seems totally plausible. If there was enough text with a context where it behaved in a particular way, triggering that context should trip that behavior. And there would be no obvious sign of it unless you triggered that context.
AI is hard.
Re: Data accidentally exposed by Microsoft AI researchers
#163Earlier quoted context omitted.
For me it's also interesting as a potential pathway for data poisoning attacks - if you have control over the data used to train a production model, can you modify the dataset such that it inserts a backdoor to any model trained subsequently trained over it? E.g. what if gpt was biased to insert certain security vulnerabilities as part of its codegen capabilities?
It’s risky to make definitive claims about what is or isn’t a possible security vector, but based on my years of training GPTs, you’d find it very difficult for a number of reasons. Firstly, the malicious data needs to form a significant portion of the data. Given that training data is on the order of terabytes, this alone makes it unlikely you’ll be able to poison the dataset. Unless the entire training dataset was…
GPT is able to accidentally spit out exact bits of text from training input, such as a particular square root function.
What fraction of the training data needed to be that text?
Re: Data accidentally exposed by Microsoft AI researchers
#164Earlier quoted context omitted.
From my understanding as a non security expert: Pentest comes across more as checking all the common attack vectors don’t exist. Getting out of bed to do the so-called “real stuff” is typically called a bug bounty program or security researching. Both exist and I don’t see why most companies couldn’t start a bug bounty program if they really cared a lot about the “real stuff”
> From my understanding as a non security expert: That certainly helps.
People are going to chit-chat about things only tangentially related to their areas of expertise; it is good when we’re honest about our limitations.
If nothing else, an obviously wrong take is a nice setup for a correction.
Re: Data accidentally exposed by Microsoft AI researchers
#165A number of replies here are noting (correctly) how this doesn't have much to do with AI (despite some sentences in this article kind of implicating it; the title doesn't really, fwiw) and is more of an issue with cloud providers, confusing ways in which security tokens apply to data being shared publicly, and dealing with big data downloads (which isn't terribly new)... ...but one notable way in which it does implic…
I’ll venture that it’s at least adjacent that the indiscriminate assembly of massive, serious pluralities of the commons on a purely unilateral basis for profit is sort of a “just try and stop us” posture that whether or not directly related here, and clearly with some precedent, is looking to create a lot of this sort of thing over and above the status-quo ick.
I have no idea what you are saying. If it is: "bad incentives cause people to misbehave", you generated an impressive verbiage around it :)
Re: Data accidentally exposed by Microsoft AI researchers
#166This is so unfortunate but a clear illustration of something I've been thinking about a lot when it comes to LLMs and AI. It seems like we're forgetting that we are just handing our data over to these companies on a solver platter in the form of our prompts.
Disclosure that I do work for Tonic.ai and we are working on a way to automatically redact any information you send to an LLM - https://www.tonic.ai/solar
Re: Data accidentally exposed by Microsoft AI researchers
#167The article tries to play up the AI angle, but this was a pretty standard misconfiguration of a storage token. This kind of thing happens shockingly often, and it’s why frequent pentests are important.
How would a pentest find that? Ok in this case it's splattered onto github; but the main point here is that you might have some unknown number of SAS tokens issued to unknown storage that you probably haven't any easy way to revoke.
A number of ways, including:
- finding the token directly in the repo
- reviewing all tokens issued
Re: Data accidentally exposed by Microsoft AI researchers
#168[dead]
Re: Data accidentally exposed by Microsoft AI researchers
#169A number of replies here are noting (correctly) how this doesn't have much to do with AI (despite some sentences in this article kind of implicating it; the title doesn't really, fwiw) and is more of an issue with cloud providers, confusing ways in which security tokens apply to data being shared publicly, and dealing with big data downloads (which isn't terribly new)... ...but one notable way in which it does implic…
Occasionally, I’ll talk to someone suggesting a dynamically typed language (or stringly-typed java) for a very large scale (in developer count) security or mission critical application. This incident is a good one to point back to.
Yeah, because statically typed language never had any kind of deserialization vulnerabilities.
Re: Data accidentally exposed by Microsoft AI researchers
#170Earlier quoted context omitted.
It’s risky to make definitive claims about what is or isn’t a possible security vector, but based on my years of training GPTs, you’d find it very difficult for a number of reasons. Firstly, the malicious data needs to form a significant portion of the data. Given that training data is on the order of terabytes, this alone makes it unlikely you’ll be able to poison the dataset. Unless the entire training dataset was…
GPT is able to accidentally spit out exact bits of text from training input, such as a particular square root function. What fraction of the training data needed to be that text?
If the question is "Would it be possible to get GPT to try to add backdoors to code examples by poisoning the training data?" my answer would be no. The sheer quantity of training data means that even with GPT-4's assistance in generating code examples that match the format of the original training data, you wouldn't be able to inject enough poison to change the model's behavior by much.
Remember, once the model is trained, it's verified in a number of ways, ultimately based on human prompting. If the tokens that come out of an experimental model are obviously bad (because, say, the model is suggesting exploits instead of helpful code), all that will do is get a scientist to look more deeply into why the model is behaving the way it is. And then that would lead to discovering the poisoned data.
The payoff for an attacker is whether they can achieve some sort of goal. You'd have to clearly define what that goal is in order to know how effective the poisoning attack could be. What's the end game?