Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data accidentally exposed by Microsoft AI researchers

wiz.io

Post reply on HN
Print view View original

171–180 of 238 posts

Re: Data accidentally exposed by Microsoft AI researchers

#171

Post by romanows »

dheera wrote: post #115
saurik wrote: post #38

A number of replies here are noting (correctly) how this doesn't have much to do with AI (despite some sentences in this article kind of implicating it; the title doesn't really, fwiw) and is more of an issue with cloud providers, confusing ways in which security tokens apply to data being shared publicly, and dealing with big data downloads (which isn't terribly new)... ...but one notable way in which it does implic…

Many people are also unaware that json is way, way, way faster than Python pickles, and human-editing-friendly. Not that you'd use it for neural net weights, but I see people use Python pickles all the time for things that json would have worked perfectly well.

Are you sure json is faster than pickle in recent python versions? That's not intuitive to me and search result blurbs seem to indicate the opposite.

Re: Data accidentally exposed by Microsoft AI researchers

#172

Post by evntdrvn »

mymac wrote: post #46
sillysaurusx wrote: post #20

The article tries to play up the AI angle, but this was a pretty standard misconfiguration of a storage token. This kind of thing happens shockingly often, and it’s why frequent pentests are important.

Pentests where people actually get out of bed to do stuff (read code, read API docs etc) and then try to really hack your system are rare. Pentests where people go through the motions, send you report with a few unimportant bits highlit while patting you on the back for your exemplary security so you can check the box on whatever audit you're going through are common.

what I always want to know when people talk about this is "what reputable companies can I actually pay to do a real pentest (without costing hundreds of thousands of dollars)."

Re: Data accidentally exposed by Microsoft AI researchers

#173

Post by btilly »

sillysaurusx wrote: post #170
btilly wrote: post #163

Earlier quoted context omitted.

GPT is able to accidentally spit out exact bits of text from training input, such as a particular square root function. What fraction of the training data needed to be that text?

If the question is "Would it be possible to get GPT to try to add backdoors to code examples by poisoning the training data?" my answer would be no. The sheer quantity of training data means that even with GPT-4's assistance in generating code examples that match the format of the original training data, you wouldn't be able to inject enough poison to change the model's behavior by much. Remember, once the model is t…

As I commented elsewhere, GPT is such a target rich security environment that it is hard to know why you would bother with this. On the other hand, advanced persistent attackers (eg the NSA) have a pretty good imagination. I could see them having both motive and means to go out of their way to achieve a particular result.

On human checks, http://www.underhanded-c.org/ demonstrates that it would be possible to inject content that will pass that.

Re: Data accidentally exposed by Microsoft AI researchers

#174

Post by pnt12 »

evntdrvn wrote: post #172
mymac wrote: post #46

Earlier quoted context omitted.

Pentests where people actually get out of bed to do stuff (read code, read API docs etc) and then try to really hack your system are rare. Pentests where people go through the motions, send you report with a few unimportant bits highlit while patting you on the back for your exemplary security so you can check the box on whatever audit you're going through are common.

what I always want to know when people talk about this is "what reputable companies can I actually pay to do a real pentest (without costing hundreds of thousands of dollars)."

I think hiring a security specialist is the way to go.

Re: Data accidentally exposed by Microsoft AI researchers

#175

Post by antonjs »

sillysaurusx wrote: post #170
btilly wrote: post #163

Earlier quoted context omitted.

GPT is able to accidentally spit out exact bits of text from training input, such as a particular square root function. What fraction of the training data needed to be that text?

If the question is "Would it be possible to get GPT to try to add backdoors to code examples by poisoning the training data?" my answer would be no. The sheer quantity of training data means that even with GPT-4's assistance in generating code examples that match the format of the original training data, you wouldn't be able to inject enough poison to change the model's behavior by much. Remember, once the model is t…

Makes me wonder if there would be a way to pollute imagenet so a particular image would always match for something like a facial recognition access control system or the like. Maybe adversarial data that would hide particular traffic patterns from an AI enabled IDS would be more plausible and something the NSA might be interested in.

Re: Data accidentally exposed by Microsoft AI researchers

#176

Post by Root_Denied »

sillysaurusx wrote: post #170
btilly wrote: post #163

Earlier quoted context omitted.

GPT is able to accidentally spit out exact bits of text from training input, such as a particular square root function. What fraction of the training data needed to be that text?

If the question is "Would it be possible to get GPT to try to add backdoors to code examples by poisoning the training data?" my answer would be no. The sheer quantity of training data means that even with GPT-4's assistance in generating code examples that match the format of the original training data, you wouldn't be able to inject enough poison to change the model's behavior by much. Remember, once the model is t…

I don't disagree with you on targeted attacks, but if you're creating output at scale then I'd say there's marginally more risk.

It's possible there's some minimum amount of poisoned data (a % or log function of a given dataset size n) that would then translate to generating a vulnerable output in x% of total outputs. If x is low enough to get past fine tuning/regression testing but high enough to still occur within the deployment space, then you've effectively created a new category of supply-chain attack.

There's probably more research that needs to be done into occurrence rate of poisoned data showing up in final output, and that result is likely specific to the AI model and/or version.

Re: Data accidentally exposed by Microsoft AI researchers

#177

Post by sneak »

sillysaurusx wrote: post #20

The article tries to play up the AI angle, but this was a pretty standard misconfiguration of a storage token. This kind of thing happens shockingly often, and it’s why frequent pentests are important.

It was so common that S3 added several features to make it really, really hard to accidentally leave a whole bucket public.

Looks like Azure hasn't done similarly.

Re: Data accidentally exposed by Microsoft AI researchers

#179

Post by sneak »

monkpit wrote: post #22
hdesh wrote: post #9

On a lighter note - I saw a chat message that started with "Hey dude! How is it going". I'm disappointed that the response was not https://nohello.net/en/ .

I strongly support the “no hello” concept but I also fear being seen as “that guy” so I never mention it. Sigh

Be that guy. In the long run it's better to be right then popular.

Re: Data accidentally exposed by Microsoft AI researchers

#180

Post by pradn »

It's not reasonable to expect human security token generation to be perfectly secure all the time. The system needs to be safe overall. The organization should have set an OrgPolicy on this entire project to prevent blanket sharing of auth tokens/credentials like this. Ideally blanket access tokens should be opt-in, not opt-out.

Google banned generation of service account keys for internally-used projects. So an awry JSON file doesn't allow access to Google data/code. This is enforced at the highest level by OrgPolicy. There's a bunch more restrictions, too.

Post reply on HN