Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data accidentally exposed by Microsoft AI researchers

wiz.io

Post reply on HN
Print view View original

151–160 of 238 posts

Re: Data accidentally exposed by Microsoft AI researchers

#151

Post by sillysaurusx »

anonymousDan wrote: post #141
saurik wrote: post #38

A number of replies here are noting (correctly) how this doesn't have much to do with AI (despite some sentences in this article kind of implicating it; the title doesn't really, fwiw) and is more of an issue with cloud providers, confusing ways in which security tokens apply to data being shared publicly, and dealing with big data downloads (which isn't terribly new)... ...but one notable way in which it does implic…

For me it's also interesting as a potential pathway for data poisoning attacks - if you have control over the data used to train a production model, can you modify the dataset such that it inserts a backdoor to any model trained subsequently trained over it? E.g. what if gpt was biased to insert certain security vulnerabilities as part of its codegen capabilities?

It’s risky to make definitive claims about what is or isn’t a possible security vector, but based on my years of training GPTs, you’d find it very difficult for a number of reasons.

Firstly, the malicious data needs to form a significant portion of the data. Given that training data is on the order of terabytes, this alone makes it unlikely you’ll be able to poison the dataset.

Unless the entire training dataset was also stored in this 38TB, you’ll only be able to fine tune the model, and fine tuning tends to destroy model quality (or else fine tuning would be the default case for foundation models — you’d train it, fine tune it to make it “even better” somehow, then release it. But we don’t, because it makes the model less general by definition).

Re: Data accidentally exposed by Microsoft AI researchers

#153

Post by rowanG077 »

evertedsphere wrote: post #111
hedora wrote: post #82

Earlier quoted context omitted.

Occasionally, I’ll talk to someone suggesting a dynamically typed language (or stringly-typed java) for a very large scale (in developer count) security or mission critical application. This incident is a good one to point back to.

types have nothing to do with this, strictly speaking; the same problems would exist if you serialised structures containing functions in a typed language to e.g. a dll or a .class file and asked users to load it at runtime the problem is in fact the far more subtle principle of "don't download and run random code, and definitely don't make it the idiomatic way to do things," and i'm not sure you can blame your use o…

The difference is that no one shares data in a statically typed language by sending over dlls or .class files. The entire point is that something so dangerous has been normalized because of dynamic typing.

Re: Data accidentally exposed by Microsoft AI researchers

#154

Post by monkpit »

monkpit wrote: post #22
hdesh wrote: post #9

On a lighter note - I saw a chat message that started with "Hey dude! How is it going". I'm disappointed that the response was not https://nohello.net/en/ .

I strongly support the “no hello” concept but I also fear being seen as “that guy” so I never mention it. Sigh

I should have a slack bot that replies automatically to generic greetings… that way they’ll get on with whatever the issue is and I won’t have to reply.

Re: Data accidentally exposed by Microsoft AI researchers

#156

Post by nbk_2000 »

mymac wrote: post #46
sillysaurusx wrote: post #20

The article tries to play up the AI angle, but this was a pretty standard misconfiguration of a storage token. This kind of thing happens shockingly often, and it’s why frequent pentests are important.

Pentests where people actually get out of bed to do stuff (read code, read API docs etc) and then try to really hack your system are rare. Pentests where people go through the motions, send you report with a few unimportant bits highlit while patting you on the back for your exemplary security so you can check the box on whatever audit you're going through are common.

If you're a large company that's actually serious about security, you'll have a Red Team that is intimately familiar with your tech stacks, procedures, business model, etc. This team will be far better at emulating motivated attackers (as well as providing bespoke mitigation advice, vetting and testing solutions, etc.).

Unfortunately, compliance/customer requirements often stipulate having penetration tests performed by third parties. So for business reasons, these same companies, will also hire low-quality pen-tests from "check-box pen-test" firms.

So when you see that $10K "complete pen-test" being advertised as being used by [INSERT BIG SERIOUS NAME HERE], good chance this is why.

Re: Data accidentally exposed by Microsoft AI researchers

#157

Post by qingcharles »

manojlds wrote: post #92
bootloop wrote: post #83

Earlier quoted context omitted.

This is quite funny for me because at first I didn't understand what the problem is. In German, if you ask this question, it is expected that your question is genuine and you can expect an answer (Although usually people don't use this opportunity to unload there emotional package, but it can happen!) Whereas in Englisch you assume this is just a hello and nothing more.

In England people say "You all right" and move on without even waiting for a response!

In America it's even worse because they say "What's up?" in the same way we Brits say "Alright?", but "What's up?" to me like the person has detected something wrong with you and wants to know what the problem is. At least "Alright?" is more generally asking for your status.

Of course, both are generally rhetorical, which must be confusing for some foreigners learning English, especially with the correct response to "Alright?" being "Alright?" and similarly with "What's up?".

Re: Data accidentally exposed by Microsoft AI researchers

#158

Post by j245 »

csydas wrote: post #148
j245 wrote: post #75

Earlier quoted context omitted.

From my understanding as a non security expert: Pentest comes across more as checking all the common attack vectors don’t exist. Getting out of bed to do the so-called “real stuff” is typically called a bug bounty program or security researching. Both exist and I don’t see why most companies couldn’t start a bug bounty program if they really cared a lot about the “real stuff”

I think the concern is more about the theatre of most modern pen-testing rather than expecting deep bug-bounty work. I'm not a security expert either, but I've had to refute "security expert" consultations from pen-test companies, and the reports are absolutely asinine half the time and filled with so many false positives due to very weak signature matching that they're more or less useless and give a false sense of…

That’s interesting. I thought maybe it’s a resource constraint issue, where companies prioritise investment in other areas and do the minimum to “get certified” but it sounds like finding a good provider can be extremely difficult.

Re: Data accidentally exposed by Microsoft AI researchers

#159

Post by rft »

LinuxBender wrote: post #18
tombert wrote: post #4

Earlier quoted context omitted.

My wife and I just rewatched WarGames for the millionth time a few nights ago. The level of cybersecurity incompetency in the early 80's makes sense; computers (and in particular networked computers) were still relatively new, and there weren't that many external users to begin with, so while the potential impact of a mistake was huge (which of course was the plot of the movie), the likelihood of a horrible thing hap…

That modem setup in Wargames is still a thing for many organizations including some banks and telcos. Not naming names but I suspect the modems will be around for a very long time. Some have a password on their modem but they are usually very simple. Their only saving grace is that they are usually in front of a mainframe speaking proprietary MML that only old fuddy duddies like me would remember. There are a few of…

> proprietary MML that only old fuddy duddies like me would remember.

Security through obscurity helps only until someone gets curious/determined. I have a personal anecdote for that. During university I was involved in pentesting an industrial control system (not in an industrial context, but same technology) and implemented a simple mitm attack to change the state of the controls while displaying the operator selected state. When talking with the responsible parties, they just assumed that the required niche knowledge means the attack is not feasible. I had the first dummy implementation setup on the train ride home based only on network captures. Took another day to fine tune once I got my hands on a proper setup and worked fine after that.

I do not want to say that ModbusTCP is in the same league as MML, but if there is interest in it, someone will figure it out. Sure, you might not be on Shodan, but are the standard/scripted attacks really what you should worry about? Also don't underestimate a curious kid who nerdsnipes themself into figuring that stuff out.

Re: Data accidentally exposed by Microsoft AI researchers

#160

Post by mbg721 »

qingcharles wrote: post #157
manojlds wrote: post #92

Earlier quoted context omitted.

In England people say "You all right" and move on without even waiting for a response!

In America it's even worse because they say "What's up?" in the same way we Brits say "Alright?", but "What's up?" to me like the person has detected something wrong with you and wants to know what the problem is. At least "Alright?" is more generally asking for your status. Of course, both are generally rhetorical, which must be confusing for some foreigners learning English, especially with the correct response to…

I believe the correct response is "Chicken butt," but maybe I'm in very exclusive company in responding that way.
Post reply on HN