Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Attacks against machine learning – an overview

elie.net

Post reply on HN
Print view View original

51–60 of 66 posts

Re: Attacks against machine learning – an overview

#51

Post by theapemachine »

bhnmmhmd wrote: post #3

It'd be great if there was a service that you could sign up for, which would "deceive" Facebook, Twitter, and other social media websites by producing false information about you. For example, if I don't want FB to know what movies I'm interested in, how about liking "random" movie pages on FB? If I don't want FB to know about my political orientations, how about run with the hare and hunt with the hounds?

It seems unlikely that this would be 100% effective though. I am pretty sure that companies like this use data from your immediate social sphere as well to make pretty relevant assumptions about you. For instance, when I buy a product, certain others of my friends will see this same product promoted to them. My friend once looked something up on Facebook, while we were in the movie theater (before the movie started ;)) and sure enough I was getting the ads for that same event as soon as I got home and looked at my phone. Of course, on a more massive scale, this could theoretically work.

Re: Attacks against machine learning – an overview

#52

Post by taurine »

Bromskloss wrote: post #38

> Model stealing techniques, which are used to “steal” (i.e., duplicate) models or recover training data membership via blackbox probing. This can be used, for example, to steal stock market prediction models I would like to hear stories about such attacks on stock market models.

There are not any, since there are no public stock market models worth copying, and no stock market model takes external input. But if they did (you could give a time-series to a model in the cloud, and it would give you predictions) then it would be possible.

Copying models is a problem for cloud-hosted pay-per-prediction image classification, not for constantly retrained stock market models that don't take external input.

Re: Attacks against machine learning – an overview

#53

Post by girvo »

itronitron wrote: post #44
nihonde wrote: post #21

Earlier quoted context omitted.

This is very much the same foundation of disinformation campaigns like strongarm regime propaganda or what is currently termed “fake news”. By attacking credibility itself, anything and nothing are equally valid. The problem with poisoning the well of your own personal data is that it also makes it easier to indict you on false pretenses.

indict you for what?

Whatever they decide to, I believe is the GPs point. When nothing is true, everything can be “true”

Re: Attacks against machine learning – an overview

#54

Post by cinquemb »

girvo wrote: post #53
itronitron wrote: post #44

Earlier quoted context omitted.

indict you for what?

Whatever they decide to, I believe is the GPs point. When nothing is true, everything can be “true”

However, if people are already doing something that is deemed illegal or not allowed by any given jurisdiction/site, making the pool of people to theoretically go after/ban order of magnitude(s) larger seems ideal.

Re: Attacks against machine learning – an overview

#55

Post by sacado2 »

pferde wrote: post #50
itronitron wrote: post #43

Earlier quoted context omitted.

In order to wash out the signal, all the service would need to do is 'like everything'. In addition to masking your interests it would also grind their algorithms to a halt if enough people did that. A lot of these algorithms gain performance due to the sparsity of the data, so if everything became connected it would negatively impact the performance of their algorithms. Anyone know how to get, or compile, a list of…

Until you get into problems (legal or personal, doesn't matter) for "liking" stuff related to child porn, terrorist propaganda or, I don't know, scientology, without even knowing about it, because it was done on your behalf by this "like automaton".

Exactly. In France people have been convicted because they "liked" illegal opinions. As if the fact that such a thing as an illegal opinion exists was not enough of a problem, it's been decided by justice that the semantic of a "like" was "I make this opinion mine".

Re: Attacks against machine learning – an overview

#56

Post by sacado2 »

itronitron wrote: post #43
bhnmmhmd wrote: post #3

It'd be great if there was a service that you could sign up for, which would "deceive" Facebook, Twitter, and other social media websites by producing false information about you. For example, if I don't want FB to know what movies I'm interested in, how about liking "random" movie pages on FB? If I don't want FB to know about my political orientations, how about run with the hare and hunt with the hounds?

In order to wash out the signal, all the service would need to do is 'like everything'. In addition to masking your interests it would also grind their algorithms to a halt if enough people did that. A lot of these algorithms gain performance due to the sparsity of the data, so if everything became connected it would negatively impact the performance of their algorithms. Anyone know how to get, or compile, a list of…

If you like everything, or if you say "my birthday is the 32nd of February", the algorithm can detect you're trying to defeat the system, and ignore you: you become a known unknown.

But if you start to like random things, or if you say "my birthdate is the 2nd of March" while it's not, you become an unknown unknown, and the algorithm must start to reason with your wrong data.

Re: Attacks against machine learning – an overview

#57

Post by sacado2 »

spunker540 wrote: post #24
bhnmmhmd wrote: post #3

It'd be great if there was a service that you could sign up for, which would "deceive" Facebook, Twitter, and other social media websites by producing false information about you. For example, if I don't want FB to know what movies I'm interested in, how about liking "random" movie pages on FB? If I don't want FB to know about my political orientations, how about run with the hare and hunt with the hounds?

How about you don’t sign up for Facebook if you don’t want them to know anything about you? I don’t really see the point of this deception.

Most of your friends, family, coworkers, etc. have your e-mail or your phone number (or both). Some (most) of them use facebook, and a significative portion of them share those information with facebook, so even if you never subscribe to the service, they know a few things about you (your name, phone number, e-mail address and some of your acquaintances), and they know they don't know anything else about you (which is some sort of information too).

Re: Attacks against machine learning – an overview

#58

Post by robertk »

tzahola wrote: post #13
joewee wrote: post #12

Earlier quoted context omitted.

Very well said, never thought about it in this way. It also nullifies a lot of propriety risk scoring models, like credit scores. I wonder what research is done around this for automated trading systems? I see an “attacker” that creates models who’s only purpose is to force another financial institution to make unprofitable trades based on reverse engineering the other traders trading modes. Eventually, if not alread…

>Eventually, if not already happening, trading becomes machines attacking other machines. Welcome to high frequency trading. You’re a bit late to the party though (around 15 years).

I believe you are wrong, if I am to believe my trader friend to whom I showed this thread and answered:

“Trivially true in the 'of course does behaviour of others matter' sense and in the 'could my actions influence others'. Not necessarily operational though.

Fine line from there to 'spoofing' (== placing trades solely with intent of engaging others to trade at price level) -- with is VERY EXPLICITLY not allowed and for which you can get fined and go to jail. Recall the case of that poor SOB out of London who was made a poster boy for the flash crash?”

It seems there is regulation against this.

Re: Attacks against machine learning – an overview

#59

Post by pferde »

sacado2 wrote: post #55
pferde wrote: post #50

Earlier quoted context omitted.

Until you get into problems (legal or personal, doesn't matter) for "liking" stuff related to child porn, terrorist propaganda or, I don't know, scientology, without even knowing about it, because it was done on your behalf by this "like automaton".

Exactly. In France people have been convicted because they "liked" illegal opinions. As if the fact that such a thing as an illegal opinion exists was not enough of a problem, it's been decided by justice that the semantic of a "like" was "I make this opinion mine".

Can you please share some links on this? All I could find was a similar case in Thailand: https://www.theguardian.com/world/2015/dec/10/thai-man-arres...

Re: Attacks against machine learning – an overview

#60

Post by Bromskloss »

taurine wrote: post #52
Bromskloss wrote: post #38

> Model stealing techniques, which are used to “steal” (i.e., duplicate) models or recover training data membership via blackbox probing. This can be used, for example, to steal stock market prediction models I would like to hear stories about such attacks on stock market models.

There are not any, since there are no public stock market models worth copying, and no stock market model takes external input. But if they did (you could give a time-series to a model in the cloud, and it would give you predictions) then it would be possible. Copying models is a problem for cloud-hosted pay-per-prediction image classification, not for constantly retrained stock market models that don't take external…

I thought it would be about observing the behaviour of a system that is trading on the market. The input to the system would consist, for example, of other people's trades.
Post reply on HN