Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I'm Beating The NSA To The Punch By Spying On Myself

fastcolabs.com

Post reply on HN
Print view View original

11–15 of 15 posts

Re: I'm Beating The NSA To The Punch By Spying On Myself

#11

Post by gabestein »

quchen wrote: post #10
gabestein wrote: post #7

Earlier quoted context omitted.

Thanks. I solemnly swear to not abuse statistics, so I'll play around and update the piece. The goal is to use this as a hook to get the public interested, then take them for the ride as I learn. That's why I tried to be really cautious about pointing out all the reasons why this particular model is bad.

I didn't mean to say you specifically were abusing statistics, it's just very easy to draw dubious conclusions. (Doing this intentionally would be the abuse I was talking about.)

I know, but I still want to take it seriously, and thank you for your criticism.

Re: I'm Beating The NSA To The Punch By Spying On Myself

#12

Post by gabestein »

quchen wrote: post #6
gabestein wrote: post #5

Earlier quoted context omitted.

Thanks for the feedback. As I'm fairly new to all of this, I'd love to hear how I can turn this into a more viable experiment. Also, to defend myself a little: I think I'm at least being responsible by in no way claiming that it's statistically valid, and in fact making the point that it's not, and citing the reasons why, several times in the article. I do stand by my point, however, that this helps the public unders…

> As I'm fairly new to all of this, I'd love to hear how I can turn this into a more viable experiment. The most important thing is getting an estimate of how good your averages are. Try to modify a couple of things, for example how do the 67% change with sample size (verification)? How does the number turn out if you feed it biased data, e.g. only male phone numbers (falsification)?

Thanks to your help, I found a huge, obvious problem that was keeping the accuracy low and remarkable stable no matter what size of training data I tried. Here's the update: http://www.fastcolabs.com/3012908/tracking/im-beating-the-ns...

Re: I'm Beating The NSA To The Punch By Spying On Myself

#13

Post by gabestein »

omonra wrote: post #8

I think the basic flaw here is that author assumes that NSA does data-mining. I personally doubt it. Rather (and here I'm going off on pure speculation myself) I think they can use the system to start with some off-line lead (say one of the Chechen bomber brothers) - and see who he was calling / emailing and then checking if there is anything valuable there. Ie - how in the past FBI would do an investigation off-line…

"As part of K.D.D., an algorithm was applied to the broader data set in efforts to detect patterns of behavior fitting models that had been previously established as being indicative of the activities of a terrorist cell."

http://m.vanityfair.com/online/eichenwald/2013/06/obama-veri...

Re: I'm Beating The NSA To The Punch By Spying On Myself

#14

Post by omonra »

gabestein wrote: post #13
omonra wrote: post #8

I think the basic flaw here is that author assumes that NSA does data-mining. I personally doubt it. Rather (and here I'm going off on pure speculation myself) I think they can use the system to start with some off-line lead (say one of the Chechen bomber brothers) - and see who he was calling / emailing and then checking if there is anything valuable there. Ie - how in the past FBI would do an investigation off-line…

"As part of K.D.D., an algorithm was applied to the broader data set in efforts to detect patterns of behavior fitting models that had been previously established as being indicative of the activities of a terrorist cell." http://m.vanityfair.com/online/eichenwald/2013/06/obama-veri...

Thank you. But the next sentence reads " It is also run against a large set of what are known as “dirty numbers”—telephones linked to terrorists either through American signals intelligence or information provided by foreign services. Even the Libyans under Qaddafi turned over huge stacks of dirty numbers to us."

So that seems to correlate with what I am saying - they have some flag generated offline that they use as a marker to find something online.

Re: I'm Beating The NSA To The Punch By Spying On Myself

#15

Post by gabestein »

omonra wrote: post #14
gabestein wrote: post #13

Earlier quoted context omitted.

"As part of K.D.D., an algorithm was applied to the broader data set in efforts to detect patterns of behavior fitting models that had been previously established as being indicative of the activities of a terrorist cell." http://m.vanityfair.com/online/eichenwald/2013/06/obama-veri...

Thank you. But the next sentence reads " It is also run against a large set of what are known as “dirty numbers”—telephones linked to terrorists either through American signals intelligence or information provided by foreign services. Even the Libyans under Qaddafi turned over huge stacks of dirty numbers to us." So that seems to correlate with what I am saying - they have some flag generated offline that they use as…

Oh, they definitely do both, I'm not disputing that.
Post reply on HN