What Does It Take to Track a Million Cell Phones?

Post by user5994461 » Wed, Jul 19, 2017, 6:14 PM UTC

tinix wrote: post #59

harlanji wrote: post #51

I did the math a while back, don't have the notes at the moment, but scaling an AWS system I built enough to collect 600m points of data each minute and compute on data within 100ms and retain it for a few minutes would run a bit over $10k usd/mo to operate. I operated it at about 3m events/min with a good amount of compute per including ip to geo lookup... Zookeeper would be the only bottleneck in this case assuming…

Using AWS is the problem here, and that's why it's so expensive. You could do this on bare metal WAY faster and more efficiently, and then you own the hardware forever, for the price you paid to do it for a month with a third party. AWS does not scale this way, you can't just throw more resources at a problem and expect to be profitable.

Post by jandrewrogers » Wed, Jul 19, 2017, 6:33 PM UTC

dajohnson89 wrote: post #52

jandrewrogers wrote: post #47

Earlier quoted context omitted.

I have not written about it. Most of the difficulty and complexity, from my perspective, is in the data science and processing required to construct an accurate population model, which requires additional data sources beyond the mobile telemetry. I designed the custom database platforms (easy for me) underneath which supported the online data processing. It isn't that difficult technically, if you have experts doing…

I'm gonna take a wild guess, and say that the NSA has a monopoly on the talent for this field.

Post by jandrewrogers » Wed, Jul 19, 2017, 6:51 PM UTC

rl3 wrote: post #55

dajohnson89 wrote: post #52

Earlier quoted context omitted.

I'm gonna take a wild guess, and say that the NSA has a monopoly on the talent for this field.

The Snowden leaks confirmed NSA has the ability to conduct co-traveler inference.[0] In other words: finding mobile devices proximate to a targeted mobile device, based on similar vectors. Perhaps even making associations in absence of targeting via patterns in device proximity over time. It probably gets real interesting when they're trying to distinguish between various modes of transit, such as a city bus, an Uber…

Post by pjc50 » Wed, Jul 19, 2017, 7:20 PM UTC

jandrewrogers wrote: post #65

rl3 wrote: post #55

Earlier quoted context omitted.

The Snowden leaks confirmed NSA has the ability to conduct co-traveler inference.[0] In other words: finding mobile devices proximate to a targeted mobile device, based on similar vectors. Perhaps even making associations in absence of targeting via patterns in device proximity over time. It probably gets real interesting when they're trying to distinguish between various modes of transit, such as a city bus, an Uber…

Graph reconstruction from space-time event data goes far beyond the above in terms of capability. You can infer relationships between people that never co-travel, infer that people have been places that are not in the event data, etc by stitching together large numbers of orthogonal event streams over long periods of time. It is straightforward to distinguish between various modes of transit analytically. The "metada…

Post by awkwarddaturtle » Wed, Jul 19, 2017, 8:30 PM UTC

jandrewrogers wrote: post #47

frandroid wrote: post #43

Earlier quoted context omitted.

Have you posted about this at length somewhere? If not, care to elaborate on what it took to design this system?

I have not written about it. Most of the difficulty and complexity, from my perspective, is in the data science and processing required to construct an accurate population model, which requires additional data sources beyond the mobile telemetry. I designed the custom database platforms (easy for me) underneath which supported the online data processing. It isn't that difficult technically, if you have experts doing…