I did the math a while back, don't have the notes at the moment, but scaling an AWS system I built enough to collect 600m points of data each minute and compute on data within 100ms and retain it for a few minutes would run a bit over $10k usd/mo to operate. I operated it at about 3m events/min with a good amount of compute per including ip to geo lookup... Zookeeper would be the only bottleneck in this case assuming good enough partitioning.
What Does It Take to Track a Million Cell Phones?
51–60 of 88 posts
Re: What Does It Take to Track a Million Cell Phones?
#52Earlier quoted context omitted.
Have you posted about this at length somewhere? If not, care to elaborate on what it took to design this system?
I have not written about it. Most of the difficulty and complexity, from my perspective, is in the data science and processing required to construct an accurate population model, which requires additional data sources beyond the mobile telemetry. I designed the custom database platforms (easy for me) underneath which supported the online data processing. It isn't that difficult technically, if you have experts doing…
I'm gonna take a wild guess, and say that the NSA has a monopoly on the talent for this field.
Re: What Does It Take to Track a Million Cell Phones?
#53Nothing worthwhile ever takes an intern and six months. Ever.
Re: What Does It Take to Track a Million Cell Phones?
#54If my phone is powered off can i be tracked? What if i remove the battery?
You can put the phone inside a RF radio blocking bag.
The bag is only going to attenuate any signal, not block it (block would imply infinite attenuation). Whether or not the attenuation provided is sufficient to prevent an adversary from receiving the signal I'm not sure. I definitely wouldn't bet my life on it. I'd want a pretty thick metal box with proper seam gaskets as a minimum.
Re: What Does It Take to Track a Million Cell Phones?
#55Earlier quoted context omitted.
I have not written about it. Most of the difficulty and complexity, from my perspective, is in the data science and processing required to construct an accurate population model, which requires additional data sources beyond the mobile telemetry. I designed the custom database platforms (easy for me) underneath which supported the online data processing. It isn't that difficult technically, if you have experts doing…
I'm gonna take a wild guess, and say that the NSA has a monopoly on the talent for this field.
The Snowden leaks confirmed NSA has the ability to conduct co-traveler inference.[0] In other words: finding mobile devices proximate to a targeted mobile device, based on similar vectors. Perhaps even making associations in absence of targeting via patterns in device proximity over time.
It probably gets real interesting when they're trying to distinguish between various modes of transit, such as a city bus, an Uber/Lyft/taxi, and a private vehicle not participating in rideshares. Of those examples, the latter would suggest the highest degree of association.
Pure speculation, but I wouldn't doubt they take a peek at ridesharing data for co-traveler inference purposes. Knowing if a rideshare driver is on or off the clock would be incredibly valuable information in that context.
[0] https://www.washingtonpost.com/apps/g/page/world/how-the-nsa...
Re: What Does It Take to Track a Million Cell Phones?
#56Earlier quoted context omitted.
I have not written about it. Most of the difficulty and complexity, from my perspective, is in the data science and processing required to construct an accurate population model, which requires additional data sources beyond the mobile telemetry. I designed the custom database platforms (easy for me) underneath which supported the online data processing. It isn't that difficult technically, if you have experts doing…
I'm gonna take a wild guess, and say that the NSA has a monopoly on the talent for this field.
[deleted]
Re: What Does It Take to Track a Million Cell Phones?
#57Earlier quoted context omitted.
I have not written about it. Most of the difficulty and complexity, from my perspective, is in the data science and processing required to construct an accurate population model, which requires additional data sources beyond the mobile telemetry. I designed the custom database platforms (easy for me) underneath which supported the online data processing. It isn't that difficult technically, if you have experts doing…
I'm gonna take a wild guess, and say that the NSA has a monopoly on the talent for this field.
Mapping is wide and common industry, just like web or finance.
The NSA only recruits in the USA. It's a fraction of the talent pool of the planet.
Re: What Does It Take to Track a Million Cell Phones?
#58Earlier quoted context omitted.
Have you posted about this at length somewhere? If not, care to elaborate on what it took to design this system?
I have not written about it. Most of the difficulty and complexity, from my perspective, is in the data science and processing required to construct an accurate population model, which requires additional data sources beyond the mobile telemetry. I designed the custom database platforms (easy for me) underneath which supported the online data processing. It isn't that difficult technically, if you have experts doing…
A great PoC is fairly doable by an intern.
Increasing the precision by tenfold will likely increase the effort by a hundred fold or more. Just because it can be made harder and more expensive doesn't mean it has to be.
At the end of the day, a bit of precision doesn't change the nature of an effective planet scale mass surveillance system.
Re: What Does It Take to Track a Million Cell Phones?
#59I did the math a while back, don't have the notes at the moment, but scaling an AWS system I built enough to collect 600m points of data each minute and compute on data within 100ms and retain it for a few minutes would run a bit over $10k usd/mo to operate. I operated it at about 3m events/min with a good amount of compute per including ip to geo lookup... Zookeeper would be the only bottleneck in this case assuming…
Using AWS is the problem here, and that's why it's so expensive. You could do this on bare metal WAY faster and more efficiently, and then you own the hardware forever, for the price you paid to do it for a month with a third party.
AWS does not scale this way, you can't just throw more resources at a problem and expect to be profitable.
Re: What Does It Take to Track a Million Cell Phones?
#60Inrix, TOMTOM, and a couple other have been providing this data as a product for at least 2 decades. There was an early provider that lead the space, but the name of that company eludes me at the moment, may have been actually purchased by inrix.
Most of those companies focused on 10m+- resolution and focused on path data to build traffic speed data for local news companies.
Only cost a couple million bucks and an extensive partnership agreement to get into the space.
There is a lot of data washing in those agreements, mostly related to preventing reverse identification.
Airsage has taken it to the next level in the more recent past with GPS based anonymized data, but data with EXTENSIVE history. The Airsage product is zip code and smaller resolution and can provide months to years of location history of an anonymous cell phone id.