I deduplicated 53,000 missing-persons reports from Venezuela's earthquake
1–3 of 3 posts
Re: I deduplicated 53,000 missing-persons reports from Venezuela's earthquake
#2I'm the founder of Tilores, the entity-resolution tool used here - so full disclosure, this is my company's product. This wasn't a paid engagement or a case study. It started because my wife is from Venezuela, and she saw people on social media pointing out that the missing-persons lists had huge numbers of duplicates.
On the data: these are public citizen-lead efforts to crowd-source the names of the missing - hosted on websites and spreadsheets. There is no official verification process behind the individual entries, which is part of why the duplicate problem existed in the first place.
An issue we have now realised is "bad actors" trying to access the data...
Happy to answer anything - methodology, false positives, data handling, whatever.
Re: I deduplicated 53,000 missing-persons reports from Venezuela's earthquake
#3Did you fix the problem with the "María" name? (At least here in Argentina) it's very very common, so most Marías don't use it. So
* María Laura Pérez
* Ma. Laura Pérez
* Laura Pérez
may be the same person. Once I learned that a gal was call "María" like 10 years after knowing her. She prefers to be call by her second name.
I know a guy that had to deduplicate the retirements registry of each province to make a national one, and he has a lot of corner cases that they had to use.
Anyway, sad to read the list has not too many duplications. :(