Fuzzy Name Matching in Postgres
info.crunchydata.com
Fuzzy Name Matching in Postgres
1–10 of 22 posts
Re: Fuzzy Name Matching in Postgres
#2Phonex, Soundex and Levenstein distance are great algorithms that hopefully more people will find out about thanks to this great article!
Re: Fuzzy Name Matching in Postgres
#3the biggest issue is postgres not having tf-idf or BM25 style relevance algorithms.
that would have been the right fit for fuzzy name matching.
Re: Fuzzy Name Matching in Postgres
#4Soundex is cool, we’re using it for a game app I'm working on.
It’s a trivia game app based on a TV show. In the TV show candidates try to guess the right answer just by saying. How it’s written is not relevant (in order to score points). However in the app users need to write it down (voice support coming soon) and can make spelling mistakes. For a lot of stuff the Levenshtein distance just doesn’t cut it.
Re: Fuzzy Name Matching in Postgres
#5Very informative and well written, it will be useful to avoid using Solr or ES for simple needs.
Re: Fuzzy Name Matching in Postgres
#6The biggest issue I can see with this approach is what happens if I’ve made a typo which changes the phonetics of the search input so that it gets excluded by the soundex prefilter, eg typing “Harrinbton” instead of “Harrington”
Re: Fuzzy Name Matching in Postgres
#7the biggest issue is postgres not having tf-idf or BM25 style relevance algorithms. that would have been the right fit for fuzzy name matching.
How would tf-idf help fuzzy match names? You would first need to decide the term youre looking for is the term you've indexed, which is what soundex &c would do.
Re: Fuzzy Name Matching in Postgres
#8Surprised there's no mention of pg_trgm for trigram-based fuzzy match.
I guess for English names, Soundex is fairly good.
Re: Fuzzy Name Matching in Postgres
#9the biggest issue is postgres not having tf-idf or BM25 style relevance algorithms. that would have been the right fit for fuzzy name matching.
How would tf-idf help fuzzy match names? You would first need to decide the term youre looking for is the term you've indexed, which is what soundex &c would do.
Soundex is a useful option for reducing the search space (the blocking stage, with aims to have a high recall).
TF-IDF is a useful option as part of a high precision pipeline _post_ blocking.
A two step pipeline is typical and necessary because the high precision pipeline, while more accurate than the blocking algorithm, suffers from being slow.
Re: Fuzzy Name Matching in Postgres
#10I've used this stuff to good effect, for years.
SOUNDEX turns 103 this year. https://en.wikipedia.org/wiki/Soundex It's used for telephone-directory and census lookup for names in American English, and is designed for high false-positive matches. It's simplistic. But it's a snap to index. In Bayes lingo it's sensitive but not selective.