I am blocked on finding a
good (defined below) way to determine whether a product description A and product description B refer to the same product.
Imagine that a product description is a n-dimensional vector like:
( manufacturerName, modelName, width, height, length, color, ...)
Now imagine you have a file with m such vectors (where m is in millions), and that not all fields in the vectors are reliable info (typos, missing info, plain wrong, etc).
What is a good way to determine which product descriptions refer to the same product.
Is this even a good approach? What is state of the art? Are there simpler ways?
Here is what I mean by good:
- robust to typos, missing info, wrong info
- efficient since both m and n are large
- updateable (e.g. if classification was done, and 10k new descriptins are added, how to efficiently update and avoid full recomputation)