Very cool! I did something similar in Canada ( https://grocerytracker.ca/ )
Oh nice!
A thorny problem in my case is that the same item is named in 3 different ways between the 3 supermarkets which makes it very hard and annoying to do a proper comparison.
Very cool! I did something similar in Canada ( https://grocerytracker.ca/ )
Oh nice! A thorny problem in my case is that the same item is named in 3 different ways between the 3 supermarkets which makes it very hard and annoying to do a proper comparison. Did you have a similar problem?
Absolutely! It’s made it difficult to implement some of the cross-retailer comparison features I would like to add. For my charts I’ve just manually selected some products, but I’ve also been trying to get a “good enough but not perfect” string comparison algorithm working.
I heard that some e-commerce sites will not block scrappers, but poison the data shown to them (e.g. subtly wrong prices). Does anyone know more about this?
Ah, I love this. Nice work! I really wish supermarkets were mandated to post this information whenever the price of a particular SKU updated. The tools that could be built with such information would do amazing things for consumers.
Oh nice! A thorny problem in my case is that the same item is named in 3 different ways between the 3 supermarkets which makes it very hard and annoying to do a proper comparison. Did you have a similar problem?
Absolutely! It’s made it difficult to implement some of the cross-retailer comparison features I would like to add. For my charts I’ve just manually selected some products, but I’ve also been trying to get a “good enough but not perfect” string comparison algorithm working.
Ah yes.
My approach so far has been to first extract the brand names (which are also not written the same way for some fcking reason!), update the strings, and then compare the remaining.
If they have a high similarity (e.g. >95%) then they could be automatically merged, and then anything between 75%-95% can be reviewed manually.
One problem that the author notes is that so much rendering is done client side via javascript.
The flip side to this is that very often you find that the data populating the site is in a very simple JSON format to facilitate easy rendering, ironically making the scraping process a lot more reliable.
Ah, I love this. Nice work! I really wish supermarkets were mandated to post this information whenever the price of a particular SKU updated. The tools that could be built with such information would do amazing things for consumers.
As someone who actively works on these kind of systems, it's a bit more complicated than that. The past few years we worked on migrating from some old system from the 80's designed for LAN use only, to a cloud based item catalogue system that finally allowed us the ability to easily make pricing info more available to consumers, such as through an app.
Absolutely! It’s made it difficult to implement some of the cross-retailer comparison features I would like to add. For my charts I’ve just manually selected some products, but I’ve also been trying to get a “good enough but not perfect” string comparison algorithm working.
Ah yes. My approach so far has been to first extract the brand names (which are also not written the same way for some fcking reason!), update the strings, and then compare the remaining. If they have a high similarity (e.g. >95%) then they could be automatically merged, and then anything between 75%-95% can be reviewed manually.
I am not by any mean an expert but maybe using some LLMs or a sentence transformer here could help to do the job?
Ah, I love this. Nice work! I really wish supermarkets were mandated to post this information whenever the price of a particular SKU updated. The tools that could be built with such information would do amazing things for consumers.
Thanks! If Greece's case is anything to go by, I doubt they'd ever accept that as it may bring to light some... questionable practices. At some point I need to deduplicate the products and plot the prices across all 3 supermarkets on the same graph as I suspect it will show some interesting trends.