Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tracking supermarket prices with Playwright

sakisv.net

Post reply on HN
Print view View original

21–30 of 213 posts

Re: Tracking supermarket prices with Playwright

#21

Post by project2501a »

nosecreek wrote: post #12
sakisv wrote: post #11

Earlier quoted context omitted.

Oh nice! A thorny problem in my case is that the same item is named in 3 different ways between the 3 supermarkets which makes it very hard and annoying to do a proper comparison. Did you have a similar problem?

Absolutely! It’s made it difficult to implement some of the cross-retailer comparison features I would like to add. For my charts I’ve just manually selected some products, but I’ve also been trying to get a “good enough but not perfect” string comparison algorithm working.

would maintaining a map of products product_x[supermarket] with 2-3 values work? I don't suspect that supermarkets would be very keen to change the name (but they might play other dirty games)

I am thinking of doing the same thing for linux packages in debian and fedora

Re: Tracking supermarket prices with Playwright

#22

Post by barryrandall »

haolez wrote: post #14

I heard that some e-commerce sites will not block scrappers, but poison the data shown to them (e.g. subtly wrong prices). Does anyone know more about this?

I never poisoned data, but I have implemented systems where clients who made requests too quickly got served data from a snapshot that only updated every 15 minutes.

Re: Tracking supermarket prices with Playwright

#23

Post by zzzzthis »

stronglikedan wrote: post #15
ikesau wrote: post #3

Ah, I love this. Nice work! I really wish supermarkets were mandated to post this information whenever the price of a particular SKU updated. The tools that could be built with such information would do amazing things for consumers.

[flagged]

[flagged]

Re: Tracking supermarket prices with Playwright

#24

Post by sakisv »

project2501a wrote: post #20
sakisv wrote: post #10

Earlier quoted context omitted.

Thanks! If Greece's case is anything to go by, I doubt they'd ever accept that as it may bring to light some... questionable practices. At some point I need to deduplicate the products and plot the prices across all 3 supermarkets on the same graph as I suspect it will show some interesting trends.

fyi, I posted this on /r/greece

Thanks!

Re: Tracking supermarket prices with Playwright

#25

Post by samstave »

stronglikedan wrote: post #15
ikesau wrote: post #3

Ah, I love this. Nice work! I really wish supermarkets were mandated to post this information whenever the price of a particular SKU updated. The tools that could be built with such information would do amazing things for consumers.

[flagged]

no - UPCs are U for a reason.

the only thing that is needed is to be able to query the UPC for the price at the location.

The only way to scan the UPC for a price per location is... at the register...

Instead - what we need is an app where every single person uploads their receipt.

Several things here - have AI scan the receipts for the same item from the same [VARIABLE] tree and look for gouging, fraud, mis prints etc.

How much is that spaghetti in the ghetto vs the grotto?

etc.

If we could have every single receipt from EVERY supplier include a QR code of that receipt that they need to have scannable for the digital json of the receipt to last as long as the return policies/laws allow.

--

I did this for cannabis labels for a company a few years ago: The cannabis had to have a publicly available PDF of the lab results for chemical testing.

I made a lable through a tiny.url for each product and the dispensary to go to that lead to the lab's PDF posting of the results..

Then when the QR was scanned - you could see which products from which dispensaries were scanned against the lab results for it and also report the sales location for them....

(there is an amazing labeling software thats free from Seagull Scientific called BarTender (as-in Barcodes):

[0] https://www.seagullscientific.com/software/starter/

Re: Tracking supermarket prices with Playwright

#26

Post by snac »

nosecreek wrote: post #7

Very cool! I did something similar in Canada ( https://grocerytracker.ca/ )

Love your site! It was a great source of inspiration with the amount of data you collect.

I did the same and made https://grocerygoose.ca/

Published the API endpoints that I “discovered” to make the app https://github.com/snacsnoc/grocery-app (see HACKING.md)

It’s an unfortunate state of affairs when devs like us have to go to such great lengths to track the price of a commodity (food).

Re: Tracking supermarket prices with Playwright

#27

Post by sakisv »

victornomad wrote: post #19
sakisv wrote: post #16

Earlier quoted context omitted.

Ah yes. My approach so far has been to first extract the brand names (which are also not written the same way for some fcking reason!), update the strings, and then compare the remaining. If they have a high similarity (e.g. >95%) then they could be automatically merged, and then anything between 75%-95% can be reviewed manually.

I am not by any mean an expert but maybe using some LLMs or a sentence transformer here could help to do the job?

I gave it a very quick try with chatgpt, but wasn't very impressed from the results.

Granted it was around January, and things may have progressed...

(Βut then again why take the easy approach when Ι can waste a few afternoons playing around with string comparisons)

Re: Tracking supermarket prices with Playwright

#28

Post by sakisv »

andrewla wrote: post #17

One problem that the author notes is that so much rendering is done client side via javascript. The flip side to this is that very often you find that the data populating the site is in a very simple JSON format to facilitate easy rendering, ironically making the scraping process a lot more reliable.

Initially that's what I wanted to do, but the first supermarket I did is sending back HTML rendered on the server side, so I abandonded this approach for the sake of "consistency".

Lately I've been thinking to bite the bullet and Just Do It, but since it's working I'm a bit reluctant to touch it.

Re: Tracking supermarket prices with Playwright

#29

Post by xyst »

Would be nice to have a price transparency of goods. It would make processes like this much more easier to track by store, and region.

For example, compare the price of oat milk at different zip codes and grocery stores. Additionally track “shrinkflation” (same price but smaller portion).

On that note, it seems you are tracking price but are you also checking the cost per gram (or ounce)? Manufacturer or store could keep price the same but offer less to the consumer. Wonder if your tool would catch this.

Re: Tracking supermarket prices with Playwright

#30

Post by andrewla »

sakisv wrote: post #28
andrewla wrote: post #17

One problem that the author notes is that so much rendering is done client side via javascript. The flip side to this is that very often you find that the data populating the site is in a very simple JSON format to facilitate easy rendering, ironically making the scraping process a lot more reliable.

Initially that's what I wanted to do, but the first supermarket I did is sending back HTML rendered on the server side, so I abandonded this approach for the sake of "consistency". Lately I've been thinking to bite the bullet and Just Do It, but since it's working I'm a bit reluctant to touch it.

For your purposes scraping the user-visible site probably makes the most sense since in the end, their users' eyes are the target.

I am typically doing one-off scraping and for that, an undocumented but clean JSON api makes things so much easier, so I've grown to enjoy sites that are unnecessarily complex in their rendering.

Post reply on HN