Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tracking supermarket prices with Playwright

sakisv.net

Post reply on HN
Print view View original

51–60 of 213 posts

Re: Tracking supermarket prices with Playwright

#51

Post by seanwilson »

> They change things in a way that doesn't make your scraper fail. Instead the scraping continues as before, visiting all the links and scraping all the products. However the way they write the prices has changed and now a bag of chips doesn't cost €1.99 but €199. To catch these changes I rely on my transformation step being as strict as possible with its inputs.

You could probably add some automated checks to not sync changes to prices/products if a sanity check fails e.g. each price shouldn't change by more than 100%, and the number of active products shouldn't change by more than 20%.

Re: Tracking supermarket prices with Playwright

#52

Post by russelg »

seoulmetro wrote: post #48
brikym wrote: post #38

I have been doing something similar for New Zealand since the start of the year with Playwright/Typescript dumping parquet files to cloud storage. I've just collecting the data I have not yet displayed it. Most of the work is getting around the reverse proxy services like Akamai and Cloudflare. At the time I wrote it I thought nobody else was doing but now I know of at least 3 start ups doing the same in NZ. It seems…

Legality of this is rocky in Australia. I dare say that NZ is the same? There are so many scrapers that come and go doing this in AU but are usually shut down by the big supermarkets. It's a cycle of usefulness and "why doesn't this exist", except it had existed many times before.

I think with the current climate wrt the big supermarkets in AU, now would be the time to push your luck. The court of public opinion will definitely not be on the supermarkets side, and the government may even step in.

Re: Tracking supermarket prices with Playwright

#53

Post by kareemm »

nosecreek wrote: post #7

Very cool! I did something similar in Canada ( https://grocerytracker.ca/ )

Was looking for one in Canada. Tried this out and it seems like some of the data is missing from where I live (halifax). Got an email I can hit you up at? Mine's in my HN profile - couldn't find yours on HN or your site.

Re: Tracking supermarket prices with Playwright

#54

Post by __MatrixMan__ »

langsoul-com wrote: post #49

The hard thing is not scraping, but getting around the increasingly sophisticated blockers. You'll need to constantly rotate residential proxies (high rated) and make sure not to exhibit data scraping patterns. Some supermarkets don't show the network requests in the network tab, so cannot just get that api response. Even then, mitm attacks with mobile app (to see the network requests and data) will also get blocked…

I wonder if we could get some legislation in place to require that they publish pricing data via an API so we don't have to tangle with the blockers at all.

Re: Tracking supermarket prices with Playwright

#55

Post by walterbell »

teruakohatu wrote: post #43
walterbell wrote: post #41

Earlier quoted context omitted.

Those who order grocery delivery online would benefit from price comparisons, because they can order from multiple stores at the same time. In addition, there's only one marketplace that has all the prices from different stores.

I think the fees they tack on for online orders would ruin ordering different products from different stores. It mostly makes sense with staples that don't perish. With fresh produce I find Pak n Save a lot more variable with quality, making online orders more risky despite the lower cost.

For those who have to order online (e.g. elderly), they are paying the fees anyway. They can avoid minimum order fees with bulk purchases of staples. Their bot/app can monitor prices and when a staple goes on sale, they can order multiple items to meet the threshold for a lower fee.

Re: Tracking supermarket prices with Playwright

#56

Post by NKosmatos »

Hey, thanks for creating https://pricewatcher.gr/en/ very much appreciated.

Nice blog post and very informative. Good to read that it costs you less than 70€ per year to run this and hope that the big supermarkets don’t block this somehow.

Have you thought of monetizing this? Perhaps with ads from the 3 big supermarkets you scrape ;-)

Re: Tracking supermarket prices with Playwright

#57

Post by gruez »

walterbell wrote: post #41
brikym wrote: post #38

I have been doing something similar for New Zealand since the start of the year with Playwright/Typescript dumping parquet files to cloud storage. I've just collecting the data I have not yet displayed it. Most of the work is getting around the reverse proxy services like Akamai and Cloudflare. At the time I wrote it I thought nobody else was doing but now I know of at least 3 start ups doing the same in NZ. It seems…

Those who order grocery delivery online would benefit from price comparisons, because they can order from multiple stores at the same time. In addition, there's only one marketplace that has all the prices from different stores.

>Those who order grocery delivery online would benefit from price comparisons, because they can order from multiple stores at the same time.

Not really, since the delivery fees/tips that you have to pay would eat up all the savings, unless maybe if you're buying for a family of 5 or something.

Re: Tracking supermarket prices with Playwright

#58

Post by nosecreek »

kareemm wrote: post #53
nosecreek wrote: post #7

Very cool! I did something similar in Canada ( https://grocerytracker.ca/ )

Was looking for one in Canada. Tried this out and it seems like some of the data is missing from where I live (halifax). Got an email I can hit you up at? Mine's in my HN profile - couldn't find yours on HN or your site.

For sure, just replace the first dot in the url from my profile with an @

Re: Tracking supermarket prices with Playwright

#59

Post by nosecreek »

baronswindle wrote: post #39
barbazoo wrote: post #31

Earlier quoted context omitted.

Grocers not putting per unit prices on the label is a pet peeve of mine. I can’t imagine any purpose not rooted in customer hostility.

In my experience, grocers always do include unit prices…at least in the USA. I’ve lived in Florida, Indiana, California, and New York, and in 35 years of life, I can’t remember ever not seeing the price per oz, per pound, per fl oz, etc. right next to the total price for food/drink and most home goods. There may be some exceptions, but I’m struggling to think of any except things where weight/volume aren’t really rel…

In Canada I think they are legally required to, but sometimes it can be frustrating if they don’t always compare like units - one product will be price per gram or 100 grams, and another price per kg. I’ve found with online shopping, the unit prices don’t take into account discounts and sale prices, which makes it harder to shop sales (in store seems to be better for this).

Re: Tracking supermarket prices with Playwright

#60

Post by lightbritefight »

baronswindle wrote: post #39
barbazoo wrote: post #31

Earlier quoted context omitted.

Grocers not putting per unit prices on the label is a pet peeve of mine. I can’t imagine any purpose not rooted in customer hostility.

In my experience, grocers always do include unit prices…at least in the USA. I’ve lived in Florida, Indiana, California, and New York, and in 35 years of life, I can’t remember ever not seeing the price per oz, per pound, per fl oz, etc. right next to the total price for food/drink and most home goods. There may be some exceptions, but I’m struggling to think of any except things where weight/volume aren’t really rel…

What they often do is put different units on the same type of good. Three chocolate bars? One will be in oz, one in lbs, one in "per unit."

They all are labelled, but it's still customer hostile to create comparison fatigue.

Post reply on HN