Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tracking supermarket prices with Playwright

sakisv.net

Post reply on HN
Print view View original

41–50 of 213 posts

Re: Tracking supermarket prices with Playwright

#41

Post by walterbell »

brikym wrote: post #38

I have been doing something similar for New Zealand since the start of the year with Playwright/Typescript dumping parquet files to cloud storage. I've just collecting the data I have not yet displayed it. Most of the work is getting around the reverse proxy services like Akamai and Cloudflare. At the time I wrote it I thought nobody else was doing but now I know of at least 3 start ups doing the same in NZ. It seems…

Those who order grocery delivery online would benefit from price comparisons, because they can order from multiple stores at the same time. In addition, there's only one marketplace that has all the prices from different stores.

Re: Tracking supermarket prices with Playwright

#42

Post by teruakohatu »

brikym wrote: post #38

I have been doing something similar for New Zealand since the start of the year with Playwright/Typescript dumping parquet files to cloud storage. I've just collecting the data I have not yet displayed it. Most of the work is getting around the reverse proxy services like Akamai and Cloudflare. At the time I wrote it I thought nobody else was doing but now I know of at least 3 start ups doing the same in NZ. It seems…

I was planning on doing the same in NZ. I would be keen to chat to you about it (email in HN profile). I am a data scientist

Did you notice anything pre and post Whittakers price increase(s)? They must have a brilliant PR firm in retainer for every major news outlet to more or less push the line that increased prices are a good thing for the consumer. I noticed more aggressive "sales" more recently, but unsure if I am just paying more attention.

My prediction is that they will decrease the size of the bars soon.

Re: Tracking supermarket prices with Playwright

#43

Post by teruakohatu »

walterbell wrote: post #41
brikym wrote: post #38

I have been doing something similar for New Zealand since the start of the year with Playwright/Typescript dumping parquet files to cloud storage. I've just collecting the data I have not yet displayed it. Most of the work is getting around the reverse proxy services like Akamai and Cloudflare. At the time I wrote it I thought nobody else was doing but now I know of at least 3 start ups doing the same in NZ. It seems…

Those who order grocery delivery online would benefit from price comparisons, because they can order from multiple stores at the same time. In addition, there's only one marketplace that has all the prices from different stores.

I think the fees they tack on for online orders would ruin ordering different products from different stores. It mostly makes sense with staples that don't perish.

With fresh produce I find Pak n Save a lot more variable with quality, making online orders more risky despite the lower cost.

Re: Tracking supermarket prices with Playwright

#44

Post by mcoliver »

Larrikin wrote: post #5
mcoliver wrote: post #4

Earlier quoted context omitted.

I wrote a chrome extension to help with this. Clips all the coupons so you don't have to do individual searches. Has resulted in some wild surprise savings when shopping. www.throwlasso.com

This looks amazing. Do you have plans to support Firefox and other browsers?

It's published as a Firefox extension and you should be able to find it by searching for Lasso but I think I need to push the latest version and update the website. Thanks for the reminder. Which other browsers would you like?

Re: Tracking supermarket prices with Playwright

#45

Post by candiddevmike »

xyst wrote: post #29

Would be nice to have a price transparency of goods. It would make processes like this much more easier to track by store, and region. For example, compare the price of oat milk at different zip codes and grocery stores. Additionally track “shrinkflation” (same price but smaller portion). On that note, it seems you are tracking price but are you also checking the cost per gram (or ounce)? Manufacturer or store could…

Imagine mandating transparent cost of goods pricing. I'd love to see farmer was paid X, manufacturer Y, and grocer added Z.

Re: Tracking supermarket prices with Playwright

#46

Post by lotsofpulp »

mcoliver wrote: post #4
lotsofpulp wrote: post #2

In the US, retail businesses are offering individualized and general coupons via the phone apps. I wonder if this pricing can be tracked, as it results in significant differences. For example, I recently purchased fruit and dairy at Safeway in the western US, and after I had everything I wanted, I searched each item in the Safeway app, and it had coupons I could apply for $1.5 to $5 off per item. The other week, my w…

I wrote a chrome extension to help with this. Clips all the coupons so you don't have to do individual searches. Has resulted in some wild surprise savings when shopping. www.throwlasso.com

Wow! This is amazing, thank you. I usually use Safari, but will give it a try.

Re: Tracking supermarket prices with Playwright

#48

Post by seoulmetro »

brikym wrote: post #38

I have been doing something similar for New Zealand since the start of the year with Playwright/Typescript dumping parquet files to cloud storage. I've just collecting the data I have not yet displayed it. Most of the work is getting around the reverse proxy services like Akamai and Cloudflare. At the time I wrote it I thought nobody else was doing but now I know of at least 3 start ups doing the same in NZ. It seems…

Legality of this is rocky in Australia. I dare say that NZ is the same?

There are so many scrapers that come and go doing this in AU but are usually shut down by the big supermarkets.

It's a cycle of usefulness and "why doesn't this exist", except it had existed many times before.

Re: Tracking supermarket prices with Playwright

#49

Post by langsoul-com »

The hard thing is not scraping, but getting around the increasingly sophisticated blockers.

You'll need to constantly rotate residential proxies (high rated) and make sure not to exhibit data scraping patterns. Some supermarkets don't show the network requests in the network tab, so cannot just get that api response.

Even then, mitm attacks with mobile app (to see the network requests and data) will also get blocked without decent cover ups.

I tried but realised it isn't worth it due to the costs and constant dev work required. In fact, some of the supermarket pricing comparison services just have (cheap labour) people scrape it

Re: Tracking supermarket prices with Playwright

#50

Post by seanthemon »

langsoul-com wrote: post #49

The hard thing is not scraping, but getting around the increasingly sophisticated blockers. You'll need to constantly rotate residential proxies (high rated) and make sure not to exhibit data scraping patterns. Some supermarkets don't show the network requests in the network tab, so cannot just get that api response. Even then, mitm attacks with mobile app (to see the network requests and data) will also get blocked…

And you couldn't use OCR and simply take an image of the product list? Not ideal, but difficult or impossible to track depending on your method.
Post reply on HN