Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tracking supermarket prices with Playwright

sakisv.net

Post reply on HN
Print view View original

61–70 of 213 posts

Re: Tracking supermarket prices with Playwright

#61

Post by MathMonkeyMan »

haolez wrote: post #14

I heard that some e-commerce sites will not block scrappers, but poison the data shown to them (e.g. subtly wrong prices). Does anyone know more about this?

This HN post had me playing around with Key Food's website. A lot of information is wrapped up in a cookie, but it looks like there isn't too much javascript rendering.

But when I hit the URLs with curl, without a cookie, I get a valid looking page, but it's just a hundred listings for "Baby Bok Choy." Maybe a test page?

After a little more fiddling, the server just responded with an empty response body. So, it looks like I'll have to use browser automation.

Re: Tracking supermarket prices with Playwright

#62

Post by immibis »

__MatrixMan__ wrote: post #54
langsoul-com wrote: post #49

The hard thing is not scraping, but getting around the increasingly sophisticated blockers. You'll need to constantly rotate residential proxies (high rated) and make sure not to exhibit data scraping patterns. Some supermarkets don't show the network requests in the network tab, so cannot just get that api response. Even then, mitm attacks with mobile app (to see the network requests and data) will also get blocked…

I wonder if we could get some legislation in place to require that they publish pricing data via an API so we don't have to tangle with the blockers at all.

Perhaps in Europe. Anywhere else, forget about it.

Re: Tracking supermarket prices with Playwright

#63

Post by odysseus »

I used to price track when I moved to a new area, but now I find it way easier to just shop at 2 markets or big box stores that consistently have low prices.

In Europe, that would probably be Aldi/Lidl.

In the U.S., maybe Costco/Trader Joe's.

For online, CamelCamelCamel/Amazon. (for health/beauty/some electronics but not food)

If you can buy direct from the manufacturer, sometimes that's even better. For example, I got a particular brand of soap I love at the soap's wholesaler site in bulk for less than half the retail price. For shampoo, buying the gallon size direct was way cheaper than buying from any retailer.

Re: Tracking supermarket prices with Playwright

#64

Post by girvo »

barbazoo wrote: post #31
xyst wrote: post #29

Would be nice to have a price transparency of goods. It would make processes like this much more easier to track by store, and region. For example, compare the price of oat milk at different zip codes and grocery stores. Additionally track “shrinkflation” (same price but smaller portion). On that note, it seems you are tracking price but are you also checking the cost per gram (or ounce)? Manufacturer or store could…

Grocers not putting per unit prices on the label is a pet peeve of mine. I can’t imagine any purpose not rooted in customer hostility.

It's required by law in Australia, which is nice

Re: Tracking supermarket prices with Playwright

#65

Post by bufferoverflow »

odysseus wrote: post #63

I used to price track when I moved to a new area, but now I find it way easier to just shop at 2 markets or big box stores that consistently have low prices. In Europe, that would probably be Aldi/Lidl. In the U.S., maybe Costco/Trader Joe's. For online, CamelCamelCamel/Amazon. (for health/beauty/some electronics but not food) If you can buy direct from the manufacturer, sometimes that's even better. For example, I g…

> In the U.S., maybe Costco/Trader Joe's.

Costco/Walmart/Aldi in my experience.

Trader Joe's is higher quality, but generally more expensive.

Re: Tracking supermarket prices with Playwright

#66

Post by DontchaKnowit »

bufferoverflow wrote: post #65
odysseus wrote: post #63

I used to price track when I moved to a new area, but now I find it way easier to just shop at 2 markets or big box stores that consistently have low prices. In Europe, that would probably be Aldi/Lidl. In the U.S., maybe Costco/Trader Joe's. For online, CamelCamelCamel/Amazon. (for health/beauty/some electronics but not food) If you can buy direct from the manufacturer, sometimes that's even better. For example, I g…

> In the U.S., maybe Costco/Trader Joe's. Costco/Walmart/Aldi in my experience. Trader Joe's is higher quality, but generally more expensive.

walmart is undisputed king of low prices and honestly in my experience the quality on their store brand stuff is pretty damn solid. and usually like half the price of comparable products. Been living off their greek yogurt for a while now. Its great

Re: Tracking supermarket prices with Playwright

#67

Post by dexwiz »

odysseus wrote: post #63

I used to price track when I moved to a new area, but now I find it way easier to just shop at 2 markets or big box stores that consistently have low prices. In Europe, that would probably be Aldi/Lidl. In the U.S., maybe Costco/Trader Joe's. For online, CamelCamelCamel/Amazon. (for health/beauty/some electronics but not food) If you can buy direct from the manufacturer, sometimes that's even better. For example, I g…

You can find ALDIs in the USA, but they are regional. Trader Joe’s is owned by the same family as ALDIs, and until recently (past 10 years) you wouldn’t see them in the same areas.

Re: Tracking supermarket prices with Playwright

#68

Post by scubadude »

teruakohatu wrote: post #42
brikym wrote: post #38

I have been doing something similar for New Zealand since the start of the year with Playwright/Typescript dumping parquet files to cloud storage. I've just collecting the data I have not yet displayed it. Most of the work is getting around the reverse proxy services like Akamai and Cloudflare. At the time I wrote it I thought nobody else was doing but now I know of at least 3 start ups doing the same in NZ. It seems…

I was planning on doing the same in NZ. I would be keen to chat to you about it (email in HN profile). I am a data scientist Did you notice anything pre and post Whittakers price increase(s)? They must have a brilliant PR firm in retainer for every major news outlet to more or less push the line that increased prices are a good thing for the consumer. I noticed more aggressive "sales" more recently, but unsure if I a…

I think Whittaker's changed their recipe some time in the last year. Whittaker's was what Cadbury used to be (good) but now I think they have both followed the same course. Markedly lower quality. This is the 200g blocks fwiw not sure about the wee 50g peanut slab.

Re: Tracking supermarket prices with Playwright

#69

Post by shiroiushi »

bufferoverflow wrote: post #65
odysseus wrote: post #63

I used to price track when I moved to a new area, but now I find it way easier to just shop at 2 markets or big box stores that consistently have low prices. In Europe, that would probably be Aldi/Lidl. In the U.S., maybe Costco/Trader Joe's. For online, CamelCamelCamel/Amazon. (for health/beauty/some electronics but not food) If you can buy direct from the manufacturer, sometimes that's even better. For example, I g…

> In the U.S., maybe Costco/Trader Joe's. Costco/Walmart/Aldi in my experience. Trader Joe's is higher quality, but generally more expensive.

Trader Joe's also only carries Trader Joe's-branded merchandise, aside from the produce. So if you're looking for something in particular that isn't a TJ item, you won't find it there.
Post reply on HN