Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tracking supermarket prices with Playwright

sakisv.net

Post reply on HN
Print view View original

81–90 of 213 posts

Re: Tracking supermarket prices with Playwright

#81

Post by z3t4 »

seanwilson wrote: post #51

> They change things in a way that doesn't make your scraper fail. Instead the scraping continues as before, visiting all the links and scraping all the products. However the way they write the prices has changed and now a bag of chips doesn't cost €1.99 but €199. To catch these changes I rely on my transformation step being as strict as possible with its inputs. You could probably add some automated checks to not sy…

Sanity checks in programming are underrated, not only are they cheap performance vice, they catch bugs early that would otherwise poison the state.

Re: Tracking supermarket prices with Playwright

#82

Post by jaza »

seoulmetro wrote: post #48
brikym wrote: post #38

I have been doing something similar for New Zealand since the start of the year with Playwright/Typescript dumping parquet files to cloud storage. I've just collecting the data I have not yet displayed it. Most of the work is getting around the reverse proxy services like Akamai and Cloudflare. At the time I wrote it I thought nobody else was doing but now I know of at least 3 start ups doing the same in NZ. It seems…

Legality of this is rocky in Australia. I dare say that NZ is the same? There are so many scrapers that come and go doing this in AU but are usually shut down by the big supermarkets. It's a cycle of usefulness and "why doesn't this exist", except it had existed many times before.

Aussie here. I hadn't heard that price scraping is only quasi-legal here and that scrapers get shut down by the big supermarkets - but then again I'm not surprised.

I'm thinking of starting a little price comparison site, mainly to compare select products at Colesworths vs Aldi (I've just started doing more regular grocery shopping at Aldi myself). But as far as I know, Aldi don't have any prices / catalogues online, so my plan is to just manually enter the data myself in the short-term, and to appeal to crowdsourcing the data in the long-term. And plan is to just make it a simple SSG site (e.g. Hugo powered), data all in simple markdown / json files, data all sourced via github pull requests.

Feel free to get in touch if you'd like to help out, or if you know of anything similar that already exists: greenash dot net dot au slash contact

Re: Tracking supermarket prices with Playwright

#83

Post by crngefest »

mechanical_bear wrote: post #79
latexr wrote: post #75

Earlier quoted context omitted.

> I scrape even app and websites data And then try to sell it back to businesses, even suggesting they use the data to train AI. You also make it sound like there’s a team manually doing all the work. https://www.economizafloripa.com.br/?q=parceria-comercial That whole page makes my view of the project go from “helpful tool for the people, to wrestle back control from corporations selling basic necessities” to “just…

Where does this lack ethics? It seems that they are providing a useful service, that they created with their hard work. People are allowed to make money with their work.

[flagged]

Re: Tracking supermarket prices with Playwright

#84

Post by odysseus »

bufferoverflow wrote: post #65
odysseus wrote: post #63

I used to price track when I moved to a new area, but now I find it way easier to just shop at 2 markets or big box stores that consistently have low prices. In Europe, that would probably be Aldi/Lidl. In the U.S., maybe Costco/Trader Joe's. For online, CamelCamelCamel/Amazon. (for health/beauty/some electronics but not food) If you can buy direct from the manufacturer, sometimes that's even better. For example, I g…

> In the U.S., maybe Costco/Trader Joe's. Costco/Walmart/Aldi in my experience. Trader Joe's is higher quality, but generally more expensive.

Occasionally you can get the same Trader Joe’s private label products rebranded as Aldi merchandise for even cheaper at Aldi.

Re: Tracking supermarket prices with Playwright

#85

Post by latexr »

mechanical_bear wrote: post #79
latexr wrote: post #75

Earlier quoted context omitted.

> I scrape even app and websites data And then try to sell it back to businesses, even suggesting they use the data to train AI. You also make it sound like there’s a team manually doing all the work. https://www.economizafloripa.com.br/?q=parceria-comercial That whole page makes my view of the project go from “helpful tool for the people, to wrestle back control from corporations selling basic necessities” to “just…

Where does this lack ethics? It seems that they are providing a useful service, that they created with their hard work. People are allowed to make money with their work.

> Where does this lack ethics?

I didn’t say it lacked ethics, I said I expected it to be driven by ethics. There’s a world of difference. I just mean it initially sounded like this was a protest project “for the people”, done in a way to take back power from big corporations, and was saddened to see it’s another generic commercial endeavour.

> People are allowed to make money with their work.

Which is why I said it’s their prerogative.

If you’re going to reply, please strive to address the points made, what was said, not what you imagine the other person said. Don’t default to thinking the other person is being dismissive or malicious.

Re: Tracking supermarket prices with Playwright

#86

Post by latexr »

crngefest wrote: post #83
mechanical_bear wrote: post #79

Earlier quoted context omitted.

Where does this lack ethics? It seems that they are providing a useful service, that they created with their hard work. People are allowed to make money with their work.

[flagged]

That was not the argument at all. Please don’t strawman. From the guidelines:

> Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes.

> Comments should get more thoughtful and substantive, not less, as a topic gets more divisive.

Re: Tracking supermarket prices with Playwright

#87

Post by ustad »

brikym wrote: post #38

I have been doing something similar for New Zealand since the start of the year with Playwright/Typescript dumping parquet files to cloud storage. I've just collecting the data I have not yet displayed it. Most of the work is getting around the reverse proxy services like Akamai and Cloudflare. At the time I wrote it I thought nobody else was doing but now I know of at least 3 start ups doing the same in NZ. It seems…

Can anyone comment how supermarkets exploit customer segmentation by updating prices? How do the time-poor and poor-poor people generally respond?

“Often they'll segment on brand loyalty vs price sensitive people; There might be 3 popular brands of chocolate and every week only one of them will be sold at a fair price.”

Re: Tracking supermarket prices with Playwright

#89

Post by brikym »

ustad wrote: post #87
brikym wrote: post #38

I have been doing something similar for New Zealand since the start of the year with Playwright/Typescript dumping parquet files to cloud storage. I've just collecting the data I have not yet displayed it. Most of the work is getting around the reverse proxy services like Akamai and Cloudflare. At the time I wrote it I thought nobody else was doing but now I know of at least 3 start ups doing the same in NZ. It seems…

Can anyone comment how supermarkets exploit customer segmentation by updating prices? How do the time-poor and poor-poor people generally respond? “Often they'll segment on brand loyalty vs price sensitive people; There might be 3 popular brands of chocolate and every week only one of them will be sold at a fair price.”

Let's say there are three brands of some item. Each week one of the brands is rotated to $1 while the others are $2. And let's also suppose that the supermarket pays 80c per item.

The smart shopper might only buy in bulk once every three weeks when his favourite brand at a lower price, or twitch to the cheapest brand every week. A hurried or lazy shopper might always pick their favourite brand every week. If they buy one item a week the lazy shopper would have spent $5, while the smart shopper has only spent $3.

They've made 60c off the smart shopper and $2.60 off the lazy shopper. By segmenting out the lazy shoppers they've made $2. The whole idea of rotating the prices is nothing to do with the cost of goods sold it's all about making shopping a pain in the ass for busy people and catching them out.

Re: Tracking supermarket prices with Playwright

#90

Post by dawnerd »

bufferoverflow wrote: post #65
odysseus wrote: post #63

I used to price track when I moved to a new area, but now I find it way easier to just shop at 2 markets or big box stores that consistently have low prices. In Europe, that would probably be Aldi/Lidl. In the U.S., maybe Costco/Trader Joe's. For online, CamelCamelCamel/Amazon. (for health/beauty/some electronics but not food) If you can buy direct from the manufacturer, sometimes that's even better. For example, I g…

> In the U.S., maybe Costco/Trader Joe's. Costco/Walmart/Aldi in my experience. Trader Joe's is higher quality, but generally more expensive.

Sams club I’ve found beats Costco in some areas but for some items Costco absolutely undercuts like crazy. Cat litter at sams is twice the price when not on sale.

I pretty much just exclusively shop at Aldi/Walmart as they have the best prices overall. Kroger owned stores and Albertsons owned are insanely overpriced. Target is a good middle ground but I can’t stand shopping there now with everything getting locked up.

Post reply on HN