Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tracking supermarket prices with Playwright

sakisv.net

Post reply on HN
Print view View original

111–120 of 213 posts

Re: Tracking supermarket prices with Playwright

#111

Post by jasomill »

jaza wrote: post #82
seoulmetro wrote: post #48

Earlier quoted context omitted.

Legality of this is rocky in Australia. I dare say that NZ is the same? There are so many scrapers that come and go doing this in AU but are usually shut down by the big supermarkets. It's a cycle of usefulness and "why doesn't this exist", except it had existed many times before.

Aussie here. I hadn't heard that price scraping is only quasi-legal here and that scrapers get shut down by the big supermarkets - but then again I'm not surprised. I'm thinking of starting a little price comparison site, mainly to compare select products at Colesworths vs Aldi (I've just started doing more regular grocery shopping at Aldi myself). But as far as I know, Aldi don't have any prices / catalogues online,…

But as far as I know, Aldi don't have any prices / catalogues online

There are a few here, but more along the lines of a flyer than a catalog:

https://www.aldi.com.au/groceries/

Aldi US has a more-or-less complete catalog online, so it might be worth crowdsourcing suggestions to the parent company to implement the same in Australia.

Re: Tracking supermarket prices with Playwright

#112

Post by showsover »

grafraf wrote: post #110
showsover wrote: post #100

Earlier quoted context omitted.

Do you have a technical writeup of your scraping approach? I'd love to read more about the challenges and solutions for them.

Unfortunately no, but i can share some insights that i hope can be of value: - Tech: Everything is hosted in AWS. We are using Golang in docker containers that does the scraping. They run on ECS Fargate spots when needed using cronjob. The scraping result is stored as a parquet in S3 and processed in our RDS Postgresql. We need to be creative and have some methods to identify that a particular product A in store 1 is…

Interesting stuff, thanks for the reply!

Do you run into issues where they block your scraping attempts or are they quite relaxed on this? Circumventing the bot detection often forces us to go for Puppeteer so we can fully control the browser, but that carries quite a heavy cost compared to using a simple HTTP requester.

Re: Tracking supermarket prices with Playwright

#113

Post by saaaaaam »

latexr wrote: post #85
mechanical_bear wrote: post #79

Earlier quoted context omitted.

Where does this lack ethics? It seems that they are providing a useful service, that they created with their hard work. People are allowed to make money with their work.

> Where does this lack ethics? I didn’t say it lacked ethics, I said I expected it to be driven by ethics. There’s a world of difference. I just mean it initially sounded like this was a protest project “for the people”, done in a way to take back power from big corporations, and was saddened to see it’s another generic commercial endeavour. > People are allowed to make money with their work. Which is why I said it’s…

I’m curious to know why you thought it “sounded like this was a protest project ‘for the people’”?

I’ve read the parent post above and looked at the website and see nothing that would make me think it’s a “protest for the people”.

It just seems a little strange when you then go on to say “strive to address… what was said, not what you imagine the other person said”.

Re: Tracking supermarket prices with Playwright

#114

Post by jasomill »

dexwiz wrote: post #67
odysseus wrote: post #63

I used to price track when I moved to a new area, but now I find it way easier to just shop at 2 markets or big box stores that consistently have low prices. In Europe, that would probably be Aldi/Lidl. In the U.S., maybe Costco/Trader Joe's. For online, CamelCamelCamel/Amazon. (for health/beauty/some electronics but not food) If you can buy direct from the manufacturer, sometimes that's even better. For example, I g…

You can find ALDIs in the USA, but they are regional. Trader Joe’s is owned by the same family as ALDIs, and until recently (past 10 years) you wouldn’t see them in the same areas.

I'd usually associate the term "regional" with chains like Meijer, Giant Eagle, and Winn-Dixie.

With 2,392 stores in 38 states plus DC[1], I'm not sure Aldi US qualifies.

[1] https://stores.aldi.us

Re: Tracking supermarket prices with Playwright

#116

Post by Larrikin »

mcoliver wrote: post #44
Larrikin wrote: post #5

Earlier quoted context omitted.

This looks amazing. Do you have plans to support Firefox and other browsers?

It's published as a Firefox extension and you should be able to find it by searching for Lasso but I think I need to push the latest version and update the website. Thanks for the reminder. Which other browsers would you like?

Personally I only care about Firefox, but I think its pretty standard to support Firefox, Chromium, and Safari.

Tried it out and works well after I figured out how to start clipping but didn't work for a couple sites I tried, mostly the financial ones like Chase and PayPal. Looking forward to the update!

Re: Tracking supermarket prices with Playwright

#117

Post by heap_perms »

RasmusFromDK wrote: post #104

Nice writeup. I've been through similar problems that you have with my contact lens price comparison website https://lenspricer.com/ that I run in ~30 countries. I have found, like you, that websites changing their HTML is a pain. One of my biggest hurdles initially was matching products across 100+ websites. Even though you think a product has a unique name, everyone puts their own twist on it. Most can be handled w…

I'm curious, can you wear contact lenses while working? I notice my eyes get tired when I look at a monitor for too long. Have you found any solutions for that?

Re: Tracking supermarket prices with Playwright

#118

Post by sumedh »

seoulmetro wrote: post #48
brikym wrote: post #38

I have been doing something similar for New Zealand since the start of the year with Playwright/Typescript dumping parquet files to cloud storage. I've just collecting the data I have not yet displayed it. Most of the work is getting around the reverse proxy services like Akamai and Cloudflare. At the time I wrote it I thought nobody else was doing but now I know of at least 3 start ups doing the same in NZ. It seems…

Legality of this is rocky in Australia. I dare say that NZ is the same? There are so many scrapers that come and go doing this in AU but are usually shut down by the big supermarkets. It's a cycle of usefulness and "why doesn't this exist", except it had existed many times before.

> Legality of this is rocky in Australia. I dare say that NZ is the same?

You might be breaking the sites terms and conditions but that does not mean its illegal.

Dan Murphy uses a similar thing, they have their own price checking algorithm.

Re: Tracking supermarket prices with Playwright

#119

Post by sumedh »

jaza wrote: post #82
seoulmetro wrote: post #48

Earlier quoted context omitted.

Legality of this is rocky in Australia. I dare say that NZ is the same? There are so many scrapers that come and go doing this in AU but are usually shut down by the big supermarkets. It's a cycle of usefulness and "why doesn't this exist", except it had existed many times before.

Aussie here. I hadn't heard that price scraping is only quasi-legal here and that scrapers get shut down by the big supermarkets - but then again I'm not surprised. I'm thinking of starting a little price comparison site, mainly to compare select products at Colesworths vs Aldi (I've just started doing more regular grocery shopping at Aldi myself). But as far as I know, Aldi don't have any prices / catalogues online,…

> get shut down by the big supermarkets

How do they shut them down?

Re: Tracking supermarket prices with Playwright

#120

Post by jasomill »

lightbritefight wrote: post #60
baronswindle wrote: post #39

Earlier quoted context omitted.

In my experience, grocers always do include unit prices…at least in the USA. I’ve lived in Florida, Indiana, California, and New York, and in 35 years of life, I can’t remember ever not seeing the price per oz, per pound, per fl oz, etc. right next to the total price for food/drink and most home goods. There may be some exceptions, but I’m struggling to think of any except things where weight/volume aren’t really rel…

What they often do is put different units on the same type of good. Three chocolate bars? One will be in oz, one in lbs, one in "per unit." They all are labelled, but it's still customer hostile to create comparison fatigue.

Worse, I've seen CVS do things like place a 180-count package of generic medication next to an identically-sized 200-count package of the equivalent name brand, with the generic costing a bit less, but with a slightly higher unit price due to the mismatched quantities.
Post reply on HN