Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tracking supermarket prices with Playwright

sakisv.net

Post reply on HN
Print view View original

131–140 of 213 posts

Re: Tracking supermarket prices with Playwright

#131

Post by shellfishgene »

RasmusFromDK wrote: post #104

Nice writeup. I've been through similar problems that you have with my contact lens price comparison website https://lenspricer.com/ that I run in ~30 countries. I have found, like you, that websites changing their HTML is a pain. One of my biggest hurdles initially was matching products across 100+ websites. Even though you think a product has a unique name, everyone puts their own twist on it. Most can be handled w…

For Germany, below the prices it says "some links may be sponsored", but it does not mark which ones. Is that even legal? Also there seem to be very few shops, are maybe all the links sponsored? Also idealo.de finds lower prices.

Re: Tracking supermarket prices with Playwright

#134

Post by RasmusFromDK »

shellfishgene wrote: post #131
RasmusFromDK wrote: post #104

Nice writeup. I've been through similar problems that you have with my contact lens price comparison website https://lenspricer.com/ that I run in ~30 countries. I have found, like you, that websites changing their HTML is a pain. One of my biggest hurdles initially was matching products across 100+ websites. Even though you think a product has a unique name, everyone puts their own twist on it. Most can be handled w…

For Germany, below the prices it says "some links may be sponsored", but it does not mark which ones. Is that even legal? Also there seem to be very few shops, are maybe all the links sponsored? Also idealo.de finds lower prices.

When I decided to put the text like that, I had looked at maybe 10-20 of the biggest price comparison websites across different countries because I of course want to make sure I respect all regulations that there are. I found that many of them don't even write anywhere that the links may be sponsored, and you have to go to the "about" page or similar to find this. I think that I actually go further than most of them when it comes to making it known that some links may be sponsored.

Now that you mention idealo, there seems to be no mention at all on a product page that they are paid by the stores, you have to click the "rank" link in the footer to be brought to a page https://www.idealo.de/aktion/ranking where they write this.

Re: Tracking supermarket prices with Playwright

#135

Post by Spivak »

sakisv wrote: post #132
joelthelion wrote: post #130

We should mutualize scraping efforts, creating a sort of Wikipedia of scraped data. I bet a ton of people and cool applications would benefit from it.

Haha all we have to do is agree on the format, right?

We already did. The format supports attaching related content, the scraped info, with the archive itself. So you get your data along with the means to generate it yourself if you want something different.

https://en.m.wikipedia.org/wiki/WARC_(file_format)

Re: Tracking supermarket prices with Playwright

#136

Post by walterbell »

seaal wrote: post #98
gruez wrote: post #57

Earlier quoted context omitted.

>Those who order grocery delivery online would benefit from price comparisons, because they can order from multiple stores at the same time. Not really, since the delivery fees/tips that you have to pay would eat up all the savings, unless maybe if you're buying for a family of 5 or something.

Instacart, Uber Eats, DoorDash all sell gift cards of $100 for $80 basically year round — when you combine that 20% with other promotions I often have deliveries that are cheaper than shopping in person.

Are those usually discounted via coupon sites?

Re: Tracking supermarket prices with Playwright

#137

Post by walterbell »

maccard wrote: post #127
seaal wrote: post #98

Earlier quoted context omitted.

Instacart, Uber Eats, DoorDash all sell gift cards of $100 for $80 basically year round — when you combine that 20% with other promotions I often have deliveries that are cheaper than shopping in person.

Uber Eats and Deliveroo all have list prices that are 15-20+% above the shelf price in the same supermarket. Plus a delivery fee, plus the "service charge", I've _never_ found it to be competitive let alone cheaper.

Some vendors offer "in-store prices" on Instacart.

Re: Tracking supermarket prices with Playwright

#139

Post by psd1 »

Alifatisk wrote: post #133

Some stores don’t have an interactive website but instead send out magazines to your email with news for the week. How would one scrape those? Anyone experienced?

Imap library to dump the attachment, pandoc to convert it to html, then DOM library to parts it statically.

Likely easier than website scraping.

Re: Tracking supermarket prices with Playwright

#140

Post by Alifatisk »

psd1 wrote: post #139
Alifatisk wrote: post #133

Some stores don’t have an interactive website but instead send out magazines to your email with news for the week. How would one scrape those? Anyone experienced?

Imap library to dump the attachment, pandoc to convert it to html, then DOM library to parts it statically. Likely easier than website scraping.

I’ll try this approach, thanks! Most magazines I’ve noticed are using a grid design, so my first thought was to somehow detect each square then OCR the product name with it’a price.
Post reply on HN