Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tracking supermarket prices with Playwright

sakisv.net

Post reply on HN
Print view View original

181–190 of 213 posts

Re: Tracking supermarket prices with Playwright

#181

Post by sakisv »

maerten wrote: post #180

Nice article! > The second kind is nastier. > > They change things in a way that doesn't make your scraper fail. Instead the scraping continues as before, visiting all the links and scraping all the products. I have found that it is best to split the task of scraping and parsing into separate processes. By saving the raw JSON or HTML, you can always go back and apply fixes to your parser. I have built a similar syste…

Thanks!

> I have found that it is best to split the task of scraping and parsing into separate processes. By saving the raw JSON or HTML, you can always go back and apply fixes to your parser.

Yes, that's exactly what I've been doing and it saved me more times than I'd care to admit!

Re: Tracking supermarket prices with Playwright

#182

Post by sakisv »

gadders wrote: post #173

This reminds me a bit of a meme that said something along the lines of "I don't want AI to draw my art, I want AI review my weekly grocery shop, workout which combinations of shops save me money, and then schedule the deliveries for me."

Ha, you can't imagine how many times I've thought of doing just that - it's just that it's somewhat blocked by other things that need to happen before I even attempt to do it

Re: Tracking supermarket prices with Playwright

#183

Post by sakisv »

hnrodey wrote: post #166

Nice job getting through all this. I kind of enjoy writing scrapers and browser automation in general. Browser automation is quite powerful and under explored/utilized by the average developer. Something I learned recently, which might help your scrapers, is the ability in Playwright to sniff the network calls made through the browser (basically, programmatic API to the Network tab of the browser). The boost is that…

...or worse, if there _is_ an API call but the response is HTML instead of a json

Re: Tracking supermarket prices with Playwright

#184

Post by sakisv »

ptrik wrote: post #157

> My CI of choice is [Concourse]( https://concourse-ci.org/ ) which describes itself as "a continuous thing-doer". While it has a bit of a learning curve, I appreciate its declarative model for the pipelines and how it versions every single input to ensure reproducible builds as much as it can. What's the thought process behind using a CI server - which I thought is mainly for builds - for what essentially is a data…

Well I'm just thinking of concourse the same way it describes itself, "a continuous thing doer".

I want something that will run some code when something happens. In my case that "something" is a specific time of day. The code will spin up a server, connect it to tailscale, run the 3 scraping jobs and then tear down the server and parse the data. Then another pipeline runs that loads the data and refreshes the caches.

Of course I'm also using it for continuously deploying my app across 2 environments, or its monitoring stack, or running terraform etc.

Basically it runs everything for me so that I don't have to.

Re: Tracking supermarket prices with Playwright

#185

Post by ElCapitanMarkla »

gadders wrote: post #173

This reminds me a bit of a meme that said something along the lines of "I don't want AI to draw my art, I want AI review my weekly grocery shop, workout which combinations of shops save me money, and then schedule the deliveries for me."

Something I was talking over with a friend a while ago was something along the lines of this.

Where you could set a list of various meals that you like to eat regularly, a list of like 20 meal options. And then the app fetches the pricing for all ingredients and works out which meals are the best value that week.

You kind of end up with a DIY HelloFresh / meal in a box service.

Re: Tracking supermarket prices with Playwright

#186

Post by dotancohen »

heap_perms wrote: post #117
RasmusFromDK wrote: post #104

Nice writeup. I've been through similar problems that you have with my contact lens price comparison website https://lenspricer.com/ that I run in ~30 countries. I have found, like you, that websites changing their HTML is a pain. One of my biggest hurdles initially was matching products across 100+ websites. Even though you think a product has a unique name, everyone puts their own twist on it. Most can be handled w…

I'm curious, can you wear contact lenses while working? I notice my eyes get tired when I look at a monitor for too long. Have you found any solutions for that?

Go try an E-Ink device. B&N Nooks are small Android tablets in disguise, you just need to install a launcher. Boox devices are also Android.

I can use an E-Ink device all day without my eyes getting tired.

Re: Tracking supermarket prices with Playwright

#187

Post by ElCapitanMarkla »

grafraf wrote: post #110
showsover wrote: post #100

Earlier quoted context omitted.

Do you have a technical writeup of your scraping approach? I'd love to read more about the challenges and solutions for them.

Unfortunately no, but i can share some insights that i hope can be of value: - Tech: Everything is hosted in AWS. We are using Golang in docker containers that does the scraping. They run on ECS Fargate spots when needed using cronjob. The scraping result is stored as a parquet in S3 and processed in our RDS Postgresql. We need to be creative and have some methods to identify that a particular product A in store 1 is…

I'm unfamiliar with the parquet format and trying to understand - are you storing the raw scraped data in that format or are you storing the result of parsing the scraped data?

Re: Tracking supermarket prices with Playwright

#188

Post by seoulmetro »

sumedh wrote: post #119
jaza wrote: post #82

Earlier quoted context omitted.

Aussie here. I hadn't heard that price scraping is only quasi-legal here and that scrapers get shut down by the big supermarkets - but then again I'm not surprised. I'm thinking of starting a little price comparison site, mainly to compare select products at Colesworths vs Aldi (I've just started doing more regular grocery shopping at Aldi myself). But as far as I know, Aldi don't have any prices / catalogues online,…

> get shut down by the big supermarkets How do they shut them down?

Threaten legal action if scraping continues or something similar as the scraping ends abruptly for every site but the prices still sit online at the supermarkets site.

Re: Tracking supermarket prices with Playwright

#189

Post by seoulmetro »

sumedh wrote: post #118
seoulmetro wrote: post #48

Earlier quoted context omitted.

Legality of this is rocky in Australia. I dare say that NZ is the same? There are so many scrapers that come and go doing this in AU but are usually shut down by the big supermarkets. It's a cycle of usefulness and "why doesn't this exist", except it had existed many times before.

> Legality of this is rocky in Australia. I dare say that NZ is the same? You might be breaking the sites terms and conditions but that does not mean its illegal. Dan Murphy uses a similar thing, they have their own price checking algorithm.

Breaking ToS is 100%, but that wouldn't stop people scraping, in all cases the people scraping are receiving something that stops them scraping while the data still is available, continuing the "why doesnt someone do this" project.

Re: Tracking supermarket prices with Playwright

#190

Post by seoulmetro »

jaza wrote: post #82
seoulmetro wrote: post #48

Earlier quoted context omitted.

Legality of this is rocky in Australia. I dare say that NZ is the same? There are so many scrapers that come and go doing this in AU but are usually shut down by the big supermarkets. It's a cycle of usefulness and "why doesn't this exist", except it had existed many times before.

Aussie here. I hadn't heard that price scraping is only quasi-legal here and that scrapers get shut down by the big supermarkets - but then again I'm not surprised. I'm thinking of starting a little price comparison site, mainly to compare select products at Colesworths vs Aldi (I've just started doing more regular grocery shopping at Aldi myself). But as far as I know, Aldi don't have any prices / catalogues online,…

Ironic that I get a few responses with "I am wanting to do the project".

I did it around 10 years ago and I think I've seen one a year since that. I didn't bother once I saw that people shipping were doing well and increasing their dataset only for it to be severely reduced later (I assume due to threats of legal action.)

Post reply on HN