Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tracking supermarket prices with Playwright

sakisv.net

Post reply on HN
Print view View original

201–210 of 213 posts

Re: Tracking supermarket prices with Playwright

#201

Post by throwaway7ahgb »

pavel_lishin wrote: post #175
heap_perms wrote: post #117

Earlier quoted context omitted.

I'm curious, can you wear contact lenses while working? I notice my eyes get tired when I look at a monitor for too long. Have you found any solutions for that?

This is very likely age-dependent. When I was in my 20s, this was absolutely not a problem. When I hit my 30s, I started wearing glasses instead of contacts basically all the time, and it wasn't a problem. Now that I'm in my 40s, I'm having to take my glasses off to read a monitor and most things that are closer than my arm's reach.

Wait until you get to 50 and you have to take OFF your glasses to read things that are small or close.

This is the most annoying part of all my vision problems.

Re: Tracking supermarket prices with Playwright

#203

Post by Tryk »

maerten wrote: post #180

Nice article! > The second kind is nastier. > > They change things in a way that doesn't make your scraper fail. Instead the scraping continues as before, visiting all the links and scraping all the products. I have found that it is best to split the task of scraping and parsing into separate processes. By saving the raw JSON or HTML, you can always go back and apply fixes to your parser. I have built a similar syste…

Awesome, have been looking for something like this!

Re: Tracking supermarket prices with Playwright

#204

Post by jfil »

I'm building something similar for 7 grocery vendor in Canada and am looking to talk with others who are doing this - my email is in my profile.

One difference: I'm recording each scraping session as a HAR file (for proving provenance). mitmproxy (mitmdump) is invaluable for that.

Re: Tracking supermarket prices with Playwright

#205

Post by koolba »

Larrikin wrote: post #141
koolba wrote: post #9

Earlier quoted context omitted.

Ha! I have the same thing as a bookmarklet for specific sites. It’s fun to watch it render the clicks.

Could you share the bookmarklet?

It’s specific to each market but took all of 3 min to write. It’s just a […document.querySelectorAll(‘a.class-for-action’)].map(x => x.click())

No smarts. No rate limiting. Just barrage everything with a click.

Re: Tracking supermarket prices with Playwright

#206

Post by seaal »

walterbell wrote: post #136
seaal wrote: post #98

Earlier quoted context omitted.

Instacart, Uber Eats, DoorDash all sell gift cards of $100 for $80 basically year round — when you combine that 20% with other promotions I often have deliveries that are cheaper than shopping in person.

Are those usually discounted via coupon sites?

No, I buy them from Costco, Sam’s Club mostly.

Re: Tracking supermarket prices with Playwright

#207

Post by pavel_lishin »

throwaway7ahgb wrote: post #201
pavel_lishin wrote: post #175

Earlier quoted context omitted.

This is very likely age-dependent. When I was in my 20s, this was absolutely not a problem. When I hit my 30s, I started wearing glasses instead of contacts basically all the time, and it wasn't a problem. Now that I'm in my 40s, I'm having to take my glasses off to read a monitor and most things that are closer than my arm's reach.

Wait until you get to 50 and you have to take OFF your glasses to read things that are small or close. This is the most annoying part of all my vision problems.

Hah, I'm already there in my 40s! I'm seriously considering getting a strap for my glasses - right now I just hook them into my shirt, but they'll occasionally fall out when I bend over for something, and it's only a matter of time before they break or go into a sewer.

Re: Tracking supermarket prices with Playwright

#208

Post by barbazoo »

nosecreek wrote: post #59
baronswindle wrote: post #39

Earlier quoted context omitted.

In my experience, grocers always do include unit prices…at least in the USA. I’ve lived in Florida, Indiana, California, and New York, and in 35 years of life, I can’t remember ever not seeing the price per oz, per pound, per fl oz, etc. right next to the total price for food/drink and most home goods. There may be some exceptions, but I’m struggling to think of any except things where weight/volume aren’t really rel…

In Canada I think they are legally required to, but sometimes it can be frustrating if they don’t always compare like units - one product will be price per gram or 100 grams, and another price per kg. I’ve found with online shopping, the unit prices don’t take into account discounts and sale prices, which makes it harder to shop sales (in store seems to be better for this).

I doubt it. Seems totally optional here where I am in BC.

Re: Tracking supermarket prices with Playwright

#209

Post by barbazoo »

baronswindle wrote: post #39
barbazoo wrote: post #31

Earlier quoted context omitted.

Grocers not putting per unit prices on the label is a pet peeve of mine. I can’t imagine any purpose not rooted in customer hostility.

In my experience, grocers always do include unit prices…at least in the USA. I’ve lived in Florida, Indiana, California, and New York, and in 35 years of life, I can’t remember ever not seeing the price per oz, per pound, per fl oz, etc. right next to the total price for food/drink and most home goods. There may be some exceptions, but I’m struggling to think of any except things where weight/volume aren’t really rel…

I live in BC, common to not see unit pricing.

Re: Tracking supermarket prices with Playwright

#210

Post by langsoul-com »

sakisv wrote: post #103
langsoul-com wrote: post #49

The hard thing is not scraping, but getting around the increasingly sophisticated blockers. You'll need to constantly rotate residential proxies (high rated) and make sure not to exhibit data scraping patterns. Some supermarkets don't show the network requests in the network tab, so cannot just get that api response. Even then, mitm attacks with mobile app (to see the network requests and data) will also get blocked…

Thankfully I'm not there yet. Since this is just a side project, if it starts demanding too much of my time too often I'll just stop it and open both the code and the data. BTW, how could the network request not appear in the network tab? For me the hardest part is to correlate and compare products across supermarkets

If they don't populate the page via Ajax or network requests. Ie server side, then no requests for supermarket data will appear.

See nextjs server side, I believe they mention that as a security thing in their docs.

In terms of comparison, most names tend to be the same. So some similarity search if it's in the same category matches good enough.

Post reply on HN