Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tracking supermarket prices with Playwright

sakisv.net

Post reply on HN
Print view View original

191–200 of 213 posts

Re: Tracking supermarket prices with Playwright

#191

Post by saaaaaam »

latexr wrote: post #153
saaaaaam wrote: post #113

Earlier quoted context omitted.

I’m curious to know why you thought it “sounded like this was a protest project ‘for the people’”? I’ve read the parent post above and looked at the website and see nothing that would make me think it’s a “protest for the people”. It just seems a little strange when you then go on to say “strive to address… what was said, not what you imagine the other person said”.

> I’m curious to know why you thought it “sounded like this was a protest project ‘for the people’”? See the sibling reply by another user, which I think explains it perfectly. https://news.ycombinator.com/item?id=41179628 > It just seems a little strange when you then go on to say “strive to address… what was said, not what you imagine the other person said”. It’s not strange at all if you pay attention to the words…

I would argue that you imagined something that the other person said - in this context, the other website. This is why your "strive to address... what was said, not what you imagine the other person said" comment sits uneasily for me.

I'm not sure if your highly condescending and somewhat reactive tone is intended or not, perhaps it's satirical, but in case you are unaware, your tone is highly condescending. Saying things like "if you pay attention to the words" and "you just have to make a genuine effort to understand" and your didactic "default to this" comes across rather badly.

Re: Tracking supermarket prices with Playwright

#192

Post by sphars »

Larrikin wrote: post #5
mcoliver wrote: post #4

Earlier quoted context omitted.

I wrote a chrome extension to help with this. Clips all the coupons so you don't have to do individual searches. Has resulted in some wild surprise savings when shopping. www.throwlasso.com

This looks amazing. Do you have plans to support Firefox and other browsers?

Looks like this is the Firefox extension: https://addons.mozilla.org/en-US/firefox/addon/lasso-coupon-...

Re: Tracking supermarket prices with Playwright

#193

Post by ElCapitanMarkla »

mishu2 wrote: post #149

Playwright is basically necessary for scraping nowadays, as the browser needs to do a lot of work before the web page becomes useful/readable. I remember scraping with HTTrack back in high school and most of the sites kept working... For my project ( https://frankendash.com/ ), I also ran into issues with dynamically generated class names which change on every site update, so in the end I just went with saving a crop…

HTTrack was fantastic, still was a couple of years ago when I used it for a small project too.

Re: Tracking supermarket prices with Playwright

#194

Post by grafraf »

filleokus wrote: post #150
grafraf wrote: post #99

We have been doing it for the Swedish market in more than 8 years. We have a website https://www.matspar.se/ , where the customer can browse all the products of all major online stores, compare the prices and add the products they want to buy in the cart. The customer can in the end of the journey compare the total price of that cart (including shipping fee) and export the cart to the store they desire to order it. I…

On the business side, what's your business model, how do you generate revenue? What's the longer term goals? (Public data shows the company have a revenue of ≈400k USD and 6 employees https://www.allabolag.se/5590076351/matspar-i-sverige-ab )

We are selling price/distribution data about the products we scrape. We do run some ads and have an affiliate deals.

The insight i can share is that the main (tech) goal is to make the product more user friendly and more aligned with the customer need as it has many pain points and we have gain some insights on the preferred customer journey.

Re: Tracking supermarket prices with Playwright

#195

Post by grafraf »

ElCapitanMarkla wrote: post #187
grafraf wrote: post #110

Earlier quoted context omitted.

Unfortunately no, but i can share some insights that i hope can be of value: - Tech: Everything is hosted in AWS. We are using Golang in docker containers that does the scraping. They run on ECS Fargate spots when needed using cronjob. The scraping result is stored as a parquet in S3 and processed in our RDS Postgresql. We need to be creative and have some methods to identify that a particular product A in store 1 is…

I'm unfamiliar with the parquet format and trying to understand - are you storing the raw scraped data in that format or are you storing the result of parsing the scraped data?

We are storing the result of the parsed scrape as parquet. I would advice to store the raw data as well in a different s3. The database should only have the data it needs and not act as a storage.

Re: Tracking supermarket prices with Playwright

#196

Post by jaza »

seoulmetro wrote: post #189
sumedh wrote: post #118

Earlier quoted context omitted.

> Legality of this is rocky in Australia. I dare say that NZ is the same? You might be breaking the sites terms and conditions but that does not mean its illegal. Dan Murphy uses a similar thing, they have their own price checking algorithm.

Breaking ToS is 100%, but that wouldn't stop people scraping, in all cases the people scraping are receiving something that stops them scraping while the data still is available, continuing the "why doesnt someone do this" project.

If it's just a matter of (not) breaking ToS, then my idea of manually collating prices shouldn't be able to be shut down (or credibly threatened with being shut down). It might be a painstaking (and unsustainable) effort, but price tags in a physical store have no ToS.

Re: Tracking supermarket prices with Playwright

#197

Post by latexr »

saaaaaam wrote: post #191
latexr wrote: post #153

Earlier quoted context omitted.

> I’m curious to know why you thought it “sounded like this was a protest project ‘for the people’”? See the sibling reply by another user, which I think explains it perfectly. https://news.ycombinator.com/item?id=41179628 > It just seems a little strange when you then go on to say “strive to address… what was said, not what you imagine the other person said”. It’s not strange at all if you pay attention to the words…

I would argue that you imagined something that the other person said - in this context, the other website. This is why your "strive to address... what was said, not what you imagine the other person said" comment sits uneasily for me. I'm not sure if your highly condescending and somewhat reactive tone is intended or not, perhaps it's satirical, but in case you are unaware, your tone is highly condescending. Saying t…

Respectfully, I don’t relish the idea of wasting more time failing to impart the difference between criticism and expressing personal disappointment. It is possible to dislike something without thinking that thing or its author are bad. Not everything is about taking a side. We used to be able to understand that.

My tone did succumb to the frustration of having the point be repeatedly strawmanned (you’ll have to turn on show dead to see every instance), which is worryingly becoming more common on HN. I accept and thank you for calling that out. While in general I appreciate HN’s simple interface, it makes it easy to miss when the person we’re talking to on a new reply is not the same as before, so I apologise for any negativity you felt from me.

I may still see your reply to this (no promises) but forgive me if I don’t reply further.

Re: Tracking supermarket prices with Playwright

#198

Post by gadders »

ElCapitanMarkla wrote: post #185
gadders wrote: post #173

This reminds me a bit of a meme that said something along the lines of "I don't want AI to draw my art, I want AI review my weekly grocery shop, workout which combinations of shops save me money, and then schedule the deliveries for me."

Something I was talking over with a friend a while ago was something along the lines of this. Where you could set a list of various meals that you like to eat regularly, a list of like 20 meal options. And then the app fetches the pricing for all ingredients and works out which meals are the best value that week. You kind of end up with a DIY HelloFresh / meal in a box service.

Yes, that would work.

"Dave, the cheapest meals for you this week are [LIST OF DINNERS]. Based on your preferred times, deliveries from Waitrose, Tesco and Sainburys are turning up at 7pm on Monday. Please check you still have the following staples in stock [EG pasta, tinned tomatoes etc]".

Re: Tracking supermarket prices with Playwright

#199

Post by throwaway7ahgb »

bane wrote: post #179
RasmusFromDK wrote: post #104

Nice writeup. I've been through similar problems that you have with my contact lens price comparison website https://lenspricer.com/ that I run in ~30 countries. I have found, like you, that websites changing their HTML is a pain. One of my biggest hurdles initially was matching products across 100+ websites. Even though you think a product has a unique name, everyone puts their own twist on it. Most can be handled w…

> One of my biggest hurdles initially was matching products across 100+ websites. Even though you think a product has a unique name, everyone puts their own twist on it. Most can be handled with regexes, but I had to manually map many of these (I used AI for some of it, but had to manually verify all of it) In the U.S. at least, big retailers will have product suppliers build slightly different SKUs for them to make…

Costco does this for sure, but Costco also creates their own products. For instance there are some variations of a package set that can only be bought at Costco, so you aren't getting the exact same box and items as anywhere else.

Re: Tracking supermarket prices with Playwright

#200

Post by throwaway7ahgb »

RasmusFromDK wrote: post #121
heap_perms wrote: post #117

Earlier quoted context omitted.

I'm curious, can you wear contact lenses while working? I notice my eyes get tired when I look at a monitor for too long. Have you found any solutions for that?

I use contact lenses basically every day, and I have had no problems working in front of screens. There's a huge difference between the different brands. Mine is one of the more expensive ones (Acuvue Oasys 1-Day), so that might be part of it, but each eye is compatible with different lenses. If I were you I would go to an optometrist and talk about this. They can also often give you free trials for different contact…

Age is important factor here, not just contract brands.

As you get older, your eyes get dryer. Also, having done Lasik and needing contacts after many years is a recipe for dry eyes.

Post reply on HN