Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tracking supermarket prices with Playwright

sakisv.net

Post reply on HN
Print view View original

121–130 of 213 posts

Re: Tracking supermarket prices with Playwright

#121

Post by RasmusFromDK »

heap_perms wrote: post #117
RasmusFromDK wrote: post #104

Nice writeup. I've been through similar problems that you have with my contact lens price comparison website https://lenspricer.com/ that I run in ~30 countries. I have found, like you, that websites changing their HTML is a pain. One of my biggest hurdles initially was matching products across 100+ websites. Even though you think a product has a unique name, everyone puts their own twist on it. Most can be handled w…

I'm curious, can you wear contact lenses while working? I notice my eyes get tired when I look at a monitor for too long. Have you found any solutions for that?

I use contact lenses basically every day, and I have had no problems working in front of screens. There's a huge difference between the different brands. Mine is one of the more expensive ones (Acuvue Oasys 1-Day), so that might be part of it, but each eye is compatible with different lenses.

If I were you I would go to an optometrist and talk about this. They can also often give you free trials for different contacts and you can find one that works for you.

Re: Tracking supermarket prices with Playwright

#122

Post by kjksf »

mt_ wrote: post #115

What about networking costs? Is it free in Hetzner?

Depends on the server.

Most have at least 20 TB of bandwidth included in the price, even the lowest $5/mo shared cpu machines. 20 TB is a gigantic amount unless you're serving videos or some such.

Some have unlimited bandwidth (I mean they are effectively limited by the speed of network connection but you don't pay for amount).

Re: Tracking supermarket prices with Playwright

#123

Post by sumedh »

grafraf wrote: post #110
showsover wrote: post #100

Earlier quoted context omitted.

Do you have a technical writeup of your scraping approach? I'd love to read more about the challenges and solutions for them.

Unfortunately no, but i can share some insights that i hope can be of value: - Tech: Everything is hosted in AWS. We are using Golang in docker containers that does the scraping. They run on ECS Fargate spots when needed using cronjob. The scraping result is stored as a parquet in S3 and processed in our RDS Postgresql. We need to be creative and have some methods to identify that a particular product A in store 1 is…

Have the sites tried to shut you down?

Re: Tracking supermarket prices with Playwright

#124

Post by Lutger »

saaaaaam wrote: post #113
latexr wrote: post #85

Earlier quoted context omitted.

> Where does this lack ethics? I didn’t say it lacked ethics, I said I expected it to be driven by ethics. There’s a world of difference. I just mean it initially sounded like this was a protest project “for the people”, done in a way to take back power from big corporations, and was saddened to see it’s another generic commercial endeavour. > People are allowed to make money with their work. Which is why I said it’s…

I’m curious to know why you thought it “sounded like this was a protest project ‘for the people’”? I’ve read the parent post above and looked at the website and see nothing that would make me think it’s a “protest for the people”. It just seems a little strange when you then go on to say “strive to address… what was said, not what you imagine the other person said”.

I had the same thought. This is why:

- 'I created a similar website', so it compares to https://pricewatcher.gr/en/.

- a big part of the discussion is in the context of inflation and price gouging

- pricewatcher presents its data publicly for all consumers to see and use, it is clearly intended as a tool for consumers to combat price gouging strategies

- 'pricewatcher.gr is an independent site which is not endorsed by any shop', nothing suggests this website is making money off consumers

- the 'similar website' however is offering exclusive access to data to businesses, at a price, in order for those business to undercut the competition and become more profitable

So the goals are almost opposite. One is to help consumers combat price gouging of supermarkets, the other is to help supermarkets become (even) more profitable. It is similar in the sense that it is also scraping data, but it's not strange to think being similar would mean they would have the same goal, which they don't.

Re: Tracking supermarket prices with Playwright

#125

Post by avador »

pcblues wrote: post #96
pcblues wrote: post #93

This is interesting because I believe the two major supermarkets in Australia can create a duopoly in anti-competitive pricing by just employing price analysis AI algorithms on each side and the algorithms will likely end up cooperating to maximise profit. This can probably be done legally through publicly obtained prices and illegally by sharing supply cost or profit per product data. The result is likely to be simi…

The word I was looking for was collusion, but done with software and without people-based collusion.

Compusion.

Re: Tracking supermarket prices with Playwright

#126

Post by grafraf »

showsover wrote: post #112
grafraf wrote: post #110

Earlier quoted context omitted.

Unfortunately no, but i can share some insights that i hope can be of value: - Tech: Everything is hosted in AWS. We are using Golang in docker containers that does the scraping. They run on ECS Fargate spots when needed using cronjob. The scraping result is stored as a parquet in S3 and processed in our RDS Postgresql. We need to be creative and have some methods to identify that a particular product A in store 1 is…

Interesting stuff, thanks for the reply! Do you run into issues where they block your scraping attempts or are they quite relaxed on this? Circumventing the bot detection often forces us to go for Puppeteer so we can fully control the browser, but that carries quite a heavy cost compared to using a simple HTTP requester.

We have been blocked a couple of times during they years, usually using proxy has been enough. We try to reach out to the stores and try to establish a friendly relationship. The feelings have been mixed depending on what store we are talking to

Re: Tracking supermarket prices with Playwright

#127

Post by maccard »

seaal wrote: post #98
gruez wrote: post #57

Earlier quoted context omitted.

>Those who order grocery delivery online would benefit from price comparisons, because they can order from multiple stores at the same time. Not really, since the delivery fees/tips that you have to pay would eat up all the savings, unless maybe if you're buying for a family of 5 or something.

Instacart, Uber Eats, DoorDash all sell gift cards of $100 for $80 basically year round — when you combine that 20% with other promotions I often have deliveries that are cheaper than shopping in person.

Uber Eats and Deliveroo all have list prices that are 15-20+% above the shelf price in the same supermarket. Plus a delivery fee, plus the "service charge", I've _never_ found it to be competitive let alone cheaper.

Re: Tracking supermarket prices with Playwright

#128

Post by grafraf »

sumedh wrote: post #123
grafraf wrote: post #110

Earlier quoted context omitted.

Unfortunately no, but i can share some insights that i hope can be of value: - Tech: Everything is hosted in AWS. We are using Golang in docker containers that does the scraping. They run on ECS Fargate spots when needed using cronjob. The scraping result is stored as a parquet in S3 and processed in our RDS Postgresql. We need to be creative and have some methods to identify that a particular product A in store 1 is…

Have the sites tried to shut you down?

We received some harsh words in the start but everything we are doing is legally and by the book.

We try to establish good relationship with the stores as the customers don't always focus on the price, but sometimes they want a specific product. We are both helping the stores and the customers to find each other. We have sent million of users over the years to the stores (not unique of course as there are only 9 million people living in Sweden)

Re: Tracking supermarket prices with Playwright

#129

Post by cynicalsecurity »

> My first thought was to use AWS, since that's what I'm most familiar with, but looking at the prices for a moderately-powerful EC2 instance (i.e. 4 cores and 8GB of RAM) it was going to cost much more than I was comfortable to spend for a side project.

Yep, AWS is hugely overrated and overpriced.

Post reply on HN