I created a similar website which got lots of interest in my city. I scrape even app and websites data using a single server at Linode with 2GB of RAM with 5 IPv4 and 1000 IPv6 (which is free) and every single product is scraped at most 40 minutes interval, never more than that with avg time of 25 minutes. I use curl impersonate and scrape JSON as much as possible because 90% of markets provide prices from Ajax calls and the other 10% I use regex to easily parse the HTML. You can check it at https://www.economizafloripa.com.br
Tracking supermarket prices with Playwright
71–80 of 213 posts
Re: Tracking supermarket prices with Playwright
#72Very cool! I did something similar in Canada ( https://grocerytracker.ca/ )
Excellent work.
Re: Tracking supermarket prices with Playwright
#73The hard thing is not scraping, but getting around the increasingly sophisticated blockers. You'll need to constantly rotate residential proxies (high rated) and make sure not to exhibit data scraping patterns. Some supermarkets don't show the network requests in the network tab, so cannot just get that api response. Even then, mitm attacks with mobile app (to see the network requests and data) will also get blocked…
And you couldn't use OCR and simply take an image of the product list? Not ideal, but difficult or impossible to track depending on your method.
You'll get blocked before even seeing the page most times.
Re: Tracking supermarket prices with Playwright
#74Earlier quoted context omitted.
In my experience, grocers always do include unit prices…at least in the USA. I’ve lived in Florida, Indiana, California, and New York, and in 35 years of life, I can’t remember ever not seeing the price per oz, per pound, per fl oz, etc. right next to the total price for food/drink and most home goods. There may be some exceptions, but I’m struggling to think of any except things where weight/volume aren’t really rel…
What they often do is put different units on the same type of good. Three chocolate bars? One will be in oz, one in lbs, one in "per unit." They all are labelled, but it's still customer hostile to create comparison fatigue.
This is such a shame, anywhere this is mandated they should mandate by mass and for medical/vitamins per mass of active ingredient
Re: Tracking supermarket prices with Playwright
#75I created a similar website which got lots of interest in my city. I scrape even app and websites data using a single server at Linode with 2GB of RAM with 5 IPv4 and 1000 IPv6 (which is free) and every single product is scraped at most 40 minutes interval, never more than that with avg time of 25 minutes. I use curl impersonate and scrape JSON as much as possible because 90% of markets provide prices from Ajax calls…
> I scrape even app and websites data
And then try to sell it back to businesses, even suggesting they use the data to train AI. You also make it sound like there’s a team manually doing all the work.
https://www.economizafloripa.com.br/?q=parceria-comercial
That whole page makes my view of the project go from “helpful tool for the people, to wrestle back control from corporations selling basic necessities” to “just another attempt to make money”. Which is your prerogative, I was just expecting something different and more ethically driven when I read the homepage.
Re: Tracking supermarket prices with Playwright
#76I have been doing something similar for New Zealand since the start of the year with Playwright/Typescript dumping parquet files to cloud storage. I've just collecting the data I have not yet displayed it. Most of the work is getting around the reverse proxy services like Akamai and Cloudflare. At the time I wrote it I thought nobody else was doing but now I know of at least 3 start ups doing the same in NZ. It seems…
Legality of this is rocky in Australia. I dare say that NZ is the same? There are so many scrapers that come and go doing this in AU but are usually shut down by the big supermarkets. It's a cycle of usefulness and "why doesn't this exist", except it had existed many times before.
Agreed. Hopefully the govs price gouging mitigation strategy includes free flow of information (allowing scraping for price comparison).
I’ve been interested in price comparison for Australia for a while, am a Product designer/manager with a concept prototype design, looking for others interested to work on it. My email is on my profile if you are.
Re: Tracking supermarket prices with Playwright
#77Earlier quoted context omitted.
Legality of this is rocky in Australia. I dare say that NZ is the same? There are so many scrapers that come and go doing this in AU but are usually shut down by the big supermarkets. It's a cycle of usefulness and "why doesn't this exist", except it had existed many times before.
I think with the current climate wrt the big supermarkets in AU, now would be the time to push your luck. The court of public opinion will definitely not be on the supermarkets side, and the government may even step in.
Agreed. Should we make something and find out?
Re: Tracking supermarket prices with Playwright
#78I would be curious if there were a price difference between what is online and physically in the store.
In Denmark there often is, things like localised sales the 4-8 times a year a specific store celebrates its birthday or similar. You can scan their PDF brochures but you would need image recognition for most of them, and some well trained recognitions to boot since they often alter their layouts and prices are listed differently.
The biggest sales come from the individual store “close to expiration” sales where items can become really cheap. These aren’t available anywhere but the stores themselves though.
Here I think the biggest challenge might be the monopoly supermarket chains have on the market. We basically two major corporations with various brands. They are extremely similar in their pricing, and even though there are two low price competitors, these don’t seem to affect the competition with the two major corporations at all. What is worse is that one of these two major corporations is “winning”, meaning that we’re heading more and more toward what will basically be a true monopoly.
Re: Tracking supermarket prices with Playwright
#79I created a similar website which got lots of interest in my city. I scrape even app and websites data using a single server at Linode with 2GB of RAM with 5 IPv4 and 1000 IPv6 (which is free) and every single product is scraped at most 40 minutes interval, never more than that with avg time of 25 minutes. I use curl impersonate and scrape JSON as much as possible because 90% of markets provide prices from Ajax calls…
> I scrape even app and websites data And then try to sell it back to businesses, even suggesting they use the data to train AI. You also make it sound like there’s a team manually doing all the work. https://www.economizafloripa.com.br/?q=parceria-comercial That whole page makes my view of the project go from “helpful tool for the people, to wrestle back control from corporations selling basic necessities” to “just…
Where does this lack ethics? It seems that they are providing a useful service, that they created with their hard work. People are allowed to make money with their work.
Re: Tracking supermarket prices with Playwright
#80What if you add all products to your shopping cart and save it as “favourites” and scrape that every other day.
You would still need a way to add all items and to check if there are new ones