Tracking supermarket prices with Playwright
161–170 of 213 posts
Re: Tracking supermarket prices with Playwright
#162> While the supermarket that I was using to test things every step of the way worked fine, one of them didn't. The reason? It was behind Akamai and they had enabled a firewall rule which was blocking requests originating from non-residential IP addresses. Why did you pick Tailscale as the solution for proxy vs scraping with something like AWS Lambda?
Didn't you answer your own question with the quote? It needs to originate from a residential IP address
Re: Tracking supermarket prices with Playwright
#163I have been doing something similar for New Zealand since the start of the year with Playwright/Typescript dumping parquet files to cloud storage. I've just collecting the data I have not yet displayed it. Most of the work is getting around the reverse proxy services like Akamai and Cloudflare. At the time I wrote it I thought nobody else was doing but now I know of at least 3 start ups doing the same in NZ. It seems…
Legality of this is rocky in Australia. I dare say that NZ is the same? There are so many scrapers that come and go doing this in AU but are usually shut down by the big supermarkets. It's a cycle of usefulness and "why doesn't this exist", except it had existed many times before.
For the other commenters here - looks like this site does the job? https://hotprices.org/
With the corresponding repo too: https://github.com/Javex/hotprices-au
Re: Tracking supermarket prices with Playwright
#164Can someone name the South-American country where they have a government price comparison website. Listing all products was required by law. Someone showed me this a decade ago. The site had many obvious issues but it did list everything. If I remember correctly it was started to stop merchants pricing things by who is buying. I forget which country it was.
Re: Tracking supermarket prices with Playwright
#165The hard thing is not scraping, but getting around the increasingly sophisticated blockers. You'll need to constantly rotate residential proxies (high rated) and make sure not to exhibit data scraping patterns. Some supermarkets don't show the network requests in the network tab, so cannot just get that api response. Even then, mitm attacks with mobile app (to see the network requests and data) will also get blocked…
I wonder if we could get some legislation in place to require that they publish pricing data via an API so we don't have to tangle with the blockers at all.
I'd prefer that governments enact legislation that prevents discriminating against IP addresses, perhaps under net neutrality laws.
For anyone with some clout/money who would like to stop corporations like Akamai and Cloudflare from unilaterally blocking IP addresses, the way that works is you file a lawsuit against the corporations and get an injunction to stop a practice (like IP blacklisting) during the legal proceedings. IANAL, so please forgive me if my terminology isn't quite right here:
https://pro.bloomberglaw.com/insights/litigation/how-to-file...
https://www.law.cornell.edu/wex/injunctive_relief
Injunctions have been used with great success for a century or more to stop corporations from polluting or destroying ecosystems. The idea is that since anyone can file an injunction, that creates an incentive for corporations to follow the law or risk having their work halted for months or years as the case proceeds.
I'd argue that unilaterally blocking IP addresses on a wide scale pollutes the ecosystem of the internet, so can't be allowed to continue.
Of course, corporations have thought of all of this, so have gone to great lengths to lobby governments and use regulatory capture to install politicians and judges who rule in their favor to pay back campaign contributions they've received from those same corporations:
https://www.crowell.com/en/insights/client-alerts/supreme-co...
https://www.mcneeslaw.com/nlrb-injunction/
So now the pressures that corporations have applied on the legal system to protect their own interests at the cost of employees, taxpayers and the environment have started to affect other industries like ours in tech.
You'll tend to hear that disruptive ideas like I've discussed are bad for business from the mainstream media and corporate PR departments, since they're protecting their own interests. That's why I feel that the heart of hacker culture is in disrupting the status quo.
Re: Tracking supermarket prices with Playwright
#166Nice job getting through all this. I kind of enjoy writing scrapers and browser automation in general. Browser automation is quite powerful and under explored/utilized by the average developer.
Something I learned recently, which might help your scrapers, is the ability in Playwright to sniff the network calls made through the browser (basically, programmatic API to the Network tab of the browser).
The boost is that you allow the website/webapp to make the API calls and then the scraper focuses on the data (rather than allowing the page to render DOM updates).
This approach falls apart if the page is doing server side rendering as there are no API calls to sniff.
Re: Tracking supermarket prices with Playwright
#167Nice article, enjoyed reading it. I’m Pier, co founder of https://Databoutique.com , which is a marketplace for web scraped data. If you’re willing to monetize your data extractions, you can list them on our website. We just started with the grocery industry and it would be great to have you on board.
Do you have data on which data is in higher demand? Do you keep a list of frequently-requested datasets?
Re: Tracking supermarket prices with Playwright
#168Earlier quoted context omitted.
I'm curious, can you wear contact lenses while working? I notice my eyes get tired when I look at a monitor for too long. Have you found any solutions for that?
I use contact lenses basically every day, and I have had no problems working in front of screens. There's a huge difference between the different brands. Mine is one of the more expensive ones (Acuvue Oasys 1-Day), so that might be part of it, but each eye is compatible with different lenses. If I were you I would go to an optometrist and talk about this. They can also often give you free trials for different contact…
> Acuvue Oasys 1-Day
I don't often wear contacts at work but I can second that these are great for "all day" wear.
Re: Tracking supermarket prices with Playwright
#169Nice writeup. I've been through similar problems that you have with my contact lens price comparison website https://lenspricer.com/ that I run in ~30 countries. I have found, like you, that websites changing their HTML is a pain. One of my biggest hurdles initially was matching products across 100+ websites. Even though you think a product has a unique name, everyone puts their own twist on it. Most can be handled w…
Do you support Canada?
Re: Tracking supermarket prices with Playwright
#170Nice writeup. I've been through similar problems that you have with my contact lens price comparison website https://lenspricer.com/ that I run in ~30 countries. I have found, like you, that websites changing their HTML is a pain. One of my biggest hurdles initially was matching products across 100+ websites. Even though you think a product has a unique name, everyone puts their own twist on it. Most can be handled w…
Doing the work we need. Every year I get fucked by my insurance company when buying a basic thing - contacts. Pricing is all over the place and coverage is usually 30% done by mail in reimbursement.
Thanks!