Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tracking supermarket prices with Playwright

sakisv.net

Post reply on HN
Print view View original

91–100 of 213 posts

Re: Tracking supermarket prices with Playwright

#91

Post by dawnerd »

barbazoo wrote: post #31
xyst wrote: post #29

Would be nice to have a price transparency of goods. It would make processes like this much more easier to track by store, and region. For example, compare the price of oat milk at different zip codes and grocery stores. Additionally track “shrinkflation” (same price but smaller portion). On that note, it seems you are tracking price but are you also checking the cost per gram (or ounce)? Manufacturer or store could…

Grocers not putting per unit prices on the label is a pet peeve of mine. I can’t imagine any purpose not rooted in customer hostility.

Or when they change what unit to display so you can’t easily cross compare.

Re: Tracking supermarket prices with Playwright

#92

Post by j45 »

brikym wrote: post #89
ustad wrote: post #87

Earlier quoted context omitted.

Can anyone comment how supermarkets exploit customer segmentation by updating prices? How do the time-poor and poor-poor people generally respond? “Often they'll segment on brand loyalty vs price sensitive people; There might be 3 popular brands of chocolate and every week only one of them will be sold at a fair price.”

Let's say there are three brands of some item. Each week one of the brands is rotated to $1 while the others are $2. And let's also suppose that the supermarket pays 80c per item. The smart shopper might only buy in bulk once every three weeks when his favourite brand at a lower price, or twitch to the cheapest brand every week. A hurried or lazy shopper might always pick their favourite brand every week. If they buy…

Bingo, an extra 50 cents to 2 dollars per item in a grocery order adds up quick, sooner or later.

Also, in-store pricing can be cheaper or not advertised as a sale, but encouraging online shopping can often be a higher price, even if it's pickup and not delivery.

"In-Store Pricing" on websites/apps are an interesting thing to track as well, it feels the more a grocery store goes towards "In-Store pricing', the higher it is.

This would be great to see in other countries.

Re: Tracking supermarket prices with Playwright

#93

Post by pcblues »

This is interesting because I believe the two major supermarkets in Australia can create a duopoly in anti-competitive pricing by just employing price analysis AI algorithms on each side and the algorithms will likely end up cooperating to maximise profit. This can probably be done legally through publicly obtained prices and illegally by sharing supply cost or profit per product data. The result is likely to be similar. Two trained AIs will maximise profit in weird ways using (super)multidimensional regression analysis (which is all AI is), and the consumer will pay for maximised profits to ostensible competitors. If the pricing data can be obtained like this, not much more is needed to implement a duopoly-focused pair of machine learning implementations.

Re: Tracking supermarket prices with Playwright

#94

Post by presentation »

latexr wrote: post #75
batata004 wrote: post #71

I created a similar website which got lots of interest in my city. I scrape even app and websites data using a single server at Linode with 2GB of RAM with 5 IPv4 and 1000 IPv6 (which is free) and every single product is scraped at most 40 minutes interval, never more than that with avg time of 25 minutes. I use curl impersonate and scrape JSON as much as possible because 90% of markets provide prices from Ajax calls…

> I scrape even app and websites data And then try to sell it back to businesses, even suggesting they use the data to train AI. You also make it sound like there’s a team manually doing all the work. https://www.economizafloripa.com.br/?q=parceria-comercial That whole page makes my view of the project go from “helpful tool for the people, to wrestle back control from corporations selling basic necessities” to “just…

It’s almost like people try to do valuable services for others in exchange for money.

Re: Tracking supermarket prices with Playwright

#95

Post by marginalia_nu »

haolez wrote: post #14

I heard that some e-commerce sites will not block scrappers, but poison the data shown to them (e.g. subtly wrong prices). Does anyone know more about this?

Yeah, by far the most reliable way of preventing bots is to silently poison the data. The harder you try to fight them in a visible fashion, the harder they become to detect. If you block them, they just come back with a hundred times as many IP addresses and u-a fingerprints.

Re: Tracking supermarket prices with Playwright

#96

Post by pcblues »

pcblues wrote: post #93

This is interesting because I believe the two major supermarkets in Australia can create a duopoly in anti-competitive pricing by just employing price analysis AI algorithms on each side and the algorithms will likely end up cooperating to maximise profit. This can probably be done legally through publicly obtained prices and illegally by sharing supply cost or profit per product data. The result is likely to be simi…

The word I was looking for was collusion, but done with software and without people-based collusion.

Re: Tracking supermarket prices with Playwright

#97

Post by eddyfromtheblok »

langsoul-com wrote: post #49

The hard thing is not scraping, but getting around the increasingly sophisticated blockers. You'll need to constantly rotate residential proxies (high rated) and make sure not to exhibit data scraping patterns. Some supermarkets don't show the network requests in the network tab, so cannot just get that api response. Even then, mitm attacks with mobile app (to see the network requests and data) will also get blocked…

Crowdsource it with a browser extension

Re: Tracking supermarket prices with Playwright

#98

Post by seaal »

gruez wrote: post #57
walterbell wrote: post #41

Earlier quoted context omitted.

Those who order grocery delivery online would benefit from price comparisons, because they can order from multiple stores at the same time. In addition, there's only one marketplace that has all the prices from different stores.

>Those who order grocery delivery online would benefit from price comparisons, because they can order from multiple stores at the same time. Not really, since the delivery fees/tips that you have to pay would eat up all the savings, unless maybe if you're buying for a family of 5 or something.

Instacart, Uber Eats, DoorDash all sell gift cards of $100 for $80 basically year round — when you combine that 20% with other promotions I often have deliveries that are cheaper than shopping in person.

Re: Tracking supermarket prices with Playwright

#99

Post by grafraf »

We have been doing it for the Swedish market in more than 8 years. We have a website https://www.matspar.se/ , where the customer can browse all the products of all major online stores, compare the prices and add the products they want to buy in the cart. The customer can in the end of the journey compare the total price of that cart (including shipping fee) and export the cart to the store they desire to order it.

I'm also one of the founders and the current CTO, so there been a lot of scraping and maintaining during the years. We are scraping over 30 million prices daily.

Re: Tracking supermarket prices with Playwright

#100

Post by showsover »

grafraf wrote: post #99

We have been doing it for the Swedish market in more than 8 years. We have a website https://www.matspar.se/ , where the customer can browse all the products of all major online stores, compare the prices and add the products they want to buy in the cart. The customer can in the end of the journey compare the total price of that cart (including shipping fee) and export the cart to the store they desire to order it. I…

Do you have a technical writeup of your scraping approach? I'd love to read more about the challenges and solutions for them.
Post reply on HN