A Google bot scrapes pricing info by adding items to carts • Hacker News Board

Re: A Google bot scrapes pricing info by adding items to carts

#81

Post by yongjik » Wed, Jul 01, 2020, 8:02 PM UTC

robots.txt, man, if you don't want search engines to visit certain part of your page, use robots.txt!

Once heard a tale of an angry site owner calling Google (back when Google itself was novel) - Google deleted his whole website! Turned out he had "DELETE" button in each page, which generated plain GET request. So Googlebot visited the site, followed links to every page, and then of course followed every link that generated GET requests - because they are supposed to be safe.

Don't be like that site owner.

Re: A Google bot scrapes pricing info by adding items to carts

#82

Post by mdoms » Wed, Jul 01, 2020, 8:05 PM UTC

saltedonion wrote: post #79

chrisan wrote: post #75

Earlier quoted context omitted.

How does that change by having the bot add items to the cart? You haven't solved anything You are still left with the same scenario as if the store listed the individual shipping price on the front page Google isn't going to know what other items you _might_ add to show you a "real" shipping cost

It matter because if you purchase multiple things your average shipping cost per item changes. If you only calculate shipping based on first item shipping cost it will be inaccurate.

But Google is only showing a result for a single item.

Re: A Google bot scrapes pricing info by adding items to carts

#83

Post by viraptor » Wed, Jul 01, 2020, 8:09 PM UTC

stingraycharles wrote: post #73

bronzeage wrote: post #15

Earlier quoted context omitted.

Makes you wonder whether the smart thing to do is just make it convenient for the bots to get the info out so they won't ruin your data and waste your bandwidth. Can't fight them, join them. Can't really stop free information flow.

They already do that, there’s all kinds of XML and JSON and whatnot standards to communicate product info, inventory, whatnot. The reason Google is doing this is because this information cannot be trusted all the time, there will always be bad actors. The process may eventually evolve in a cat-and-mouse game, where malicious e-commerce sites try to detect these Google crawlers and serve different price info to them,…

Given the possibility you get detected and it impacts your organic search ranking... I'm not sure any serious vendor would risk it. And if they do, let them burn.

Re: A Google bot scrapes pricing info by adding items to carts

#84

Post by soganess » Wed, Jul 01, 2020, 8:12 PM UTC

For people saying this to calculate the final price with shipping and tax, it's not (or at least not entirely). It is for this new sales conversion dark pattern where prices aren't listed until you add to cart.

Ebay sellers are particularly bad offenders: https://www.ebay.com/itm/Open-Box-Certified-Samsung-Galaxy-1...

Re: A Google bot scrapes pricing info by adding items to carts

#85

Post by an_opabinia » Wed, Jul 01, 2020, 8:13 PM UTC

soganess wrote: post #84

For people saying this to calculate the final price with shipping and tax, it's not (or at least not entirely). It is for this new sales conversion dark pattern where prices aren't listed until you add to cart. Ebay sellers are particularly bad offenders: https://www.ebay.com/itm/Open-Box-Certified-Samsung-Galaxy-1...

That clearly has a see details button that shows the price.

Re: A Google bot scrapes pricing info by adding items to carts

#86

Post by nkozyra » Wed, Jul 01, 2020, 8:14 PM UTC

soganess wrote: post #84

For people saying this to calculate the final price with shipping and tax, it's not (or at least not entirely). It is for this new sales conversion dark pattern where prices aren't listed until you add to cart. Ebay sellers are particularly bad offenders: https://www.ebay.com/itm/Open-Box-Certified-Samsung-Galaxy-1...

What am I missing on here? That item has the price listed without having to Add To Cart.

Re: A Google bot scrapes pricing info by adding items to carts

#87

Post by baybal2 » Wed, Jul 01, 2020, 8:17 PM UTC

Funny, a one quick gig I did in my college years was to write a shopping bot protection against "guaranteed lowest price" scraper like tigerdirect, or RFD.

Back then, the goal was exactly the opposite.

Re: A Google bot scrapes pricing info by adding items to carts

#88

Post by soganess » Wed, Jul 01, 2020, 8:19 PM UTC

an_opabinia wrote: post #85

soganess wrote: post #84

For people saying this to calculate the final price with shipping and tax, it's not (or at least not entirely). It is for this new sales conversion dark pattern where prices aren't listed until you add to cart. Ebay sellers are particularly bad offenders: https://www.ebay.com/itm/Open-Box-Certified-Samsung-Galaxy-1...

That clearly has a see details button that shows the price.

Most dark patterns have a non-intuitive way of circumventing them (the small-font faded-color "no, thank you" button comes to mind). That is Ebay's.

Other examples here: https://ux.stackexchange.com/questions/83050/price-too-low-t...

Amazon example from a few years ago: https://lh5.googleusercontent.com/ztyT6xTPaTr9TtP8LwlRJBE6RV...

Re: A Google bot scrapes pricing info by adding items to carts

#89

Post by gruez » Wed, Jul 01, 2020, 8:19 PM UTC

the_pwner224 wrote: post #62

jiofih wrote: post #40

Earlier quoted context omitted.

Maybe I haven’t done enough online shopping recently, but as far back as I remember this used to be the norm: enter postcode to calculate shipping, get precise final price without even adding to cart. Is it not the case anymore?

On many websites it still is. But recently many smaller independent stores use the Shopify platform where shipping is the penultimate step, before billing. You have to give address, email(!), phone number(!, mandatory), etc. before getting the price. I normally just use a fake email and number to get shipping price, and then do the actual checkout in incognito. Pretty sure if you do enter your real info and don't con…

I'm pretty sure most sites (eg. Amazon) does the same thing. Probably for the reason you mentioned: so they can have your contact information to send you spam later.

Re: A Google bot scrapes pricing info by adding items to carts

#90

Post by YetAnotherNick » Wed, Jul 01, 2020, 8:20 PM UTC

yongjik wrote: post #81

robots.txt, man, if you don't want search engines to visit certain part of your page, use robots.txt! Once heard a tale of an angry site owner calling Google (back when Google itself was novel) - Google deleted his whole website! Turned out he had "DELETE" button in each page, which generated plain GET request. So Googlebot visited the site, followed links to every page, and then of course followed every link that ge…

How do I use robots.txt to tell google to not add item to the shopping cart?