Live data from Hacker News

A Google bot scrapes pricing info by adding items to carts

wsj.com

91–100 of 291 posts

Re: A Google bot scrapes pricing info by adding items to carts

#91
Genuine question, is this not considered a DoS attack?

Let's imagine I have my online stock linked to limited physical items/assets, ex tickets for a show, which will get reserved for a period of time. This will be preventing genuine clients from buying them.

Re: A Google bot scrapes pricing info by adding items to carts

#92
post #63

Earlier quoted context omitted.

It's not rude at a consumer level, where (in general) you're at least considering making the purchase. It's arguably rude at a bot level, depending on the frequency, where there is 0% chance of conversion.

The entire purpose of the bot is to provide listings to consumers who are looking to buy. If it was consumer journalist doing it to get the price for a news article (in a for-profit publication) about the product, would it be “rude”? If not, how is it for Google bot?

Because bots will do it at a much larger scale than individual humans. The first law of web robotics applies here: the bot should not harm the website it's crawling, or through inaction allow it to come to harm.

I didn't read the article due to the paywall, but I assume that the problem is that the problem is that these goods are reserved for that (non)-customer until the shopping cart times out? That is directly costing the merchant money, either in lost sales or having to maintain extra inventory.

So yeah, that bot really should have been programmed to end the session with an empty basket one way or another.

Re: A Google bot scrapes pricing info by adding items to carts

#93
post #81

robots.txt, man, if you don't want search engines to visit certain part of your page, use robots.txt! Once heard a tale of an angry site owner calling Google (back when Google itself was novel) - Google deleted his whole website! Turned out he had "DELETE" button in each page, which generated plain GET request. So Googlebot visited the site, followed links to every page, and then of course followed every link that ge…

That has nothing to do with robots.txt, the problem is doing things in response to GET requests. I've said it before and I'll say it again: you do not do things on GET requests.

Re: A Google bot scrapes pricing info by adding items to carts

#94

Genuine question, is this not considered a DoS attack? Let's imagine I have my online stock linked to limited physical items/assets, ex tickets for a show, which will get reserved for a period of time. This will be preventing genuine clients from buying them.

Possibly it is lower traffic than a full on dos?

Re: A Google bot scrapes pricing info by adding items to carts

#95
post #86

For people saying this to calculate the final price with shipping and tax, it's not (or at least not entirely). It is for this new sales conversion dark pattern where prices aren't listed until you add to cart. Ebay sellers are particularly bad offenders: https://www.ebay.com/itm/Open-Box-Certified-Samsung-Galaxy-1...

What am I missing on here? That item has the price listed without having to Add To Cart.

The modal that pops up is not in the dom until you click the "See details" link, which has target="javascript:;". The "Add to cart" button is an actual link. I wouldn't be surprised if Google just doesn't want to run javascript to extract pricing information if it doesn't necessarily have to.

Re: A Google bot scrapes pricing info by adding items to carts

#96
post #81

robots.txt, man, if you don't want search engines to visit certain part of your page, use robots.txt! Once heard a tale of an angry site owner calling Google (back when Google itself was novel) - Google deleted his whole website! Turned out he had "DELETE" button in each page, which generated plain GET request. So Googlebot visited the site, followed links to every page, and then of course followed every link that ge…

[deleted]

Re: A Google bot scrapes pricing info by adding items to carts

#97
post #81

robots.txt, man, if you don't want search engines to visit certain part of your page, use robots.txt! Once heard a tale of an angry site owner calling Google (back when Google itself was novel) - Google deleted his whole website! Turned out he had "DELETE" button in each page, which generated plain GET request. So Googlebot visited the site, followed links to every page, and then of course followed every link that ge…

How do I use robots.txt to tell google to not add item to the shopping cart?

Erm... hide the shopping cart page behind robots.txt?

Re: A Google bot scrapes pricing info by adding items to carts

#98
post #94

Genuine question, is this not considered a DoS attack? Let's imagine I have my online stock linked to limited physical items/assets, ex tickets for a show, which will get reserved for a period of time. This will be preventing genuine clients from buying them.

Possibly it is lower traffic than a full on dos?

Yes, in regards to traffic. But it's still denying me from providing a service to real customers.

Re: A Google bot scrapes pricing info by adding items to carts

#99
post #81

robots.txt, man, if you don't want search engines to visit certain part of your page, use robots.txt! Once heard a tale of an angry site owner calling Google (back when Google itself was novel) - Google deleted his whole website! Turned out he had "DELETE" button in each page, which generated plain GET request. So Googlebot visited the site, followed links to every page, and then of course followed every link that ge…

How do I use robots.txt to tell google to not add item to the shopping cart?

Well, theoretically, your Add To Cart button could have an href with a path that’s banned in robots.txt, but overridden with JS.

But most online stores should be happy to have Google crawling their prices and showing up under the Shopping results.

Post reply on HN