A Google bot scrapes pricing info by adding items to carts • Hacker News Board

Re: A Google bot scrapes pricing info by adding items to carts

#101

Post by rubyron » Wed, Jul 01, 2020, 8:39 PM UTC

disposekinetics wrote: post #31

tacon wrote: post #18

Would it be too much for Google to program the bot to get the final price, and then delete all the items from the cart? Seems rather rude, even for Google.

Is abandoning a cart really rude behavior? I sometimes do it just to see if they'll spam me as a test of if I want to do business with a site.

An abandoned cart reminder email sent a few hours later has a ridiculously high conversion rate - around 15% in my experience. Online vendors aren’t going to stop that practice, especially when the big e-commerce platforms make it easy to do.

Re: A Google bot scrapes pricing info by adding items to carts

#102

Post by jogjayr » Wed, Jul 01, 2020, 8:39 PM UTC

the_pwner224 wrote: post #65

rkagerer wrote: post #42

Are there legal implications to Google bots transacting with websites under false pretenses? I mean their normal web crawler identifies itself as such. Here, I feel like they're committing (very) minor fraud by putting in fake shopper information and actively hiding their identity. Not a big deal if it were just some Joe Schmoe somewhere, but at their scale might it border on harassment? The robot equivalent of a pra…

Probably a violation of the CFAA. Lots of people hate it because they think it's overreaching, and lots of companies use it to legally threaten scrapers and security research. But in this case Google is doing mass unauthorized use of other people's computers.

I think that's outdated information. ToS violations aren't prosecutable under CFAA since April.[1]

1. https://www.eff.org/deeplinks/2020/04/federal-judge-rules-it...

Re: A Google bot scrapes pricing info by adding items to carts

#103

Post by Spivak » Wed, Jul 01, 2020, 8:43 PM UTC

inetknght wrote: post #23

kqr wrote: post #17

Earlier quoted context omitted.

Or quietly detect the bots and feed them junk data after they've gone through the hoops. Not saying it's the better option, but knowing the business maybe the more likely one.

> Or quietly detect the bots and feed them junk data Just a few customers with alternative browsers being detected as "bots" will poison your reputation and income stream.

As someone who uses an alternative browser… I kinda doubt it. As a group we wouldn’t even move the analytics needle let alone revenue.

Re: A Google bot scrapes pricing info by adding items to carts

#104

Post by Spivak » Wed, Jul 01, 2020, 8:45 PM UTC

alpha_squared wrote: post #52

cglace wrote: post #47

Earlier quoted context omitted.

How do you show a final price if you don't have all the information needed: tax locality, shipping preference, total cart value discounts etc.

If I'm remembering right, Best Buy used to have "deals" on items that they "couldn't show you" until item was in cart. They may still be doing this. Best Buy's justification for it was that its agreements with manufacturers prevented it from displaying items below certain prices on their site. I'd never seen this elsewhere to know how pervasive these agreements were (or if Best Buy was just taking losses on certain i…

KitchenAid is famous for these kinds of agreements as a measure to ensure the public perception of their value doesn’t go down whenever there are sales.

Re: A Google bot scrapes pricing info by adding items to carts

#105

Post by hrktb » Wed, Jul 01, 2020, 8:49 PM UTC

chrisan wrote: post #75

mmcconnell1618 wrote: post #43

Earlier quoted context omitted.

Not all shipping charges can be calculated ahead of time. For example, you may offer free shipping on orders over $50. You may charge $9.99 for the first item, $5.99 for each additional item. You may charge by weight of the whole order. You may have oversized items or packages that can be combined to reduce shipping charges. Some items may ship together as OTR Freight, while others can go via the local postal service…

How does that change by having the bot add items to the cart? You haven't solved anything You are still left with the same scenario as if the store listed the individual shipping price on the front page Google isn't going to know what other items you _might_ add to show you a "real" shipping cost

I’d assume parent’s point is regarding the “which is ridiculous because e-commerce storefronts should do that in the first place without going through the whole checkout process.” part.

There’s a lot of legitimate case were showing shipping price upfront is just not doable or valuable to the customer.

BTW there are a surprising amount of shops for specialized goods that won’t even list the final price at the end. The customer places an order, and they update it with a finalized price after a human looks at the content, and from there the customer is free to pay the transaction or give up the order.

Re: A Google bot scrapes pricing info by adding items to carts

#106

Post by Ruthalas » Wed, Jul 01, 2020, 8:59 PM UTC

gruez wrote: post #14

whoisjuan wrote: post #8

This bot is simply trying to get the final price (with tax and shipping) which is ridiculous because e-commerce storefronts should do that in the first place without going through the whole checkout process. I always have found that kind of shady but it's probably known to increase conversions. What I found interesting is that this an open attack vector for e-commerces. Multiple bots can hit a website and start addin…

> This bot is simply trying to get the final price (with tax and shipping) which is ridiculous because e-commerce storefronts should do that in the first place without going through the whole checkout process. It's usually not possible because you don't know how much the shipping + taxes are until the customer enters the billing information.

I will bail from the purchasing process on sites that are unwilling to give me a final price before I enter payment information.

A postal code should suffice, and I'm not providing more personal information if the site is unwilling to say what I'll be charged upfront.

Re: A Google bot scrapes pricing info by adding items to carts

#107

Post by [deleted] » Wed, Jul 01, 2020, 9:14 PM UTC

the_pwner224 wrote: post #62

jiofih wrote: post #40

Earlier quoted context omitted.

Maybe I haven’t done enough online shopping recently, but as far back as I remember this used to be the norm: enter postcode to calculate shipping, get precise final price without even adding to cart. Is it not the case anymore?

On many websites it still is. But recently many smaller independent stores use the Shopify platform where shipping is the penultimate step, before billing. You have to give address, email(!), phone number(!, mandatory), etc. before getting the price. I normally just use a fake email and number to get shipping price, and then do the actual checkout in incognito. Pretty sure if you do enter your real info and don't con…

[deleted]

Re: A Google bot scrapes pricing info by adding items to carts

#108

Post by mkl » Wed, Jul 01, 2020, 9:14 PM UTC

stronglikedan wrote: post #28

gruez wrote: post #14

Earlier quoted context omitted.

> This bot is simply trying to get the final price (with tax and shipping) which is ridiculous because e-commerce storefronts should do that in the first place without going through the whole checkout process. It's usually not possible because you don't know how much the shipping + taxes are until the customer enters the billing information.

I just used a website that had a simple form at the bottom of the cart. It had one text input for the postal code, and a button to get the rates to that postal code based on what was in my cart. IMHO, this is how it's done right, since all you need to know is general location and weight.

Is that US only? You need country too.

Re: A Google bot scrapes pricing info by adding items to carts

#109

Post by Mizza » Wed, Jul 01, 2020, 9:17 PM UTC

vmateixeira wrote: post #91

Genuine question, is this not considered a DoS attack? Let's imagine I have my online stock linked to limited physical items/assets, ex tickets for a show, which will get reserved for a period of time. This will be preventing genuine clients from buying them.

I'm thinking - if I forbid this in my site's Terms of Service, will DoJ go after Google for CFAA violations like they did to Aaron?

Re: A Google bot scrapes pricing info by adding items to carts

#110

Post by crazygringo » Wed, Jul 01, 2020, 9:17 PM UTC

whoisjuan wrote: post #8

This bot is simply trying to get the final price (with tax and shipping) which is ridiculous because e-commerce storefronts should do that in the first place without going through the whole checkout process. I always have found that kind of shady but it's probably known to increase conversions. What I found interesting is that this an open attack vector for e-commerces. Multiple bots can hit a website and start addin…

Not only that, but certain brands on certain sites won't show the price, with that message "add to cart to see the price!"

I've heard varying explanations as to why, but at the end of the day it doesn't matter. Adding to the cart is the only way to scrape the price.