A Google bot scrapes pricing info by adding items to carts • Hacker News Board

Re: A Google bot scrapes pricing info by adding items to carts

#71

Post by shadowgovt » Wed, Jul 01, 2020, 7:50 PM UTC

the_pwner224 wrote: post #65

rkagerer wrote: post #42

Are there legal implications to Google bots transacting with websites under false pretenses? I mean their normal web crawler identifies itself as such. Here, I feel like they're committing (very) minor fraud by putting in fake shopper information and actively hiding their identity. Not a big deal if it were just some Joe Schmoe somewhere, but at their scale might it border on harassment? The robot equivalent of a pra…

Probably a violation of the CFAA. Lots of people hate it because they think it's overreaching, and lots of companies use it to legally threaten scrapers and security research. But in this case Google is doing mass unauthorized use of other people's computers.

If I'm doing price comparison between online vendors, I will---as a human---put some items in the cart and get right to the edge of checkout to determine what my final bill would be. I may not close the sale if I'm looking at a better option elsewhere.

How is what I'm doing materially different from what Google's doing? Is scale a factor that matters for CFAA?

Re: A Google bot scrapes pricing info by adding items to carts

#72

Post by whoisjuan » Wed, Jul 01, 2020, 7:51 PM UTC

alpha_squared wrote: post #52

cglace wrote: post #47

Earlier quoted context omitted.

How do you show a final price if you don't have all the information needed: tax locality, shipping preference, total cart value discounts etc.

If I'm remembering right, Best Buy used to have "deals" on items that they "couldn't show you" until item was in cart. They may still be doing this. Best Buy's justification for it was that its agreements with manufacturers prevented it from displaying items below certain prices on their site. I'd never seen this elsewhere to know how pervasive these agreements were (or if Best Buy was just taking losses on certain i…

They still do it.

Re: A Google bot scrapes pricing info by adding items to carts

#73

Post by stingraycharles » Wed, Jul 01, 2020, 7:51 PM UTC

bronzeage wrote: post #15

whoisjuan wrote: post #8

This bot is simply trying to get the final price (with tax and shipping) which is ridiculous because e-commerce storefronts should do that in the first place without going through the whole checkout process. I always have found that kind of shady but it's probably known to increase conversions. What I found interesting is that this an open attack vector for e-commerces. Multiple bots can hit a website and start addin…

Makes you wonder whether the smart thing to do is just make it convenient for the bots to get the info out so they won't ruin your data and waste your bandwidth. Can't fight them, join them. Can't really stop free information flow.

They already do that, there’s all kinds of XML and JSON and whatnot standards to communicate product info, inventory, whatnot. The reason Google is doing this is because this information cannot be trusted all the time, there will always be bad actors.

The process may eventually evolve in a cat-and-mouse game, where malicious e-commerce sites try to detect these Google crawlers and serve different price info to them, but let’s hope it doesn’t get this far.

Re: A Google bot scrapes pricing info by adding items to carts

#74

Post by dragonwriter » Wed, Jul 01, 2020, 7:51 PM UTC

jawns wrote: post #63

disposekinetics wrote: post #31

Earlier quoted context omitted.

Is abandoning a cart really rude behavior? I sometimes do it just to see if they'll spam me as a test of if I want to do business with a site.

It's not rude at a consumer level, where (in general) you're at least considering making the purchase. It's arguably rude at a bot level, depending on the frequency, where there is 0% chance of conversion.

The entire purpose of the bot is to provide listings to consumers who are looking to buy.

If it was consumer journalist doing it to get the price for a news article (in a for-profit publication) about the product, would it be “rude”? If not, how is it for Google bot?

Re: A Google bot scrapes pricing info by adding items to carts

#75

Post by chrisan » Wed, Jul 01, 2020, 7:53 PM UTC

mmcconnell1618 wrote: post #43

whoisjuan wrote: post #8

This bot is simply trying to get the final price (with tax and shipping) which is ridiculous because e-commerce storefronts should do that in the first place without going through the whole checkout process. I always have found that kind of shady but it's probably known to increase conversions. What I found interesting is that this an open attack vector for e-commerces. Multiple bots can hit a website and start addin…

Not all shipping charges can be calculated ahead of time. For example, you may offer free shipping on orders over $50. You may charge $9.99 for the first item, $5.99 for each additional item. You may charge by weight of the whole order. You may have oversized items or packages that can be combined to reduce shipping charges. Some items may ship together as OTR Freight, while others can go via the local postal service…

How does that change by having the bot add items to the cart? You haven't solved anything

You are still left with the same scenario as if the store listed the individual shipping price on the front page

Google isn't going to know what other items you _might_ add to show you a "real" shipping cost

Re: A Google bot scrapes pricing info by adding items to carts

#76

Post by shadowgovt » Wed, Jul 01, 2020, 7:53 PM UTC

inetknght wrote: post #25

bluGill wrote: post #20

Earlier quoted context omitted.

They need to be non traceable. If I'm doing something underhanded with pricing information I want to detect Google and other such bots and give them different information.

You really think it's wise to lie to your customers?

I mean... The vast bulk of digital capitalism is information asymmetries. Lies by omission are nearly the currency of e-commerce.

Is it wise to intentionally refrain from offering the same price to a human you'd offer to the Google-bot? Possibly. Depends on your goals.

Re: A Google bot scrapes pricing info by adding items to carts

#77

Post by withinboredom » Wed, Jul 01, 2020, 7:53 PM UTC

maltelandwehr wrote: post #50

vmception wrote: post #2

That sparked a funny idea in my head, what if we tricked product managers industry wide to follow KPIs and A/B tests that resulted in a better user experience for consumers, instead of experiences that coincidentally slightly upticked "engagement". Because it seems like this mystery shopper is already doing that.

„Messing up your competitors A/B test“ is not unheard of as a tactic in highly competitive ecommerce settings.

Do software engineers actually implement that? That seems pretty immoral. I'd rather let them run the a/b test and steal whatever solution they end up with.

Re: A Google bot scrapes pricing info by adding items to carts

#78

Post by whoisjuan » Wed, Jul 01, 2020, 7:54 PM UTC

bravoetch wrote: post #4

TL;DR - it's a google bot

This is the problem that I have with HN editorializing titles. This comment made perfect sense and was useful before they changed the title, but now that is changed it looks like the poster is an idiot who is just saying what the title says and some people downvote it.

I know HN is not very keen on adding features, but this is one that is missing for the sake of transparency (seeing if the original title was editorialized)

I understand that the original title was click-bait trash and this one makes sense, but it would be nice to understand how it changed so certain comments don’t get de-contextualized.

But I guess is the same problem with editing comments.

Re: A Google bot scrapes pricing info by adding items to carts

#79

Post by saltedonion » Wed, Jul 01, 2020, 7:59 PM UTC

chrisan wrote: post #75

mmcconnell1618 wrote: post #43

Earlier quoted context omitted.

Not all shipping charges can be calculated ahead of time. For example, you may offer free shipping on orders over $50. You may charge $9.99 for the first item, $5.99 for each additional item. You may charge by weight of the whole order. You may have oversized items or packages that can be combined to reduce shipping charges. Some items may ship together as OTR Freight, while others can go via the local postal service…

How does that change by having the bot add items to the cart? You haven't solved anything You are still left with the same scenario as if the store listed the individual shipping price on the front page Google isn't going to know what other items you _might_ add to show you a "real" shipping cost

It matter because if you purchase multiple things your average shipping cost per item changes. If you only calculate shipping based on first item shipping cost it will be inaccurate.

Re: A Google bot scrapes pricing info by adding items to carts

#80

Post by enriquto » Wed, Jul 01, 2020, 8:01 PM UTC

gruez wrote: post #14

whoisjuan wrote: post #8

This bot is simply trying to get the final price (with tax and shipping) which is ridiculous because e-commerce storefronts should do that in the first place without going through the whole checkout process. I always have found that kind of shady but it's probably known to increase conversions. What I found interesting is that this an open attack vector for e-commerces. Multiple bots can hit a website and start addin…

> This bot is simply trying to get the final price (with tax and shipping) which is ridiculous because e-commerce storefronts should do that in the first place without going through the whole checkout process. It's usually not possible because you don't know how much the shipping + taxes are until the customer enters the billing information.

> It's usually not possible because you don't know how much the shipping + taxes are until the customer enters the billing information.

Sure, they show you a different content depending on your IP address and lots of shady heuristics, but when it comes to estimate a shipping cost, it is absolutely impossible: you can just be anywhere, who knows where you are. I say bullshit.