A Google bot scrapes pricing info by adding items to carts • Hacker News Board

Re: A Google bot scrapes pricing info by adding items to carts

#121

Post by Alupis » Wed, Jul 01, 2020, 9:35 PM UTC

malux85 wrote: post #118

Alupis wrote: post #117

The real problem with this is from the merchant side of things. This bot generates thousands of "Abandoned Carts" on one of our sites... thousands... We send cart reminders to Abandoned Carts after a few days, sometimes with a coupon offer to complete checkout. This bot is responsible for thousands of bounced emails each week, which impacts our metrics with Mandrill among other things. Maybe we shouldn't care, but it…

Can’t you useragent sniff the bot and cut it off? If u want help coding this or advice happy to help (for free)

If you cut it off, you get penalized in Google Merchant Tools, and possibly have your product feed suppressed, which will dramatically impact your search visibility for both text and product searches. It can also impact your Google Ads if you link that with your product feed, and more.

So, effectively, no you cannot cut this bot off.

To make it worse, the bot doesn't always follow the same pattern. Sometimes slightly different names, addresses, etc.

We initially thought it was fraud attempts, but none of them actually attempt a checkout. They just enter all their info on the checkout page, get the final quote, and bail.

It would have been nice if Google told people about this instead of it just happening. Or allowed you to schedule a time slot for it to do what it's going to do.

Re: A Google bot scrapes pricing info by adding items to carts

#122

Post by malux85 » Wed, Jul 01, 2020, 9:37 PM UTC

Alupis wrote: post #121

malux85 wrote: post #118

Earlier quoted context omitted.

Can’t you useragent sniff the bot and cut it off? If u want help coding this or advice happy to help (for free)

If you cut it off, you get penalized in Google Merchant Tools, and possibly have your product feed suppressed, which will dramatically impact your search visibility for both text and product searches. It can also impact your Google Ads if you link that with your product feed, and more. So, effectively, no you cannot cut this bot off. To make it worse, the bot doesn't always follow the same pattern. Sometimes slightly…

Good lord, that sucks, hopefully a google engineer is in this thread...

Re: A Google bot scrapes pricing info by adding items to carts

#123

Post by vmateixeira » Wed, Jul 01, 2020, 9:37 PM UTC

Mizza wrote: post #109

vmateixeira wrote: post #91

Genuine question, is this not considered a DoS attack? Let's imagine I have my online stock linked to limited physical items/assets, ex tickets for a show, which will get reserved for a period of time. This will be preventing genuine clients from buying them.

I'm thinking - if I forbid this in my site's Terms of Service, will DoJ go after Google for CFAA violations like they did to Aaron?

Yeah.. probably depend$ on how loud you can make yourself heard..

RIP Aaron

Re: A Google bot scrapes pricing info by adding items to carts

#124

Post by webmaven » Wed, Jul 01, 2020, 9:43 PM UTC

justinwp wrote: post #27

Protip: You will often get a discount coupon if you go through most of the checkout process(need to provide email), but wait a couple days. Many stores automate abandoned checkout promotions.

For a while there were registrars that gave a discount when you abandoned your cart.

Re: A Google bot scrapes pricing info by adding items to carts

#125

Post by miohtama » Wed, Jul 01, 2020, 9:44 PM UTC

whoisjuan wrote: post #8

This bot is simply trying to get the final price (with tax and shipping) which is ridiculous because e-commerce storefronts should do that in the first place without going through the whole checkout process. I always have found that kind of shady but it's probably known to increase conversions. What I found interesting is that this an open attack vector for e-commerces. Multiple bots can hit a website and start addin…

All EU prices must be tax included.

All hospitality prices must include cleaning and service fees.

It is only the US were hidden fees and charges may apply and the price regulation, if any, is more tilted towards corporations.

Only upfront shipping charge is tricky because it depends on so many factors.

Re: A Google bot scrapes pricing info by adding items to carts

#126

Post by ksk » Wed, Jul 01, 2020, 9:45 PM UTC

amelius wrote: post #112

It would be cool if Google could manage to become a storefront for the entire web, thereby eliminating Amazon.

Between the two, personally, I would rather an advertising company like Google be eliminated, or at-least be regulated to protect user privacy. For me, it's easy enough to avoid Amazon, harder to avoid Google since every website uses them to spy on users.

Re: A Google bot scrapes pricing info by adding items to carts

#127

Post by Xelbair » Wed, Jul 01, 2020, 9:49 PM UTC

chrisan wrote: post #75

mmcconnell1618 wrote: post #43

Earlier quoted context omitted.

Not all shipping charges can be calculated ahead of time. For example, you may offer free shipping on orders over $50. You may charge $9.99 for the first item, $5.99 for each additional item. You may charge by weight of the whole order. You may have oversized items or packages that can be combined to reduce shipping charges. Some items may ship together as OTR Freight, while others can go via the local postal service…

How does that change by having the bot add items to the cart? You haven't solved anything You are still left with the same scenario as if the store listed the individual shipping price on the front page Google isn't going to know what other items you _might_ add to show you a "real" shipping cost

And what's even more interesting - human would do exactly the same thing.

Add items to card, check the total price and then decide whether to buy it(i remember trying to order some stuff form one japanese plamo store - and it didn't provide exact prices before checkout. I went through the process, but even the cheapest option for delivery was way too high - as 2x price of the whole order)

Re: A Google bot scrapes pricing info by adding items to carts

#128

Post by px1999 » Wed, Jul 01, 2020, 9:50 PM UTC

Alupis wrote: post #117

The real problem with this is from the merchant side of things. This bot generates thousands of "Abandoned Carts" on one of our sites... thousands... We send cart reminders to Abandoned Carts after a few days, sometimes with a coupon offer to complete checkout. This bot is responsible for thousands of bounced emails each week, which impacts our metrics with Mandrill among other things. Maybe we shouldn't care, but it…

Do your users consent to contact before you send them reminders or coupons? If not, you've earned your bounce rate.

Re: A Google bot scrapes pricing info by adding items to carts

#129

Post by cortesoft » Wed, Jul 01, 2020, 9:53 PM UTC

Alupis wrote: post #117

The real problem with this is from the merchant side of things. This bot generates thousands of "Abandoned Carts" on one of our sites... thousands... We send cart reminders to Abandoned Carts after a few days, sometimes with a coupon offer to complete checkout. This bot is responsible for thousands of bounced emails each week, which impacts our metrics with Mandrill among other things. Maybe we shouldn't care, but it…

I personally find the 'email abandoned carts' behavior to be a dark pattern

Re: A Google bot scrapes pricing info by adding items to carts

#130

Post by dzmien » Wed, Jul 01, 2020, 9:54 PM UTC

Alupis wrote: post #121

malux85 wrote: post #118

Earlier quoted context omitted.

Can’t you useragent sniff the bot and cut it off? If u want help coding this or advice happy to help (for free)

If you cut it off, you get penalized in Google Merchant Tools, and possibly have your product feed suppressed, which will dramatically impact your search visibility for both text and product searches. It can also impact your Google Ads if you link that with your product feed, and more. So, effectively, no you cannot cut this bot off. To make it worse, the bot doesn't always follow the same pattern. Sometimes slightly…

I wonder why Google didn't include a "cleanup" routine that empties the shopping cart after the data is collected. It seems like it would be a trivial thing to do, unless I am missing something. I guess the answer is because it would not benefit them in any way.