Live data from Hacker News

A Google bot scrapes pricing info by adding items to carts

wsj.com

131–140 of 291 posts

Re: A Google bot scrapes pricing info by adding items to carts

#131

Genuine question, is this not considered a DoS attack? Let's imagine I have my online stock linked to limited physical items/assets, ex tickets for a show, which will get reserved for a period of time. This will be preventing genuine clients from buying them.

You can always update your robots.txt or block the Googlebot UA. (lol)

Re: A Google bot scrapes pricing info by adding items to carts

#132
post #128
post #117

The real problem with this is from the merchant side of things. This bot generates thousands of "Abandoned Carts" on one of our sites... thousands... We send cart reminders to Abandoned Carts after a few days, sometimes with a coupon offer to complete checkout. This bot is responsible for thousands of bounced emails each week, which impacts our metrics with Mandrill among other things. Maybe we shouldn't care, but it…

Do your users consent to contact before you send them reminders or coupons? If not, you've earned your bounce rate.

Of course we have consent. Not sure what kind of question that is?

Violate CAN-SPAM Act and risk a $16,000 fine per instance? No legit business is going to do that.

Just ask Papa Johns how painful those fines/settlements can be[1].

[1] https://topclassactions.com/lawsuit-settlements/lawsuit-news...

Re: A Google bot scrapes pricing info by adding items to carts

#133

Earlier quoted context omitted.

„Messing up your competitors A/B test“ is not unheard of as a tactic in highly competitive ecommerce settings.

Do software engineers actually implement that? That seems pretty immoral. I'd rather let them run the a/b test and steal whatever solution they end up with.

Eh, I am sure you could convince yourself it isn't immoral... everyone in HN seems to think things like google analytics are bad because of the privacy implications, and doesn't have a problem blocking them (which would also 'mess with a/b tests'). You could just argue that you are hindering their user spying.

Not a great argument, but good enough to allow a developer to sleep at night.

Re: A Google bot scrapes pricing info by adding items to carts

#134
post #117

The real problem with this is from the merchant side of things. This bot generates thousands of "Abandoned Carts" on one of our sites... thousands... We send cart reminders to Abandoned Carts after a few days, sometimes with a coupon offer to complete checkout. This bot is responsible for thousands of bounced emails each week, which impacts our metrics with Mandrill among other things. Maybe we shouldn't care, but it…

I personally find the 'email abandoned carts' behavior to be a dark pattern

Dark pattern or not, it's super effective - particularly when accompanied with a coupon ;)

Besides, the user is opting-in to receiving these emails. They don't have to provide an email address - so some are probably playing the game and seeing if they get a coupon or not.

As an aside - if the internet worked the way SV hipster brogrammers thought it should work, nobody would use it.

Yes, ads are crazy effective - you can ignore or block them, we don't care because enough people don't block them and are happy to click.

Yes, emails are crazy effective - you can ignore or opt-out or never opt-in, we don't care, you just cost us money if you're not engaged anyway so we'd rather you not be on our mailing list.

Re: A Google bot scrapes pricing info by adding items to carts

#135
When and why did news cease being news and start being short stories and opinion? This entire article could have been cut down to the last few paragraphs and nothing of value would have been lost.

Look at The New York Times in 1921 [0]. Generally the stories are factual and to the point. The entire front page seems to be pure news. There's very little storytelling here, at most there are a few timelines of events.

Look at The New York Times today [1]. There's a bunch of factual and useful Coronavirus information but ~15% of the page is dedicated to "Opinion", the second article appears to be pure speculation, the third article is a bunch of storytime fluff around a little bit of news and the front page has a mix of actual news and opinion pieces being passed off as news.

When did this happen? Why? Did people lose interest in actual news? Is there less actual news to report?

Perhaps this is regional? Take for example the story about the San Quentin prison. NYTimes [2] has the same drawn out nonsense as this Google story while Aljazeera [3] adds a lot of background but sticks to factual reporting.

[0]: https://archive.org/details/NYTimes_jul16_31_1921

[1]: http://archive.is/oiiXU

[2]: https://www.nytimes.com/2020/06/30/us/san-quentin-prison-cor...

[3]: https://www.aljazeera.com/news/2020/07/san-quentin-prison-se...

Re: A Google bot scrapes pricing info by adding items to carts

#136
post #121

Earlier quoted context omitted.

Can’t you useragent sniff the bot and cut it off? If u want help coding this or advice happy to help (for free)

If you cut it off, you get penalized in Google Merchant Tools, and possibly have your product feed suppressed, which will dramatically impact your search visibility for both text and product searches. It can also impact your Google Ads if you link that with your product feed, and more. So, effectively, no you cannot cut this bot off. To make it worse, the bot doesn't always follow the same pattern. Sometimes slightly…

It sounds like you interpreted malux85 suggestion as cutting the bot off from putting things in the cart altogether. What I understood is that, given it's recognizable by user agent, those carts can be marked as created by a bot to exclude it from statistics and reminder mails only.

Re: A Google bot scrapes pricing info by adding items to carts

#137
post #130
post #121

Earlier quoted context omitted.

If you cut it off, you get penalized in Google Merchant Tools, and possibly have your product feed suppressed, which will dramatically impact your search visibility for both text and product searches. It can also impact your Google Ads if you link that with your product feed, and more. So, effectively, no you cannot cut this bot off. To make it worse, the bot doesn't always follow the same pattern. Sometimes slightly…

I wonder why Google didn't include a "cleanup" routine that empties the shopping cart after the data is collected. It seems like it would be a trivial thing to do, unless I am missing something. I guess the answer is because it would not benefit them in any way.

We'd still have an abandoned cart, since the session was created and held some sort of data - but it would be far less disruptive for sure.

Empty Abandoned Carts are useless anyway (for stats and other things - we track bounce rates in other ways), so that would be a large improvement from what is going on right now.

Re: A Google bot scrapes pricing info by adding items to carts

#138
post #40

Earlier quoted context omitted.

Maybe I haven’t done enough online shopping recently, but as far back as I remember this used to be the norm: enter postcode to calculate shipping, get precise final price without even adding to cart. Is it not the case anymore?

On many websites it still is. But recently many smaller independent stores use the Shopify platform where shipping is the penultimate step, before billing. You have to give address, email(!), phone number(!, mandatory), etc. before getting the price. I normally just use a fake email and number to get shipping price, and then do the actual checkout in incognito. Pretty sure if you do enter your real info and don't con…

Phone number is required or otherwise highly encouraged by some shippers, like FedEx.

Re: A Google bot scrapes pricing info by adding items to carts

#139
post #117

The real problem with this is from the merchant side of things. This bot generates thousands of "Abandoned Carts" on one of our sites... thousands... We send cart reminders to Abandoned Carts after a few days, sometimes with a coupon offer to complete checkout. This bot is responsible for thousands of bounced emails each week, which impacts our metrics with Mandrill among other things. Maybe we shouldn't care, but it…

>We send cart reminders to Abandoned Carts after a few days, sometimes with a coupon offer to complete checkout.

I consider this spammy behaviour, and mark the emails as such. I can only hope this discourages such practices in the future.

Re: A Google bot scrapes pricing info by adding items to carts

#140
post #135

When and why did news cease being news and start being short stories and opinion? This entire article could have been cut down to the last few paragraphs and nothing of value would have been lost. Look at The New York Times in 1921 [0]. Generally the stories are factual and to the point. The entire front page seems to be pure news. There's very little storytelling here, at most there are a few timelines of events. Lo…

Opinion masquerading as news sells, see Fox News, MSNBC, et al.
Post reply on HN