A Google bot scrapes pricing info by adding items to carts • Hacker News Board

Re: A Google bot scrapes pricing info by adding items to carts

#201

Post by mmcconnell1618 » Thu, Jul 02, 2020, 1:01 AM UTC

bhuber wrote: post #190

mmcconnell1618 wrote: post #43

Earlier quoted context omitted.

Not all shipping charges can be calculated ahead of time. For example, you may offer free shipping on orders over $50. You may charge $9.99 for the first item, $5.99 for each additional item. You may charge by weight of the whole order. You may have oversized items or packages that can be combined to reduce shipping charges. Some items may ship together as OTR Freight, while others can go via the local postal service…

Even assuming you already know the customer's shipping address and ignore the multiple-items problem, this is still difficult to accomplish from a computational complexity perspective. Calculating shipping cost is likely at least an order of magnitude more expensive than simply looking up list prices in a database - you have to look up the customer's address, go through a bunch of tax rules, figure out the shipping c…

In many cases, this is not only computationally expensive on the server, it also requires one or more requests to external APIs which further slows things down. Imagine needing to query the API of your tax vendor, then also your shipping provider of choice for every single item displayed on a page. Even if you did this client side, asynchronously, it would be a lot of extra requests for something that most shoppers won't even pay attention to.

I ran an ecommerce platform company for many years and you had merchants with very complex shipping and tax schemes, or you had merchants that made it super simple with a basic rate table. The complex merchants had margin on every order at the cost of processing external API calls. The simple rate table merchants had great margin on some, lost money on others but were happy with their average shipping margin.

Re: A Google bot scrapes pricing info by adding items to carts

#202

Post by three_seagrass » Thu, Jul 02, 2020, 1:03 AM UTC

mmcconnell1618 wrote: post #43

whoisjuan wrote: post #8

This bot is simply trying to get the final price (with tax and shipping) which is ridiculous because e-commerce storefronts should do that in the first place without going through the whole checkout process. I always have found that kind of shady but it's probably known to increase conversions. What I found interesting is that this an open attack vector for e-commerces. Multiple bots can hit a website and start addin…

Not all shipping charges can be calculated ahead of time. For example, you may offer free shipping on orders over $50. You may charge $9.99 for the first item, $5.99 for each additional item. You may charge by weight of the whole order. You may have oversized items or packages that can be combined to reduce shipping charges. Some items may ship together as OTR Freight, while others can go via the local postal service…

Is Google indexing the shipping cost?

I don't see how this is relevant because ecommerce sites will change the price in the cart (or reveal it) before shipping is even calculated.

Re: A Google bot scrapes pricing info by adding items to carts

#203

Post by dragonwriter » Thu, Jul 02, 2020, 1:05 AM UTC

Youden wrote: post #135

When and why did news cease being news and start being short stories and opinion? This entire article could have been cut down to the last few paragraphs and nothing of value would have been lost. Look at The New York Times in 1921 [0]. Generally the stories are factual and to the point. The entire front page seems to be pure news. There's very little storytelling here, at most there are a few timelines of events. Lo…

> When and why did news cease being news and start being short stories and opinion?

As radio and then TV (both of which favor emotion over information) displaced print as the major avenues for “news”, shaping expectations even of print media.

Re: A Google bot scrapes pricing info by adding items to carts

#204

Post by randomdude402 » Thu, Jul 02, 2020, 1:17 AM UTC

alpha_squared wrote: post #52

cglace wrote: post #47

Earlier quoted context omitted.

How do you show a final price if you don't have all the information needed: tax locality, shipping preference, total cart value discounts etc.

If I'm remembering right, Best Buy used to have "deals" on items that they "couldn't show you" until item was in cart. They may still be doing this. Best Buy's justification for it was that its agreements with manufacturers prevented it from displaying items below certain prices on their site. I'd never seen this elsewhere to know how pervasive these agreements were (or if Best Buy was just taking losses on certain i…

I used to build and manage ecomm sites. We had several manufacturers/brands that we had to agree not to openly display a retail price below a certain amount.

Incidentally, for almost every brand in our industry, that number was 1.8X listed wholesale.

We could sell for less, but not list a price of less. Implementing "add to cart to see price" was good enough at the time to keep them happy.

Re: A Google bot scrapes pricing info by adding items to carts

#205

Post by fennecfoxen » Thu, Jul 02, 2020, 1:28 AM UTC

krustyburger wrote: post #188

supernova87a wrote: post #174

Earlier quoted context omitted.

Maybe you don't know this, but the "A-hed" article of the WSJ is the humorous, light-hearted take on some cultural phenomenon that appears every couple of days. It's got a distinct separation (graphically) from the rest of the news, and is written not to be taken too seriously. (It's not so apparent in the online version, if you haven't read it before). So you don't have to worry that it's some broad decline in journ…

All of the links to the A-hed section in the WSJ article link to a 404. Is there an updated link?

https://www.wsj.com/news/types/a-hed works for me right now, as do all of these:

https://www.wsj.com/articles/baseball-stadiums-major-league-... "Baseball Stadiums Are Closed to Fans—but This Guy’s Balcony Is Open for Business"

https://www.wsj.com/articles/americans-ponder-how-to-throw-a... "Americans Ponder How to Throw a Party"

https://www.wsj.com/articles/how-do-doctors-treating-coronav... "How Do Doctors Treating Coronavirus Relax? By Playing the Game ‘Pandemic’"

https://www.wsj.com/articles/a-scientist-turned-the-coronavi... "A Scientist Turned the Coronavirus Into Music—Here’s What It Sounds Like"

https://www.wsj.com/articles/the-covid-15-lockdowns-are-lift... "The Covid 15: Lockdowns Are Lifting, and Our Clothes Don’t Fit"

https://www.wsj.com/articles/oversize-sneakers-are-hot-but-v... "Oversize Sneakers Are Hot but Very Hard to Wear. ‘I Just Can’t Balance.’"

Re: A Google bot scrapes pricing info by adding items to carts

#206

Post by [deleted] » Thu, Jul 02, 2020, 1:38 AM UTC

galacticaactual wrote: post #173

gretch wrote: post #156

Earlier quoted context omitted.

This theory doesn't hold up against the test of time. We used to have tyrant monarchs - Genghis Khan would roam the steppe and cutting people's heads off and that would be normal every day life. There are still bad things today, but more people have more rights and a higher quality of life.

It's unclear what you're rebutting here. My position is that on a cyclic basis, there will always be societal elements that will trend towards the negative and nefarious as long as it is tolerated. The fact that the global minimum for such things manifested themselves much more severely (as in your example) in the past does not change the fact that it still occurs today.

[deleted]

Re: A Google bot scrapes pricing info by adding items to carts

#207

Post by abiogenesis » Thu, Jul 02, 2020, 1:46 AM UTC

mmcconnell1618 wrote: post #43

whoisjuan wrote: post #8

This bot is simply trying to get the final price (with tax and shipping) which is ridiculous because e-commerce storefronts should do that in the first place without going through the whole checkout process. I always have found that kind of shady but it's probably known to increase conversions. What I found interesting is that this an open attack vector for e-commerces. Multiple bots can hit a website and start addin…

Not all shipping charges can be calculated ahead of time. For example, you may offer free shipping on orders over $50. You may charge $9.99 for the first item, $5.99 for each additional item. You may charge by weight of the whole order. You may have oversized items or packages that can be combined to reduce shipping charges. Some items may ship together as OTR Freight, while others can go via the local postal service…

True, but you can at least show the total shipping cost for the shopping cart, given a zip code or a similar indication, without completing the whole account creation/checkout process.

Re: A Google bot scrapes pricing info by adding items to carts

#208

Post by judge2020 » Thu, Jul 02, 2020, 1:48 AM UTC

gruez wrote: post #89

the_pwner224 wrote: post #62

Earlier quoted context omitted.

On many websites it still is. But recently many smaller independent stores use the Shopify platform where shipping is the penultimate step, before billing. You have to give address, email(!), phone number(!, mandatory), etc. before getting the price. I normally just use a fake email and number to get shipping price, and then do the actual checkout in incognito. Pretty sure if you do enter your real info and don't con…

I'm pretty sure most sites (eg. Amazon) does the same thing. Probably for the reason you mentioned: so they can have your contact information to send you spam later.

Haven't received any "items in your cart" emails from Amazon peronally, but I will end up seeing Ads for the products later on.

Re: A Google bot scrapes pricing info by adding items to carts

#209

Post by dannyw » Thu, Jul 02, 2020, 1:48 AM UTC

withinboredom wrote: post #77

maltelandwehr wrote: post #50

Earlier quoted context omitted.

„Messing up your competitors A/B test“ is not unheard of as a tactic in highly competitive ecommerce settings.

Do software engineers actually implement that? That seems pretty immoral. I'd rather let them run the a/b test and steal whatever solution they end up with.

Consider companies like Uber....

Re: A Google bot scrapes pricing info by adding items to carts

#210

Post by abiogenesis » Thu, Jul 02, 2020, 1:53 AM UTC

soganess wrote: post #84

For people saying this to calculate the final price with shipping and tax, it's not (or at least not entirely). It is for this new sales conversion dark pattern where prices aren't listed until you add to cart. Ebay sellers are particularly bad offenders: https://www.ebay.com/itm/Open-Box-Certified-Samsung-Galaxy-1...

Google disagrees with you:

> When The Wall Street Journal contacted Google in June, a spokesman at the internet giant, after a few days of digging, provided an update: The mystery shopper is a bot of its own creation. The purpose: making sure the all-in price for the product, including tax and shipping, matches the listing on its Google Shopping platform or in advertisements.