Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Google bot scrapes pricing info by adding items to carts

wsj.com

Post reply on HN
Print view View original

181–190 of 291 posts

Re: A Google bot scrapes pricing info by adding items to carts

#181

Post by milliondollar »

supernova87a wrote: post #174
Youden wrote: post #135

When and why did news cease being news and start being short stories and opinion? This entire article could have been cut down to the last few paragraphs and nothing of value would have been lost. Look at The New York Times in 1921 [0]. Generally the stories are factual and to the point. The entire front page seems to be pure news. There's very little storytelling here, at most there are a few timelines of events. Lo…

Maybe you don't know this, but the "A-hed" article of the WSJ is the humorous, light-hearted take on some cultural phenomenon that appears every couple of days. It's got a distinct separation (graphically) from the rest of the news, and is written not to be taken too seriously. (It's not so apparent in the online version, if you haven't read it before). So you don't have to worry that it's some broad decline in journ…

Amen. Learning how to read the news is a skill just like everything else. The direct links of the internet make it harder, but one still needs to try and understand the context of the publication and how the article is presented. A-heads are entertainment.

Re: A Google bot scrapes pricing info by adding items to carts

#182

Post by ebg13 »

Youden wrote: post #135

When and why did news cease being news and start being short stories and opinion? This entire article could have been cut down to the last few paragraphs and nothing of value would have been lost. Look at The New York Times in 1921 [0]. Generally the stories are factual and to the point. The entire front page seems to be pure news. There's very little storytelling here, at most there are a few timelines of events. Lo…

I'm sure that it predates Fox News Channel by a lot, after all, news business are business first and news second, but it would be misguided to ignore the fact that FNC has completely dominated viewership numbers in the US almost since its inception by doing exactly what you're complaining about. Opinion is a format that engages large numbers of people and therefore makes money extremely well.

Re: A Google bot scrapes pricing info by adding items to carts

#183

Post by fwip »

the_pwner224 wrote: post #62
jiofih wrote: post #40

Earlier quoted context omitted.

Maybe I haven’t done enough online shopping recently, but as far back as I remember this used to be the norm: enter postcode to calculate shipping, get precise final price without even adding to cart. Is it not the case anymore?

On many websites it still is. But recently many smaller independent stores use the Shopify platform where shipping is the penultimate step, before billing. You have to give address, email(!), phone number(!, mandatory), etc. before getting the price. I normally just use a fake email and number to get shipping price, and then do the actual checkout in incognito. Pretty sure if you do enter your real info and don't con…

Also increasing conversion rates via the sunk-cost fallacy. If I see up front that the air conditioner I'm ordering costs $30 to ship, I'll check another site. But if I already decided on this one and I just did all the work to go find my credit card, enter my billing info - maybe I'll just say "ehhh. Fine." and purchase it anyways.

Re: A Google bot scrapes pricing info by adding items to carts

#184

Post by [deleted] »

whoisjuan wrote: post #8

This bot is simply trying to get the final price (with tax and shipping) which is ridiculous because e-commerce storefronts should do that in the first place without going through the whole checkout process. I always have found that kind of shady but it's probably known to increase conversions. What I found interesting is that this an open attack vector for e-commerces. Multiple bots can hit a website and start addin…

[deleted]

Re: A Google bot scrapes pricing info by adding items to carts

#185

Post by marcinzm »

amelius wrote: post #112

It would be cool if Google could manage to become a storefront for the entire web, thereby eliminating Amazon.

I sure would love for my ability buy goods to be blocked by some badly coded ML algorithm with my only recourse being to yell about it on social media. Yeah, I'll take Amazon any day of the week.

Re: A Google bot scrapes pricing info by adding items to carts

#186

Post by IfOnlyYouKnew »

Youden wrote: post #135

When and why did news cease being news and start being short stories and opinion? This entire article could have been cut down to the last few paragraphs and nothing of value would have been lost. Look at The New York Times in 1921 [0]. Generally the stories are factual and to the point. The entire front page seems to be pure news. There's very little storytelling here, at most there are a few timelines of events. Lo…

Are you really complaining about 15% of their front page featuring their opinion section?

Your 1921 NYT example has "Broker is Slain at Bride's Door By Her Gardener", which the Times would never feature prominently today, because it's basically yellow press.

It also features some French pilot's unverified claims about a new record – something that would see HN storm the barricades today.

There's a story there, right at the top, about a group of visitors of the Capitol being disappointed. I'm not entirely sure if it's trying to make fun of the congresswoman it mentions. But it does sound a lot more like today's opinion pieces than a straight news story.

Re: A Google bot scrapes pricing info by adding items to carts

#188

Post by krustyburger »

supernova87a wrote: post #174
Youden wrote: post #135

When and why did news cease being news and start being short stories and opinion? This entire article could have been cut down to the last few paragraphs and nothing of value would have been lost. Look at The New York Times in 1921 [0]. Generally the stories are factual and to the point. The entire front page seems to be pure news. There's very little storytelling here, at most there are a few timelines of events. Lo…

Maybe you don't know this, but the "A-hed" article of the WSJ is the humorous, light-hearted take on some cultural phenomenon that appears every couple of days. It's got a distinct separation (graphically) from the rest of the news, and is written not to be taken too seriously. (It's not so apparent in the online version, if you haven't read it before). So you don't have to worry that it's some broad decline in journ…

All of the links to the A-hed section in the WSJ article link to a 404. Is there an updated link?

Re: A Google bot scrapes pricing info by adding items to carts

#189

Post by fiblye »

totony wrote: post #175
perl4ever wrote: post #165

Earlier quoted context omitted.

Do you find the following inaccurate or biased, and if so, for the uninformed, briefly why? "Campaigns initiated in his lifetime include those against the Qara Khitai, Khwarezmia, and the Western Xia and Jin dynasties, and raids into Medieval Georgia, the Kievan Rus', and Volga Bulgaria. These campaigns were often accompanied by large-scale massacres of the civilian populations, especially in the Khwarazmian- and Wes…

From what I've seen Ganghis Khan was brutal against non-surrendering nations, but beneficial for those who followed him. E.g. you woulsn't see him cutting his subjects heads, and his brutality was pretty much an intimidation tactic so others army would join him.

That's how authoritarian regimes usually work. Do everything they say and you'll be okay. The problem is a decent number of people don't want to be absolutely subservient.

Re: A Google bot scrapes pricing info by adding items to carts

#190

Post by bhuber »

mmcconnell1618 wrote: post #43
whoisjuan wrote: post #8

This bot is simply trying to get the final price (with tax and shipping) which is ridiculous because e-commerce storefronts should do that in the first place without going through the whole checkout process. I always have found that kind of shady but it's probably known to increase conversions. What I found interesting is that this an open attack vector for e-commerces. Multiple bots can hit a website and start addin…

Not all shipping charges can be calculated ahead of time. For example, you may offer free shipping on orders over $50. You may charge $9.99 for the first item, $5.99 for each additional item. You may charge by weight of the whole order. You may have oversized items or packages that can be combined to reduce shipping charges. Some items may ship together as OTR Freight, while others can go via the local postal service…

Even assuming you already know the customer's shipping address and ignore the multiple-items problem, this is still difficult to accomplish from a computational complexity perspective. Calculating shipping cost is likely at least an order of magnitude more expensive than simply looking up list prices in a database - you have to look up the customer's address, go through a bunch of tax rules, figure out the shipping cost (however that works, I honestly don't know but I assume it's non-trivial), etc. Now consider the fact that prices displayed at checkout make up a tiny fraction of prices that are requested by the site. Every time an item appears anywhere on the site you probably want to display a price with it. So now your infrastructure costs for handling pricing requests go up by an order of magnitude since all of them no require expensive pricing computation, whereas only a tiny fraction did before.

On top of all this, if all you're displaying is list price you can cache that very effectively and significantly reduce the load on your backend, probably by at least another order of magnitude. As with many things, items loaded on ecommerce sites tend to follow a Pareto distribution, for which caching is very effective. Adding a shipping address to the mix will destroy this caching ability, so not only are your requests 10x more expensive, 10x more of them now make it to the backend. There are various tricks you can do to try to have your cake and eat it too, but none of them are easy or simple. At the end of the day, while this is definitely a useful and desirable feature for customers, it has significant cost for both development time and hardware.

TL;DR this is actually a much more difficult technical problem to solve cost effectively at scale than it initially appears.

Post reply on HN