Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Google bot scrapes pricing info by adding items to carts

wsj.com

Post reply on HN
Print view View original

241–250 of 291 posts

Re: A Google bot scrapes pricing info by adding items to carts

#241

Post by jollofricepeas »

Youden wrote: post #135

When and why did news cease being news and start being short stories and opinion? This entire article could have been cut down to the last few paragraphs and nothing of value would have been lost. Look at The New York Times in 1921 [0]. Generally the stories are factual and to the point. The entire front page seems to be pure news. There's very little storytelling here, at most there are a few timelines of events. Lo…

When has journalism ever stuck to facts? When has the “news” been more than short stories and opinion?

Hint: Never

Read: A History of News by Mitchell Stephens https://openlibrary.org/works/OL1854941W/A_history_of_news

Read: https://en.m.wikipedia.org/wiki/Yellow_journalism

Read: https://en.m.wikipedia.org/wiki/History_of_French_journalism

Re: A Google bot scrapes pricing info by adding items to carts

#242

Post by ehsankia »

yongjik wrote: post #81

robots.txt, man, if you don't want search engines to visit certain part of your page, use robots.txt! Once heard a tale of an angry site owner calling Google (back when Google itself was novel) - Google deleted his whole website! Turned out he had "DELETE" button in each page, which generated plain GET request. So Googlebot visited the site, followed links to every page, and then of course followed every link that ge…

I'm curious though, why didn't Google properly tag their robot in this case? Or was the reporter did not know about user agents? It seems strange for them to crawl with a "mystery bot".

Re: A Google bot scrapes pricing info by adding items to carts

#243

Post by kube-system »

ehsankia wrote: post #240
gruez wrote: post #14

Earlier quoted context omitted.

> This bot is simply trying to get the final price (with tax and shipping) which is ridiculous because e-commerce storefronts should do that in the first place without going through the whole checkout process. It's usually not possible because you don't know how much the shipping + taxes are until the customer enters the billing information.

I don't know if this is true, but I've been many websites that actually claim that they have an extra deal they can't show you until you put the item in the cart. I used to see those a lot a few years ago, not sure if it was a real legal thing or just a trick to get people to add it, but that was definitely a thing and not related to tax+shipping.

It is a contractual obligation with the product manufacturer.

Something like: we will allow you to sell our product but only if you don't advertise discounts more than x%.

Re: A Google bot scrapes pricing info by adding items to carts

#244

Post by kube-system »

wdb wrote: post #61
gruez wrote: post #14

Earlier quoted context omitted.

> This bot is simply trying to get the final price (with tax and shipping) which is ridiculous because e-commerce storefronts should do that in the first place without going through the whole checkout process. It's usually not possible because you don't know how much the shipping + taxes are until the customer enters the billing information.

They could at least include the tax in the price. That's normally fixed depending on the item category e.g. low or high tax rate. Only the US is being weird with its taxes.

Canada has provincial sales taxes.

It's really not that weird when you consider the US is not a unitary state.

Re: A Google bot scrapes pricing info by adding items to carts

#245

Post by kube-system »

soganess wrote: post #84

For people saying this to calculate the final price with shipping and tax, it's not (or at least not entirely). It is for this new sales conversion dark pattern where prices aren't listed until you add to cart. Ebay sellers are particularly bad offenders: https://www.ebay.com/itm/Open-Box-Certified-Samsung-Galaxy-1...

This is more likely just a contractual MAP (Minimum Advertised Price) policy by the manufacturer, not a dark pattern that is of the retailer's choosing.

https://www.thebalancesmb.com/what-is-minimum-advertised-pri...

Re: A Google bot scrapes pricing info by adding items to carts

#246

Post by hnburnsy »

asdf21 wrote: post #221
supernova87a wrote: post #174

Earlier quoted context omitted.

Maybe you don't know this, but the "A-hed" article of the WSJ is the humorous, light-hearted take on some cultural phenomenon that appears every couple of days. It's got a distinct separation (graphically) from the rest of the news, and is written not to be taken too seriously. (It's not so apparent in the online version, if you haven't read it before). So you don't have to worry that it's some broad decline in journ…

>The WSJ is one of the few quite reputable news rooms out there. Hahhahahah I think that myth evaporated when they spend months trying to "expose" Pewdiepie.

All the news doesn't have to be serious. Hope you didn't miss the WSJ's reporting on Theranos which potentially saved lives by exposing Theranos' fraudulent lab tests.

Re: A Google bot scrapes pricing info by adding items to carts

#247

Post by manigandham »

toomanybeersies wrote: post #239
dahfizz wrote: post #215

Earlier quoted context omitted.

> It's not so apparent in the online version, if you haven't read it before. I think this is the core issue that leads to sentiment like OPs. Real news still exists, but it's the highly editorialized and opinionated articles that are shared more widely. News agencies universally are terrible at obviously differentiating the two to users. 99% of people when they open an article scroll directly to the content. But any…

Is it the news agency's fault, or the reader's fault? In this case, it really isn't clear that it's not "real news". But I see plenty of people on social media sharing articles from news sites where they're clearly marked as an editorial or opinion piece, and believing or treating them as if they were exactly the same as a news article that attempts to paint a neutral picture of the facts. Do newspapers need to put a…

Perhaps newspapers should stop running opinions and editorials altogether, or move them to an entirely different brand.

Re: A Google bot scrapes pricing info by adding items to carts

#248

Post by ehnto »

whoisjuan wrote: post #8

This bot is simply trying to get the final price (with tax and shipping) which is ridiculous because e-commerce storefronts should do that in the first place without going through the whole checkout process. I always have found that kind of shady but it's probably known to increase conversions. What I found interesting is that this an open attack vector for e-commerces. Multiple bots can hit a website and start addin…

There's a lot that goes into when and where you can show the final item price.

Assuming a simple product, you don't know where the user lives so you can't apply the correct taxes yet. In AUS this is easy because it's the same nationally. But in the US there are dozens of tax combinations that could be applied depending on the location.

Shipping obviously depends on where you live and what you're buying, and few places charge per item shipping these days so it doesn't even make sense to include shipping in a single item cost.

Then you have customer group discounts, some customers get different pricing when they are logged in, even item combinations in the cart can have different prices, you get the idea. It is usually not possible to calculate ahead of time.

Re: A Google bot scrapes pricing info by adding items to carts

#249

Post by ehnto »

chrisan wrote: post #75
mmcconnell1618 wrote: post #43

Earlier quoted context omitted.

Not all shipping charges can be calculated ahead of time. For example, you may offer free shipping on orders over $50. You may charge $9.99 for the first item, $5.99 for each additional item. You may charge by weight of the whole order. You may have oversized items or packages that can be combined to reduce shipping charges. Some items may ship together as OTR Freight, while others can go via the local postal service…

How does that change by having the bot add items to the cart? You haven't solved anything You are still left with the same scenario as if the store listed the individual shipping price on the front page Google isn't going to know what other items you _might_ add to show you a "real" shipping cost

No one said the bot is getting good data. I assume it's trying to get the best possible outcome by adding to the cart, but I doubt it's getting the real final price for many merchants even by doing that.

Re: A Google bot scrapes pricing info by adding items to carts

#250

Post by VonGallifrey »

milliondollar wrote: post #181
supernova87a wrote: post #174

Earlier quoted context omitted.

Maybe you don't know this, but the "A-hed" article of the WSJ is the humorous, light-hearted take on some cultural phenomenon that appears every couple of days. It's got a distinct separation (graphically) from the rest of the news, and is written not to be taken too seriously. (It's not so apparent in the online version, if you haven't read it before). So you don't have to worry that it's some broad decline in journ…

Amen. Learning how to read the news is a skill just like everything else. The direct links of the internet make it harder, but one still needs to try and understand the context of the publication and how the article is presented. A-heads are entertainment.

This is the "You are holding it wrong" opinion I guess.

If A-heads are entertainment then why does it say A-head and not Entertainment? I am not at fault here if I don't know what the hell A-head is supposed to mean and just think that this news site is kinda garbage.

What exactly about the presentation of this article was I supposed to pick up on and see that it is entertainment. It is the exact same presentation than their "U.S. Seeks to Seize Iranian Fuel Bound for Venezuela" article. Is that one news or is it also entertainment?

Post reply on HN