Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Google bot scrapes pricing info by adding items to carts

wsj.com

Post reply on HN
Print view View original

211–220 of 291 posts

Re: A Google bot scrapes pricing info by adding items to carts

#211

Post by abiogenesis »

cortesoft wrote: post #133
withinboredom wrote: post #77

Earlier quoted context omitted.

Do software engineers actually implement that? That seems pretty immoral. I'd rather let them run the a/b test and steal whatever solution they end up with.

Eh, I am sure you could convince yourself it isn't immoral... everyone in HN seems to think things like google analytics are bad because of the privacy implications, and doesn't have a problem blocking them (which would also 'mess with a/b tests'). You could just argue that you are hindering their user spying. Not a great argument, but good enough to allow a developer to sleep at night.

True, but this is not "messing up your competitors A/B test".

Re: A Google bot scrapes pricing info by adding items to carts

#212

Post by gnuisance »

mmcconnell1618 wrote: post #201
bhuber wrote: post #190

Earlier quoted context omitted.

Even assuming you already know the customer's shipping address and ignore the multiple-items problem, this is still difficult to accomplish from a computational complexity perspective. Calculating shipping cost is likely at least an order of magnitude more expensive than simply looking up list prices in a database - you have to look up the customer's address, go through a bunch of tax rules, figure out the shipping c…

In many cases, this is not only computationally expensive on the server, it also requires one or more requests to external APIs which further slows things down. Imagine needing to query the API of your tax vendor, then also your shipping provider of choice for every single item displayed on a page. Even if you did this client side, asynchronously, it would be a lot of extra requests for something that most shoppers w…

Having worked with a major eCom platform as well, this is exactly the standard case. Both shipping and tax are complex problems which do not have a simple solution for scraping by a search engine.

Shipping is often highly dependent on the location of the buyer and often involves full estimate calls from each carriers APIs (USPS, FedEx, UPS). The only major data point I would focus on is whether the shipping is free or flat rate.

Tax is even more complicated. Merchants often outsource tax calculations to a third-party service such as Avalara, which calculates unbelievably complex taxing schemes even down to the zip code, as tax laws are becoming increasingly more complex.

Because of these reasons taxes and shipping are not widely useful data points for search engines. That may change in the future, however. I could imagine it becoming another SEO topic to be accounted for, similar to meta tags on product pages.

Re: A Google bot scrapes pricing info by adding items to carts

#213

Post by bcrosby95 »

input_sh wrote: post #171
shultays wrote: post #152

Earlier quoted context omitted.

I find not including tax better tbh. That way people are being reminded how much tax they are with every purchase.

Every EU bill should come with the price without VAT (value-added tax) and with the VAT applied, leading to the same result. The only difference is that the VAT is a certain % depending on your country (mostly 15-25%), making it easy for a merchant to calculate it with a single data point. Percentage is also always a fixed round number, so you can calculate it in your head when you stumble upon "+ VAT" somewhere with…

Yeah. It's easy in the EU.

My favorite example of how absurd it can get in the US: one side of my friend's street has 9.5% sales tax, the other side has 7.25%. same state, city, and zip.

Re: A Google bot scrapes pricing info by adding items to carts

#214

Post by harry8 »

supernova87a wrote: post #174
Youden wrote: post #135

When and why did news cease being news and start being short stories and opinion? This entire article could have been cut down to the last few paragraphs and nothing of value would have been lost. Look at The New York Times in 1921 [0]. Generally the stories are factual and to the point. The entire front page seems to be pure news. There's very little storytelling here, at most there are a few timelines of events. Lo…

Maybe you don't know this, but the "A-hed" article of the WSJ is the humorous, light-hearted take on some cultural phenomenon that appears every couple of days. It's got a distinct separation (graphically) from the rest of the news, and is written not to be taken too seriously. (It's not so apparent in the online version, if you haven't read it before). So you don't have to worry that it's some broad decline in journ…

> The WSJ is one of the few quite reputable news rooms out there.

The WSJ is owned by Rupert Murdoch. The credibility of their newsroom begins being compromised by his owning it. He will destroy its credibility utterly by selling it for politicial influence in news reporting. Just as he has everywhere he has bought media. The particular example of compromised credibility that comes to mind is the Times of London which is now Murdoch propaganda (all be it vastly more polite than fox news) where it used to do credible news reporting. Times reporting now can still be excellent but has a "be cautious" flag on it that it used not to have in the days prior to Mudoch. The man has become vastly worse in the past couple of decades as has everything he touches.

Re: A Google bot scrapes pricing info by adding items to carts

#215

Post by dahfizz »

supernova87a wrote: post #174
Youden wrote: post #135

When and why did news cease being news and start being short stories and opinion? This entire article could have been cut down to the last few paragraphs and nothing of value would have been lost. Look at The New York Times in 1921 [0]. Generally the stories are factual and to the point. The entire front page seems to be pure news. There's very little storytelling here, at most there are a few timelines of events. Lo…

Maybe you don't know this, but the "A-hed" article of the WSJ is the humorous, light-hearted take on some cultural phenomenon that appears every couple of days. It's got a distinct separation (graphically) from the rest of the news, and is written not to be taken too seriously. (It's not so apparent in the online version, if you haven't read it before). So you don't have to worry that it's some broad decline in journ…

> It's not so apparent in the online version, if you haven't read it before.

I think this is the core issue that leads to sentiment like OPs. Real news still exists, but it's the highly editorialized and opinionated articles that are shared more widely. News agencies universally are terrible at obviously differentiating the two to users.

99% of people when they open an article scroll directly to the content. But any discerning features (in this case, a small font A-hed link) are tucked way at the top. In this case, the A-hed link takes you to the A-hed home page but still does not offer any context to what A-hed is.

> So you don't have to worry that it's some broad decline in journalistic standards (at least based on this).

As long as the WSJ does such a bad job at separating "WSJ the proper news room" and "A-hed the not proper news room", their brand will suffer. OP is proof of that, and I think it's safe to assume the average HN reader is more astute than the average citizen. A tiny link to nowhere useful is not enough of a UI change for us to blame the user. The onus is on the news agencies to do a better job giving context for articles.

Re: A Google bot scrapes pricing info by adding items to carts

#216

Post by amadeuspagel »

harry8 wrote: post #214
supernova87a wrote: post #174

Earlier quoted context omitted.

Maybe you don't know this, but the "A-hed" article of the WSJ is the humorous, light-hearted take on some cultural phenomenon that appears every couple of days. It's got a distinct separation (graphically) from the rest of the news, and is written not to be taken too seriously. (It's not so apparent in the online version, if you haven't read it before). So you don't have to worry that it's some broad decline in journ…

> The WSJ is one of the few quite reputable news rooms out there. The WSJ is owned by Rupert Murdoch. The credibility of their newsroom begins being compromised by his owning it. He will destroy its credibility utterly by selling it for politicial influence in news reporting. Just as he has everywhere he has bought media. The particular example of compromised credibility that comes to mind is the Times of London whic…

Murdoch bought the WSJ in 2007. When is he going to start destroying its credibility "utterly"?

Re: A Google bot scrapes pricing info by adding items to carts

#217

Post by dylz »

doe88 wrote: post #195

I wouldn't fault them for that, I've observed some sites most likely are gaming the system by detecting and providing Google bots with artificially lower prices so that they would appear in indexes summaries and then when you access the product, its real price is always higher than the one reported in the index.

yep, I see this type of behaviour constantly - faked prices for Gbot, fake prices on Cache, significantly higher price for end user.

It's also infuriating to sort by price and get inflated fake shipping prices to "make up the total"

Re: A Google bot scrapes pricing info by adding items to carts

#219

Post by SquishyPanda23 »

amadeuspagel wrote: post #216
harry8 wrote: post #214

Earlier quoted context omitted.

> The WSJ is one of the few quite reputable news rooms out there. The WSJ is owned by Rupert Murdoch. The credibility of their newsroom begins being compromised by his owning it. He will destroy its credibility utterly by selling it for politicial influence in news reporting. Just as he has everywhere he has bought media. The particular example of compromised credibility that comes to mind is the Times of London whic…

Murdoch bought the WSJ in 2007. When is he going to start destroying its credibility "utterly"?

Uh have you read their commentary/opinions? Half the time they come off as if they're trolling.

I'm sure at one point they were a thinking man's newspaper. At this point they're just fan service for people who have drunk the koolaid but can't stomach Fox's mass market approach.

Re: A Google bot scrapes pricing info by adding items to carts

#220

Post by harry8 »

amadeuspagel wrote: post #216
harry8 wrote: post #214

Earlier quoted context omitted.

> The WSJ is one of the few quite reputable news rooms out there. The WSJ is owned by Rupert Murdoch. The credibility of their newsroom begins being compromised by his owning it. He will destroy its credibility utterly by selling it for politicial influence in news reporting. Just as he has everywhere he has bought media. The particular example of compromised credibility that comes to mind is the Times of London whic…

Murdoch bought the WSJ in 2007. When is he going to start destroying its credibility "utterly"?

He started in 2007. The date when you consider it utterly destroyed is up to the reader.
Post reply on HN