Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scrape like the big boys

incolumitas.com

Post reply on HN
Print view View original

111–120 of 196 posts

Re: Scrape like the big boys

#111

Post by kwhitefoot »

killingtime74 wrote: post #95
wolverine876 wrote: post #85

Earlier quoted context omitted.

> But our most fruitful effort was when we removed limits and started giving "bad" data. By bad I mean alter the price up or down by a small percentage. ... If the customer made a transaction on the altered figure we we informed them and took it at the correct price. Is that legal? It would be a big blow to trust if I was the customer, but that's without knowing what you were selling and in what market.

It’s legal if it’s in the contract. Standard for contracts to allow for mistakes and confirmations of prices

It's not mistake if you do it deliberately!

Re: Scrape like the big boys

#113

Post by nkozyra »

neals wrote: post #68

In a particularly hard to scrape website, using some kind of bot protection that I just couldn't reliably get working (if anybody wants to know what that was exactly, I'll go and check it) I now have a small Intel NUC running with firefox that listens to a local server and uses Temper Monkey to perform commands. Works like a charm and I can actualy see what it's doing and where it's going wrong. (though it's not scal…

You can use chromium/chrome/cdp and turn headless off and see the same thing.

Re: Scrape like the big boys

#114

Post by krzyk »

ebbp wrote: post #2

Having spent a week battling a particularly inconsiderate scraping attempt, I’m quite unsurprised by the juvenile tone and fairly glib approach to the ethics of bots/scraping presented by the piece. For the site I work for, about 20-30% of our monthly hosting costs go towards servicing bot/scraping traffic. We’ve generally priced this into the cost of doing business, as we’ve prioritised making our site as freely acc…

As a programmer that just sometimes wants to check if given item is available in store I would like to be able to use API for that. But if it is not available one has to scrape.

Re: Scrape like the big boys

#115

Post by echelon »

matheusmoreira wrote: post #104
rootusrootus wrote: post #65

Earlier quoted context omitted.

I know a guy at Nike that had to deal with a similar problem. As I recall, they basically gave in -- instead of trying to fight the scrapers, they built them an API so they'd quit trashing the performance of the retail site with all the scraping.

Yes. That's exactly what everyone should do.

If data is your competitive advantage or product, then what? Accept that your market no longer exists and that there's no way to stop theft?

Re: Scrape like the big boys

#116

Post by wolverine876 »

texasbigdata wrote: post #72
jadell wrote: post #56

Earlier quoted context omitted.

I run a large scraper farm against several large sites. They're not online shops, and we don't compete with them. But they do have hundreds of thousands of data points that we use to provide reports and analytics for our clients, who also do not compete with the sites. I absolutely would pay for an API that provides that data. I'd be willing to pay 10x more than the cost of maintaining and running the scrapers. But t…

Building and maintaining the scraper is the not cost they would use to measure it internally. It’s the cost to build the API, and support it and perhaps any perverse incentive it creates where even more data flows out to competitors.

And the cost of being scraped.

Re: Scrape like the big boys

#117

Post by paco3346 »

RobSm wrote: post #83
paco3346 wrote: post #62

Earlier quoted context omitted.

It was a hybrid of low-effort vulnerability scanning and targeted inventory scraping. Many dealerships in the automotive space will pay gray-hat third parties to scrape and compile data on their competitors. The irony for us as a provider is that it's one of our customers (party A) paying a third party to scrape data from another one of our customers (party B) which in turn affects the performance of party A's site.…

And how do you get your 'inventory data'? Aren't you scraping (or using scraped data) yourself? Oh the irony :)

No, we're a contracted provider for these customers. They ingest their data into our network through APIs or CSVs.

Re: Scrape like the big boys

#118

Post by ianbutler »

echelon wrote: post #115
matheusmoreira wrote: post #104

Earlier quoted context omitted.

Yes. That's exactly what everyone should do.

If data is your competitive advantage or product, then what? Accept that your market no longer exists and that there's no way to stop theft?

You're going to need to explain how scraping publicly available information on a website is theft.

If information is your competitive advantage maybe you shouldn't have it on a publicly accessible website, and should instead stick it behind an API with pay tiers and a very clear license regarding what you may do with it as an end user.

Note, a simple sign up being required to view a website makes it not publicly available information any longer and you can cover usage, again, in a license.

Then you have a whole bunch of legal avenues you can use to protect your work. Assuming you can afford it that is.

Re: Scrape like the big boys

#119

Post by thatwasunusual »

scarygliders wrote: post #31
ebbp wrote: post #2

Having spent a week battling a particularly inconsiderate scraping attempt, I’m quite unsurprised by the juvenile tone and fairly glib approach to the ethics of bots/scraping presented by the piece. For the site I work for, about 20-30% of our monthly hosting costs go towards servicing bot/scraping traffic. We’ve generally priced this into the cost of doing business, as we’ve prioritised making our site as freely acc…

Right with you there. I had a particularly bad time not so long ago, when a customer's site - a shop - was brought to its knees because someone, probably a competitor, hired some scraper-company of some sort to scrape every product and price. The scraper would systematically go through every single product page. And by scraper, I mean - 100's of them. All at the same time, using the old trick of 1 scraper requesting…

It sucks when this happens, but it's easily avoidable by using a caching frontend of some sort.

My favorite is Varnish,[0] which I have used with great success for _many_ web sites throughout the years. Even a web site that 10+ millions of requests per day ran from a single web server for a long time a decade-ish ago.

[0] https://varnish-cache.org/

Re: Scrape like the big boys

#120

Post by gonzo41 »

rootusrootus wrote: post #65
biosed wrote: post #37

I used to lead Sys Eng for a FTSE 100 company. Our data was valuable but only for a short amount of time. We were constantly scraped which cost us in hosting etc. We even seen competitors use our figures (good ones used it to offset their prices, bad ones just used it straight). As the article suggest, we couldn't block mobile operator IPs, some had over 100k customers behind them. Forcing the users to login did litt…

I know a guy at Nike that had to deal with a similar problem. As I recall, they basically gave in -- instead of trying to fight the scrapers, they built them an API so they'd quit trashing the performance of the retail site with all the scraping.

I think there's an opportunity for a new JS framework to have something like randomly generated dom that will always display the page and elements the same to a human but constantly break paths for computers.

Like displaying a table with semantic elements, then divs, then using an iframe with css grid and floating values over the top.

This almost seems like a problem for AI to solve.

Post reply on HN