Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scrape like the big boys

incolumitas.com

Post reply on HN
Print view View original

171–180 of 196 posts

Re: Scrape like the big boys

#171

Post by jadell »

texasbigdata wrote: post #72
jadell wrote: post #56

Earlier quoted context omitted.

I run a large scraper farm against several large sites. They're not online shops, and we don't compete with them. But they do have hundreds of thousands of data points that we use to provide reports and analytics for our clients, who also do not compete with the sites. I absolutely would pay for an API that provides that data. I'd be willing to pay 10x more than the cost of maintaining and running the scrapers. But t…

Building and maintaining the scraper is the not cost they would use to measure it internally. It’s the cost to build the API, and support it and perhaps any perverse incentive it creates where even more data flows out to competitors.

For all intents and purposes, this isn't competitive data for them. There aren't really competitors in the space anyway, the barrier to entry is ridiculous. In fact, by law, operators in the industry are required to share this particular data with each other and industry regulators. But they don't share it with outside parties in the aggregate form we need it in. Hence, the scraping.

Re: Scrape like the big boys

#172

Post by histriosum »

ransom1538 wrote: post #74
biosed wrote: post #37

I used to lead Sys Eng for a FTSE 100 company. Our data was valuable but only for a short amount of time. We were constantly scraped which cost us in hosting etc. We even seen competitors use our figures (good ones used it to offset their prices, bad ones just used it straight). As the article suggest, we couldn't block mobile operator IPs, some had over 100k customers behind them. Forcing the users to login did litt…

I love the honey pot approach. Put tons of valued hrefs on the page that are invisible (css) that the scrapper would find. Then just rate limit that ip address and randomize the data coming back. Profit.

I think this falls into the "arms race" trap, though. If you can make an href invisible via CSS, then the scraper can certainly be written to understand CSS, and thus filter out the invisible hrefs..

Re: Scrape like the big boys

#173

Post by cbsmith »

matheusmoreira wrote: post #104
rootusrootus wrote: post #65

Earlier quoted context omitted.

I know a guy at Nike that had to deal with a similar problem. As I recall, they basically gave in -- instead of trying to fight the scrapers, they built them an API so they'd quit trashing the performance of the retail site with all the scraping.

Yes. That's exactly what everyone should do.

Well, not EXACTLY. The exactly should be to just do WebSub/PuSH. No need to invent your own thing and hope that bots learn how to use it properly.

Re: Scrape like the big boys

#174

Post by cbsmith »

simondotau wrote: post #125
ianbutler wrote: post #118

Earlier quoted context omitted.

You're going to need to explain how scraping publicly available information on a website is theft. If information is your competitive advantage maybe you shouldn't have it on a publicly accessible website, and should instead stick it behind an API with pay tiers and a very clear license regarding what you may do with it as an end user. Note, a simple sign up being required to view a website makes it not publicly avai…

> You're going to need to explain how scraping publicly available information on a website is theft. Seriously? Do I need to explain why a song doesn’t enter the public domain when it is played on the radio?

No, but there is a legit philosophical argument about theft when it comes to copyright. There are two ways to look at theft: acquiring something you didn't earn vs. someone losing something they did earn. Generally, we tend to focus on the latter. From that perspective, "copying" is really not "theft", and arguably "copyright" does more net societal harm than any benefit it provides.

Re: Scrape like the big boys

#175

Post by matheusmoreira »

cbsmith wrote: post #173
matheusmoreira wrote: post #104

Earlier quoted context omitted.

Yes. That's exactly what everyone should do.

Well, not EXACTLY. The exactly should be to just do WebSub/PuSH. No need to invent your own thing and hope that bots learn how to use it properly.

Agreed. What I mean is people need to stop fighting these pointless battles.

Re: Scrape like the big boys

#176

Post by matheusmoreira »

echelon wrote: post #169
matheusmoreira wrote: post #128

Earlier quoted context omitted.

Do I need to explain that copyright is practically unenforceable in the 21st century? Data is trivially copied and there's nothing you can do to fight that, no amount of laws will ever make it non-trivial again. Even if you successfully sue somebody for this, it won't stop them. At some point people are gonna have to accept this.

What are copyright strikes, then?

Not even a thing in my country. Likely easy to avoid in others.

Re: Scrape like the big boys

#177

Post by abc03 »

rdtwo wrote: post #166
abc03 wrote: post #52

I scrap government sites a lot as they don't provide apis. For mobile proxies, I use the proxidize dongles and mobinet.io (free, with Android devices). As stated in the article, with cgNAT it's basically impossible to block them as in my case, half the country couldn't access the sites anymore (if you place them in several locations and use one carrier each there).

Is it just one ip per dongle at a time? Or can you have multiple ips on the same device.

Just one IP at a time but you can change every 5 Min or more if you like

Re: Scrape like the big boys

#178

Post by achillesheels »

Grimm1 wrote: post #124
achillesheels wrote: post #122

Earlier quoted context omitted.

It is copyright information, no? So technically it is intellectual property theft if the scraping use is for commercial purposes.

No? If you place information publicly on a website it's pretty much free game, no copyright violation, especially regarding user generated information. That's my take, but legally it's a gray area and it's still going back and forth in the courts (at least in the US) but for a while before a decision was vacated by the supreme court scraping publicly available information on a site was legally protected and seemingly…

If we are to live in a mutually prosperous society, how is the labor and therefore well-being of the content creator improved by a web scraper? Does this precedent not injury future opportunities for exercising one’s life to making website data available for others to scrape?

By my understanding any website with a copyright disclaimer warrants their data as exclusively their own and are granting permission for other web users to generate it, ie people are not entitled to share their web data with anyone. So if they are, and we agree that it’s good that they do, and continue to create information for others to know, how do we avoid the implicit harm in extracting data without nothing being given in return but possibly harming the internet’s experience for everyone accessing the same information?

Re: Scrape like the big boys

#179

Post by wiether »

joekrill wrote: post #14

A little pet-peeve I have is when an obscure(ish) acronym is used and never defined. Is SERP a well-known acronym? Perhaps this is a niche blog and I'm not the intended audience.

On HN I'm used to SE meaning Software Engineer so I came up with "Software Engineer Ranting Board" before asking Google to give me the SERP that would provide me with the true meaning of SERP.

Re: Scrape like the big boys

#180

Post by incolumitas »

max002 wrote: post #127

Its easy to detect chrome headless so scraping with it is not really how "big" boys do it :D the only scrapers/bots that are really hard to detect are the ones running and controlling real browser and not chromium. I do a lot od research aggainst abitbot systems, some times is friday night. If you spend each one in pub it doesnt mean your normal.

:D
Post reply on HN