Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping in 2016

franciskim.co

Post reply on HN
Print view View original

261–270 of 402 posts

Re: Web Scraping in 2016

#261

Post by collinmanderson »

tangue wrote: post #247
Analemma_ wrote: post #210

Earlier quoted context omitted.

Why is it a double standard? Google scraping usually benefits the site with increased traffic and revenue, in a way most other scraping does not. Saying "you can scrape me if it benefits me" isn't totally in keeping with the principles of the open web, but it's not hypocritical.

Google is basically algorithms built on top of a scraping service. It's unfair to competitors (and potential disruptors) to restrict access to data that Google can fetch without limits.

Maybe we (scrappers) just need to market ourselves as search engines. _Indexing_ is what we're doing. :)

Re: Web Scraping in 2016

#262

Post by Guest98123 »

encoderer wrote: post #259
Guest98123 wrote: post #258

Earlier quoted context omitted.

Google usually does 300 pages a minute on my site. In total, bots were loading about 1,000 pages a minute.

That is a lot! It's also still an order of magnitude less than big content sites. Not taking anything away from what must be a successful website to get a consistent 300 pages/minute crawl rate, but only to illustrate magnitude.

I was curious, so I just checked the stats through webmaster tools. For the last 90 days, the low is 450,000 daily crawled pages, average is 650,000, and yesterday was the high of 1,130,000 (780 per minute). Ouch.

Re: Web Scraping in 2016

#263

Post by encoderer »

Guest98123 wrote: post #262
encoderer wrote: post #259

Earlier quoted context omitted.

That is a lot! It's also still an order of magnitude less than big content sites. Not taking anything away from what must be a successful website to get a consistent 300 pages/minute crawl rate, but only to illustrate magnitude.

I was curious, so I just checked the stats through webmaster tools. For the last 90 days, the low is 450,000 daily crawled pages, average is 650,000, and yesterday was the high of 1,130,000 (780 per minute). Ouch.

Have you seen correlation between rankings and crawl rate?

Re: Web Scraping in 2016

#264

Post by ori_b »

Nicksil wrote: post #44
minimaxir wrote: post #25

Earlier quoted context omitted.

> tl;dr: Scraping isn't bad / blackhat as long as you aren't affecting their service or business. Analyzing data that you're not allowed to access gives you/your company a competitive advantage, which is affecting their service/business even if it's not posted/distributed publically.

I don't follow your argument. How does one get their scraper access to data they would otherwise not be able to access through 'normal' browsing techniques?

By ignoring robots.txt and bypassing capchas.

Re: Web Scraping in 2016

#265

Post by [deleted] »

nreece wrote: post #230

At Feedity ( https://feedity.com ), we "index" webpages to generate custom feeds. Over the years, we've designed our system to use a mix of technologies like .NET (C#) and node.js, and implemented a bunch of tweaks and optimizations for seamless & scalable access to public content.

[deleted]

Re: Web Scraping in 2016

#266

Post by elmigranto »

BrandonMarc wrote: post #161
elmigranto wrote: post #52

Earlier quoted context omitted.

> To fight scrapers, we show some values as images that look like text You are fighting screen readers more than anything; as well as legitimate plugins, form autofills, etc. If this is for captcha, you are fighting all the users as well. > And we insert random (non-visible) html and css classes in our site to screw with em, and use randomized css classnames. Legitimate browser plugins, etc. I'd just use electron or…

> What you effectively doing is wasting time on useless stuff. This is even more useless than trying to prevent copying of DVDs or pirating games. Can you be so sure? The Union blockade of the Confederacy had plenty of holes, and smugglers / privateers / blockade-runners made good money getting through (when they survived) ... but that doesn't mean the blockade wasn't effective all the same at weakening the Confedera…

Do you really not see the difference between military blockade and randomizing CSS classes?

Re: Web Scraping in 2016

#267

Post by macromaniac »

prashnts wrote: post #10

A neat trick I sometimes use to "scrape" data from sites that use jquery ajax to load data is to plug in a middleware in jquery xhr: $.ajaxSetup({ dataFilter: function (data, type) { if (this.url === 'some url that you want to watch!') { // Do anything with the data here awesomeMethod(this.data) } return data } }) I remember last using it with an infinite-scroll page with a periodic callback that scrolled the page do…

Another trick: You can hover over elements in chrome with F12 and the inspect tool, then right click > copy > copy selector and chrome will generate one for you that way you don't have to actually do any work.

With a selector it's easy to grab data, here's a linux command that gets every user that posted in this thread:

  lynx -base -source 'https://news.ycombinator.com/item?id=12345693' | hxnormalize -x | \
    hxselect -c -s '\n' "td > table > tbody > tr > td.default > div:nth-child(1) > span > a.hnuser"
Here are the most frequent commenters:

     27 cookiecaper
     22 franciskim
      6 fake-name
      4 niftich
      4 flukus
      4 elmigranto
      4 downandout
      3 tedunangst
      3 siegecraft
      3 muglug
      3 minimaxir
      3 madamelic

Re: Web Scraping in 2016

#268

Post by Steeeve »

hchasestevens wrote: post #61
cookiecaper wrote: post #50

Earlier quoted context omitted.

Eh, this is pretty weak. Scrapers are no different from other browsing devices. The web speaks HTTP. There's no reason that using another HTTP browser would cause any disparate impact just by virtue of not being a conventional desktop browser -- you've thrown out a pretty absurd hypothetical. In fact, scrapers usually cause less impact because they usually don't download images or execute JavaScript. I did an analysi…

A small counterpoint to this -- in the airline industry, it's relatively commonplace for seat reservations to be made for a user _before_ payment has occurred. In this case, if you're mirroring normal browser activity, you can (temporarily) reduce availability on a flight, potentially even bumping up the price for other, legitimate users, and almost certainly causing the airline to incur costs beyond normal bandwidth…

I've had this happen regularly enough that I developed the habit of finding a fare in the morning and then returning at 11pm ready to buy.

Re: Web Scraping in 2016

#269

Post by ccvannorman »

franciskim wrote: post #244
minimaxir wrote: post #2

Keep in mind that companies have sued for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: http://www.informationweek.com/software/social/linkedin-sues... OKCupid did a DMCA takedown for researchers releasing scraped data: https://www.engadget.com/2016/05/17/publicly-released-okcupi... Since both of these incidents, I now only scrape if it's a) through the API follow…

On the ethics side, I don't scrape large amounts of data - eg. giving clients lead gen (x leads for y dollars) - in fact, I have never done a scraping job and don't intend to do those jobs for profit. For me it's purely for personal use and my little side projects. I don't even like the word scraping because it comes loaded with so many negative connotations (which sparked this whole comment thread) - and for a good…

This is exactly my response to "you can't legally scrape my site because of TOS." I don't think anyone has a legal right to tell me HOW I use their service. Making "browsing my website using a script you wrote yourself" illegal is akin to "You cannot use the tab key to tab between fields on my website, you must only use the touchpad to move the cursor over each field individually."

It's baloney.

Re: Web Scraping in 2016

#270

Post by cookiecaper »

esrauch wrote: post #234
gnud wrote: post #128

Earlier quoted context omitted.

That notice is typically in the footer, and a screen reader will reach the nav-bar before mentioning the TOS notice. Even for sighted people, the notice is often easy to miss - and this is by design.

It's by design because the vast majority of people don't care about that information and it makes the website worse for them to have a big ToS banner at the top of your page. I don't think many websites have a secret ToS that they hope you won't read, I think most of them don't even know what their own ToS say. I signed my lease on a site with an explicit checkmark for ToS that said I agreed I would only use exactly…

Which is plenty amusing, until one of those companies is suing you in a court of law for violating said ToS.
Post reply on HN