Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping in 2016

franciskim.co

Post reply on HN
Print view View original

251–260 of 402 posts

Re: Web Scraping in 2016

#251

Post by devbug »

greglindahl wrote: post #209
buro9 wrote: post #177

Earlier quoted context omitted.

The UK has a database law: https://en.wikibooks.org/wiki/UK_Database_Law#Database_Right If you scrape, and effectively reconstitute a database, then so long as the database originally had a "substantial investment" in it's "obtaining, verifying or presenting the contents" then yup... you have breached the database right, which is a modified form of copyright. You may access said database (via the web), but as soon as…

Actually, such database laws are rare. The US and Canada don't have one. See Feist v. Rural Telephone for an example of databases getting scraped & the scraper winning in court.

Exactly. You can't copyright facts.

Re: Web Scraping in 2016

#252

Post by fake-name »

nickysielicki wrote: post #246
fake-name wrote: post #195

Earlier quoted context omitted.

Yeah, running this thing publicly would be a huge mess from a copyright perspective, since it literally re-hosts everything as a core part of how it works. As it is, I think I'm OK, since it's basically just a "website DVR" type thing, for my own use. Really, if nothing else, the project has been enormously educational for me. I've learnt a boatload about distributed systems, learned a bit of SQL, dicked about with d…

This project is really cool. Last year I was looking into open source projects that implement something like Readability so that I could scrape articles from my RSS feeds and turn them into plaintext. But I didn't find anything that blew me away. The best I got was stealing the implementation from Firefox, and I lost interest before I could make it worthwhile. (Now revisiting the idea, I wonder why I never thought of…

It's probably GPL, I'll have to figure out my dependencies and see what it's infected with. I tend to err BSD on my own cruft.

This isn't quite as fancy as readability, though I integrated a port of readability for a while. Now I just write a ruleset for a site that has stuff that interests me.

Re: Web Scraping in 2016

#253

Post by zappo2938 »

franciskim wrote: post #22
prashnts wrote: post #10

A neat trick I sometimes use to "scrape" data from sites that use jquery ajax to load data is to plug in a middleware in jquery xhr: $.ajaxSetup({ dataFilter: function (data, type) { if (this.url === 'some url that you want to watch!') { // Do anything with the data here awesomeMethod(this.data) } return data } }) I remember last using it with an infinite-scroll page with a periodic callback that scrolled the page do…

Yup I've been known to do this as well :) I'd have a Node.js + Mongo endpoint ready on the other side.

Why not user Nightmare with Node.js + Mongo?

Here is an example of injecting a jQuery script into a page with jQuery loaded and getting nicely formatted information returned. [1]

[1]https://github.com/adam-s/playboy-fm/blob/master/server/scra...

Re: Web Scraping in 2016

#254

Post by paulddraper »

malz wrote: post #184
hobs wrote: post #13

Earlier quoted context omitted.

This post is kind of crazy, aggrandizing bad behavior and misuse of other's resources against their will. Scraping against the TOS is super bad netizen stuff, and I dont think people should be posting positive reviews of people doing this. Breaking captchas and the like is basically blackhat work and should be looked down upon, not congratulated as I see in this thread.

> misuse of other's resources against their will Nonsense, there is no implication that this activity is illicit. Many sites (I have worked with hundreds) are happy to be included in my service, but don't have the technical ability to provide a data feed. They were delighted when I told them I could aggregate their content without any extra work on their part. We respect TOS, we respect robots.txt and so on. Just bec…

"Don't shoot the messenger for pointing out the lock on your door can be picked"

Re: Web Scraping in 2016

#255

Post by flukus »

hobs wrote: post #13
minimaxir wrote: post #2

Keep in mind that companies have sued for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: http://www.informationweek.com/software/social/linkedin-sues... OKCupid did a DMCA takedown for researchers releasing scraped data: https://www.engadget.com/2016/05/17/publicly-released-okcupi... Since both of these incidents, I now only scrape if it's a) through the API follow…

This post is kind of crazy, aggrandizing bad behavior and misuse of other's resources against their will. Scraping against the TOS is super bad netizen stuff, and I dont think people should be posting positive reviews of people doing this. Breaking captchas and the like is basically blackhat work and should be looked down upon, not congratulated as I see in this thread.

> This post is kind of crazy, aggrandizing bad behavior and misuse of other's resources against their will.

How so? I send a web request, they send me the content in a response. If they aren't happy with that then they should refuse my request.

Re: Web Scraping in 2016

#257

Post by chucksmash »

baddox wrote: post #152
chucksmash wrote: post #103

Earlier quoted context omitted.

Have run into exactly this before. Wrote a scraper that retrieved results from a trivia league website. Tried to be a polite scraper (<1 request per second) but the site still crashed - even with 5 seconds of sleep between requests. They were doing something weird with DB connection management (maybe just forgetting to close it and letting it timeout? I remember figuring it out but it's been quite a while) and so aft…

It seems like you could easily hit those scaling issues by manually browsing the website. While I agree that it sucks to take down a site by scraping, in that specific case it sounds like the performance issues are their fault and not yours. That said, once I realized the effect my scraping had, I would (hopefully) cease my scraping.

So the thing is, I could totally believe they never saw this traffic pattern under normal load. I'd expect bar trivia scores in a certain mid-sized US city are one of those niche things where you have a very low number of uniques but each unique then pokes around on 9 or 10 pages while they're there. The fact that the site didn't crash during normal browsing was what originally led me to speculate they were maintaining an open DB connection per session. If that was indeed the issue, I could totally imagine they'd only rarely (never?) had 100+ "concurrent..ish" unique visitors.

Re: Web Scraping in 2016

#258

Post by Guest98123 »

encoderer wrote: post #242
codeddesign wrote: post #219

Earlier quoted context omitted.

Double standard? The difference is that Google Bot is built on being unobtrusive. I can easily built a scraper that will quickly ddos a site. Linkedin for example...if they allow 10,000 people to send 100 scraping requests per second everyday then that is stolen bandwidth that Linkedin has to pay for and the scrapers get free data. The difference is that Google has standards in which site's unusually benefit from, no…

I agree that Googlebot is well behaved. When it detects your site is slowing down, it will back itself off. Unfortunately, this is often to your detriment. In my experience, on a large site, Google will often slurp as much as you let it, upwards of hundreds of pages per second.

Google usually does 300 pages a minute on my site. In total, bots were loading about 1,000 pages a minute.

Re: Web Scraping in 2016

#259

Post by encoderer »

Guest98123 wrote: post #258
encoderer wrote: post #242

Earlier quoted context omitted.

I agree that Googlebot is well behaved. When it detects your site is slowing down, it will back itself off. Unfortunately, this is often to your detriment. In my experience, on a large site, Google will often slurp as much as you let it, upwards of hundreds of pages per second.

Google usually does 300 pages a minute on my site. In total, bots were loading about 1,000 pages a minute.

That is a lot! It's also still an order of magnitude less than big content sites. Not taking anything away from what must be a successful website to get a consistent 300 pages/minute crawl rate, but only to illustrate magnitude.

Re: Web Scraping in 2016

#260

Post by pyre »

twa927 wrote: post #73
minimaxir wrote: post #2

Keep in mind that companies have sued for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: http://www.informationweek.com/software/social/linkedin-sues... OKCupid did a DMCA takedown for researchers releasing scraped data: https://www.engadget.com/2016/05/17/publicly-released-okcupi... Since both of these incidents, I now only scrape if it's a) through the API follow…

How can TOS have legal power for the case scraping? A website is a public property. If I'm visiting it without logging in, I don't have a chance to accept TOS. Imagine a hotel that makes guests sign a document saying they will not make photographs of the building. If I'm not a guest, I can take photographs of it and I can't even know that would be illegal.

A better metaphor for this would be the "Sunday Flyers" that come in the newspaper (e.g. for big box stores like Best Buy). They sent that information to you, they can not then attempt to restrict how you use that information (though they have tried to claim copyright over pricing against sites that aggregate the flyers).
Post reply on HN