Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping in 2016

franciskim.co

Post reply on HN
Print view View original

241–250 of 402 posts

Re: Web Scraping in 2016

#241

Post by nercht12 »

mgalka wrote: post #225
nmstoker wrote: post #214

Earlier quoted context omitted.

With a risk of stating the obvious, this is a double standard simply because there are two standards - one for Google and one for others. I can't speak for the poster you were replying to, but whilst I see it as logical self-interested behaviour by site owners, it still feels unfair.

Whether or not it technically qualifies as a "double standard," in practice I don't see anything inherently unfair about it. If a stranger enters my house without my permission, that's trespassing. But there's nothing unfair about letting in someone who I invite over.

That's a terrible analogy. Your home is private, websites are not. The fact is that websites are posted online for all to see, so it's more like saying certain people at a park may take pictures while others are not allowed. That's unfair. If everyone could take pictures, it would be fair. Yes, someone with an old bright bulb camera might be annoying people, but nobody said "fair" meant all players would be nice or that having a "fair" policy would somehow be more beneficial to the website owner. It's not, that's why site owners are selective. So they have a double standard, but it's for their benefit, not that of the site visitors (be they human or bot).

Re: Web Scraping in 2016

#242

Post by encoderer »

codeddesign wrote: post #219
tangue wrote: post #200

Earlier quoted context omitted.

It would be ok if it wasn't "You can't scrap my site. Unless of course you're Google" this double standard drives me mad.

Double standard? The difference is that Google Bot is built on being unobtrusive. I can easily built a scraper that will quickly ddos a site. Linkedin for example...if they allow 10,000 people to send 100 scraping requests per second everyday then that is stolen bandwidth that Linkedin has to pay for and the scrapers get free data. The difference is that Google has standards in which site's unusually benefit from, no…

I agree that Googlebot is well behaved. When it detects your site is slowing down, it will back itself off. Unfortunately, this is often to your detriment.

In my experience, on a large site, Google will often slurp as much as you let it, upwards of hundreds of pages per second.

Re: Web Scraping in 2016

#243

Post by downandout »

ChuckMcM wrote: post #229
downandout wrote: post #206

Earlier quoted context omitted.

Actually it's been held that TOS violations are NOT subject to the criminal provisions of the CFAA.

Are you referring to the MySpace case? Or the July 2016 decision by the Ninth Circuit ( https://cdn.ca9.uscourts.gov/datastore/opinions/2016/07/05/1... ). In US v Nosal it seems like they come down in favor of a CFAA violation if the user acts in an unauthorized way. The author of the piece talks about bypassing captcha's which are, in one interpretation, a demand for authorization (by proving that you are a human an…

The case you're referring to is an entirely different set of circumstances. From the text:

"The panel held that the defendant, a former employee whose computer access credentials were revoked, acted “without authorization” in violation of the CFAA when he or his former employee co-conspirators used the login credentials of a current employee to gain access to computer data owned by the former employer and to circumvent the revocation of access. "

I think that case is unambiguous - this guy was using someone else's credentials to access secured systems after having been explicitly told that he could not. I was referring to the MySpace case.

I don't think these two cases are in conflict; IMO they are very different. Additionally, for our purposes in this comment thread, we're talking about scraping of publicly available websites by outside parties, not by former employees whose access has been explicitly revoked. That is different than either of these cases.

Re: Web Scraping in 2016

#244

Post by franciskim »

minimaxir wrote: post #2

Keep in mind that companies have sued for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: http://www.informationweek.com/software/social/linkedin-sues... OKCupid did a DMCA takedown for researchers releasing scraped data: https://www.engadget.com/2016/05/17/publicly-released-okcupi... Since both of these incidents, I now only scrape if it's a) through the API follow…

On the ethics side, I don't scrape large amounts of data - eg. giving clients lead gen (x leads for y dollars) - in fact, I have never done a scraping job and don't intend to do those jobs for profit.

For me it's purely for personal use and my little side projects. I don't even like the word scraping because it comes loaded with so many negative connotations (which sparked this whole comment thread) - and for a good reason - it's reflective of how the the demand in the market. People want cheap leads to spam, and that's bad use of technology.

Generally I tend to focus more on words and phases like 'automation' and 'scripting a bot'. I'm just automating my life, I'm writing a bot to replace what I would have to do on a daily basis - like looking on Facebook for some gifs and videos then manually posting them to my site. Would I spend an hour each and every day doing this? No, I'm much more lazier than that.

Who is anyone to tell me what I can and can't automate in my life?

Re: Web Scraping in 2016

#245

Post by atmosx »

siegecraft wrote: post #164
fake-name wrote: post #49

I do a significant amount of scraping for hobby projects, albeit mostly open websites. As a result, I've gotten pretty good a circumventing rate-limiting and most other controls. I suspect I'm one of those bad people your parents tell you to avoid - by that I mean I completely ignore robots.txt. At this point, my architecture has settled on a distributed RPC system with a rotating swarm of clients. I use RabbitMQ for…

I'm sure there'd be a ton of people that would love to pay to use your platform (who cares if the source is available, I don't want to run my own because once the code is written, it's ops thats hard). But then I suppose it would be hard to stay unnoticed.

Similar, paid solution: https://scrapinghub.com/crawlera/

Re: Web Scraping in 2016

#246

Post by nickysielicki »

fake-name wrote: post #195
siegecraft wrote: post #164

Earlier quoted context omitted.

I'm sure there'd be a ton of people that would love to pay to use your platform (who cares if the source is available, I don't want to run my own because once the code is written, it's ops thats hard). But then I suppose it would be hard to stay unnoticed.

Yeah, running this thing publicly would be a huge mess from a copyright perspective, since it literally re-hosts everything as a core part of how it works. As it is, I think I'm OK, since it's basically just a "website DVR" type thing, for my own use. Really, if nothing else, the project has been enormously educational for me. I've learnt a boatload about distributed systems, learned a bit of SQL, dicked about with d…

This project is really cool. Last year I was looking into open source projects that implement something like Readability so that I could scrape articles from my RSS feeds and turn them into plaintext. But I didn't find anything that blew me away. The best I got was stealing the implementation from Firefox, and I lost interest before I could make it worthwhile. (Now revisiting the idea, I wonder why I never thought of passing a user-agent from a mobile browser... Probably would have helped a lot.)

I see you don't have a license listed on GitHub. Do you have a license in mind for these?

Re: Web Scraping in 2016

#247

Post by tangue »

Analemma_ wrote: post #210
tangue wrote: post #200

Earlier quoted context omitted.

It would be ok if it wasn't "You can't scrap my site. Unless of course you're Google" this double standard drives me mad.

Why is it a double standard? Google scraping usually benefits the site with increased traffic and revenue, in a way most other scraping does not. Saying "you can scrape me if it benefits me" isn't totally in keeping with the principles of the open web, but it's not hypocritical.

Google is basically algorithms built on top of a scraping service. It's unfair to competitors (and potential disruptors) to restrict access to data that Google can fetch without limits.

Re: Web Scraping in 2016

#248

Post by profeta »

Analemma_ wrote: post #210
tangue wrote: post #200

Earlier quoted context omitted.

It would be ok if it wasn't "You can't scrap my site. Unless of course you're Google" this double standard drives me mad.

Why is it a double standard? Google scraping usually benefits the site with increased traffic and revenue, in a way most other scraping does not. Saying "you can scrape me if it benefits me" isn't totally in keeping with the principles of the open web, but it's not hypocritical.

because i may also come to the point where i am a direct competitor to google, but i will never get there because i can't scrap any site like they can.

your next argument may very well be a very racist one with the very same excuse you used above.

Re: Web Scraping in 2016

#249

Post by clevernickname »

fake-name wrote: post #165
kough wrote: post #124

Earlier quoted context omitted.

Yet another incredible technical achievement due to someone's quest for more porn ( https://github.com/fake-name/AutoTriever/blob/master/setting... ).

That's a separate project: - https://github.com/fake-name/ExHentai-Archival - https://github.com/fake-name/PatreonArchiver - https://github.com/fake-name/xA-Scraper - https://github.com/fake-name/DanbooruScraper Or... well, 4 separate projects. Whoops? At one point, a friend and I were looking at trying to basically replicate the google deep-dream neural net thing, only with a training set of porn. It turns out getti…

You're doing god's work.

Re: Web Scraping in 2016

#250

Post by slrz »

ysavir wrote: post #130
minimaxir wrote: post #82

Earlier quoted context omitted.

That analogy is not equitable. If you take photographs of a building while on the building's property , they have the right to tell you to stop, or call the police to escort you off if you refuse to do so.

Let's go with a more apt analogy: If you're entering a country, do its laws not apply to you until you've seen a copy of them? "Oh, sorry, no one told me theft is illegal here. Where does it say that? Oh, I see. Okay. I'll stop now. Thanks for letting me know." If you cross the border without necessary documents, does that country have no right to detain you, simply because you haven't checked the laws? Just because…

But when you plan on using it in a way that might not be intended, and you don't check the terms of service, well, that's on you.

I don't need to check your terms of services if I'm doing something that I'm allowed to do by law anyway; the TOS cannot deny me those rights (they might, of course, grant me additional rights provided that I follow certain conditions).

Post reply on HN