Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping in 2016

franciskim.co

Post reply on HN
Print view View original

281–290 of 402 posts

Re: Web Scraping in 2016

#281

Post by Trundle »

Raphmedia wrote: post #55
Nicksil wrote: post #44

Earlier quoted context omitted.

I don't follow your argument. How does one get their scraper access to data they would otherwise not be able to access through 'normal' browsing techniques?

Example I know of: You can scrape your competitor's Facebook pages since their creation and output nice graphs of which posts generated what kind of likes and suscriptions. This data is usually limited to the owner of the page.

If someone was to sit down and use paper + pencil + time to accomplish the same thing, would you still have issue with it? It's publicly available data. Should you also not watch your competitors television ads or walk in to their physical store and browse around?

Re: Web Scraping in 2016

#282

Post by lugg »

bendbro wrote: post #220
nmstoker wrote: post #214

Earlier quoted context omitted.

With a risk of stating the obvious, this is a double standard simply because there are two standards - one for Google and one for others. I can't speak for the poster you were replying to, but whilst I see it as logical self-interested behaviour by site owners, it still feels unfair.

There isn't: the function for this standard includes expected benefit as an input. Every standard has inputs, so that certainly isn't the quality for making something a double standard. The only remaining quality is how unfair it feels, so it would probably be better to just address that, since it is obviously the only thing you disagree about.

With that logic decreased wages for women are not a double standard due to the potential for maternity leave affecting their output at work.

This is a double standard plain and simple, and a very dangerous one at that.

Re: Web Scraping in 2016

#283

Post by sjwright »

tangue wrote: post #200
minimaxir wrote: post #2

Keep in mind that companies have sued for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: http://www.informationweek.com/software/social/linkedin-sues... OKCupid did a DMCA takedown for researchers releasing scraped data: https://www.engadget.com/2016/05/17/publicly-released-okcupi... Since both of these incidents, I now only scrape if it's a) through the API follow…

It would be ok if it wasn't "You can't scrap my site. Unless of course you're Google" this double standard drives me mad.

As the owner of a large website, I don't care what you think. I block by default and whitelist when I decide it's in my interest.

If you don't think this is reasonable, chances are you've never run a large website, or analyzed the logs of a large website. You'd be astonished how much robotic activity you'll receive. If left unchecked it can easily swamp legitimate traffic.

Unless you have a way for me to automatically identify "honourable" scrapers such as yourself as distinct from the thousands upon thousands of extremely dodgy scrapers from across the world, my policy shall remain.

Re: Web Scraping in 2016

#284

Post by uptown »

fake-name wrote: post #49

I do a significant amount of scraping for hobby projects, albeit mostly open websites. As a result, I've gotten pretty good a circumventing rate-limiting and most other controls. I suspect I'm one of those bad people your parents tell you to avoid - by that I mean I completely ignore robots.txt. At this point, my architecture has settled on a distributed RPC system with a rotating swarm of clients. I use RabbitMQ for…

What would the rough costs be to run the 800k UA scenario?

Re: Web Scraping in 2016

#285

Post by sjwright »

nercht12 wrote: post #241
mgalka wrote: post #225

Earlier quoted context omitted.

Whether or not it technically qualifies as a "double standard," in practice I don't see anything inherently unfair about it. If a stranger enters my house without my permission, that's trespassing. But there's nothing unfair about letting in someone who I invite over.

That's a terrible analogy. Your home is private, websites are not. The fact is that websites are posted online for all to see, so it's more like saying certain people at a park may take pictures while others are not allowed. That's unfair. If everyone could take pictures, it would be fair. Yes, someone with an old bright bulb camera might be annoying people, but nobody said "fair" meant all players would be nice or t…

How about the analogy of an art gallery disallowing photography? Is the gallery being hypocritical when they allow the local paper to take photos for publicity, or when they permit an archivist that has a known reputation to take photos for archival purposes?

Re: Web Scraping in 2016

#286

Post by _c_ »

tangue wrote: post #200
minimaxir wrote: post #2

Keep in mind that companies have sued for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: http://www.informationweek.com/software/social/linkedin-sues... OKCupid did a DMCA takedown for researchers releasing scraped data: https://www.engadget.com/2016/05/17/publicly-released-okcupi... Since both of these incidents, I now only scrape if it's a) through the API follow…

It would be ok if it wasn't "You can't scrap my site. Unless of course you're Google" this double standard drives me mad.

Worse is that Google tries to stop scraping. It's like they don't want anyone to see past the first page of results.

They could scrape your website and then they prevent you form scraping your own data back.

The whole process is silly; it reflects the duct tape and chicken wire nature of the www.

No one should have to "scrape" or "crawl".

Data should be put into a open universal format (no tags) and submitted when necessary (rsynced) to a public access archive, mirrored around the world.

This to bridge the gap until we reach a more content addressable system (cf. location based).

Clients (text readers, media players, whatever) can download and transform the universally formatted data into markup, binary, etc. -- whatever they wish, but all the design creativity and complexity of "web pages" or "web apps" can be handled at the network edge, client-side.

"Crawling" should not be necessary.

No one should have to store HTML tags and other window dressing for data.

Dream on.

Re: Web Scraping in 2016

#287

Post by sjwright »

profeta wrote: post #248
Analemma_ wrote: post #210

Earlier quoted context omitted.

Why is it a double standard? Google scraping usually benefits the site with increased traffic and revenue, in a way most other scraping does not. Saying "you can scrape me if it benefits me" isn't totally in keeping with the principles of the open web, but it's not hypocritical.

because i may also come to the point where i am a direct competitor to google, but i will never get there because i can't scrap any site like they can. your next argument may very well be a very racist one with the very same excuse you used above.

And if you have some way to identify yourself as a potential competitor to google and not some jackass trying to scrape email addresses or spam comments forms, I'm all ears.

Re: Web Scraping in 2016

#288

Post by IANAD »

> But if you are automating your exact actions that happen via a browser, can this be blocked?

Yes, by checking times between actions and number of actions in a time period, and blocking atypical activity. I was IP banned from a site once for a few months, after trying to scrape it too much and hitting links on the site that were hidden from humans.

The random wait settings specified in the post are better than nothing, but still too flimsy. You would need to put hours between requests, only request during a certain 15 hour periods, take days off, and eventually you aren't scraping regularly enough to do much good.

Scraping is not an API, and I should know- I used to do it for a living. Its unreliable. It requires constant maintenance. APIs can break too, but they are meant for the sort of consumption you are trying for.

If you scrape for a living, only do it as a side job.

Re: Web Scraping in 2016

#289

Post by sjwright »

madamelic wrote: post #18
hobs wrote: post #13

Earlier quoted context omitted.

This post is kind of crazy, aggrandizing bad behavior and misuse of other's resources against their will. Scraping against the TOS is super bad netizen stuff, and I dont think people should be posting positive reviews of people doing this. Breaking captchas and the like is basically blackhat work and should be looked down upon, not congratulated as I see in this thread.

>Scraping against the TOS is super bad netizen stuff, and I dont think people should be posting positive reviews of people doing this. Breaking captchas and the like is basically blackhat work and should be looked down upon, not congratulated as I see in this thread. Not really. Scraping, in my opinion, isn't black hat unless you are actually affecting their service or stealing info. If you are slamming the site with…

> Scraping, in my opinion, isn't black hat unless you are ... stealing info.

And as a webmaster, how can I tell the difference before it's too late?

Re: Web Scraping in 2016

#290

Post by wumpus »

cookiecaper wrote: post #277
greglindahl wrote: post #209

Earlier quoted context omitted.

Actually, such database laws are rare. The US and Canada don't have one. See Feist v. Rural Telephone for an example of databases getting scraped & the scraper winning in court.

To be clear, this wasn't a scraper in the networked computer sense. It's actually a perfect example of how meatspace safeguards don't translate because law is not equipped to handle the nature of cyberspace.

I don't see how that's true at all. Running a meatspace telephone book through a sheet-fed scanner and OCR isn't wildly different from scraping a website.
Post reply on HN