Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping in 2016

franciskim.co

Post reply on HN
Print view View original

291–300 of 402 posts

Re: Web Scraping in 2016

#291

Post by sandGorgon »

minimaxir wrote: post #2

Keep in mind that companies have sued for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: http://www.informationweek.com/software/social/linkedin-sues... OKCupid did a DMCA takedown for researchers releasing scraped data: https://www.engadget.com/2016/05/17/publicly-released-okcupi... Since both of these incidents, I now only scrape if it's a) through the API follow…

The code is pretty cool. Thanks for releasing that! May I ask why you built your own scraper infrastructure and not build it on top of a known framework like scrapy (which is in python as well).

Re: Web Scraping in 2016

#292

Post by encoderer »

Guest98123 wrote: post #274
encoderer wrote: post #263

Earlier quoted context omitted.

Have you seen correlation between rankings and crawl rate?

This particular site is top 5,000 Alexa. The content changes every minute, and Google is fast at picking up those changes. The last cache of the homepage was 7 minutes ago from Google. There's definitely a correlation between my sites' Google rankings, their organic traffic, and their crawl rate. The other sites I run are Alexa top 30,000 and top 100,000. They all feature dynamically changing content, but Google is d…

I've seen rankings drops following drops in crawl rate.

Re: Web Scraping in 2016

#293

Post by greglindahl »

sjwright wrote: post #287
profeta wrote: post #248

Earlier quoted context omitted.

because i may also come to the point where i am a direct competitor to google, but i will never get there because i can't scrap any site like they can. your next argument may very well be a very racist one with the very same excuse you used above.

And if you have some way to identify yourself as a potential competitor to google and not some jackass trying to scrape email addresses or spam comments forms, I'm all ears.

A majority of the websites that blekko, a google competitor, contacted to ask for robots.txt access ignored us.

Re: Web Scraping in 2016

#294

Post by pault »

esac wrote: post #279
pault wrote: post #217

Earlier quoted context omitted.

You can also build a chrome extension if you need to navigate to multiple pages and use a long-running scraping process. I've done this several times and it's really easy to get one up and running if you use an extension boilerplate (30 minutes tops).

do you have something? i was going to write the very same extension (but distributed so i could add it to my pc and my friends) but never did that

This is the boilerplate I used last time: http://extensionizr.com

Re: Web Scraping in 2016

#295

Post by greglindahl »

fareesh wrote: post #174
hk__2 wrote: post #135

Earlier quoted context omitted.

Yes, everybody scrapes the prices of the others.

With stuff like Facebook opengraph (e.g. og:price) and other meta tags meant to help search engines and social networks get this sort of data to display inline, do you think it's inevitable that complex scraping will no longer be needed in a practical sense since everyone will be inadvertently optimizing their markup in a way that you could write a really simple parser to grab the data?

Given that FB's bot identifies itself, no, eventually some websites will present og: markup only to FB's bot.

Re: Web Scraping in 2016

#297

Post by sp527 »

lugg wrote: post #282
bendbro wrote: post #220

Earlier quoted context omitted.

There isn't: the function for this standard includes expected benefit as an input. Every standard has inputs, so that certainly isn't the quality for making something a double standard. The only remaining quality is how unfair it feels, so it would probably be better to just address that, since it is obviously the only thing you disagree about.

With that logic decreased wages for women are not a double standard due to the potential for maternity leave affecting their output at work. This is a double standard plain and simple, and a very dangerous one at that.

Your example is a case of discrimination, but the economic rationale is unquestionable. There is a tremendous upfront cost for new employees, who are not valuable contributors for some lengthy ramp up period and furthermore accrue experience over the course of employment. So the lifetime value curve for any given employee is typically skewed left.

Re: Web Scraping in 2016

#298

Post by cookiecaper »

wumpus wrote: post #290
cookiecaper wrote: post #277

Earlier quoted context omitted.

To be clear, this wasn't a scraper in the networked computer sense. It's actually a perfect example of how meatspace safeguards don't translate because law is not equipped to handle the nature of cyberspace.

I don't see how that's true at all. Running a meatspace telephone book through a sheet-fed scanner and OCR isn't wildly different from scraping a website.

It's different because you don't contact another party's server to do it. The CFAA makes it illegal to "exceed authorized access" to networked computers. "Authorized access" is whatever the server's owner says it is. That's why the copyright status of factual accumulations isn't a protection for internet scraping.

If Feist v. Rural occurred now and Rural, like most companies, kept their information in a database online, Feist would lose not for copyright infringement, but for exceeding authorized access to Rural's server.

Re: Web Scraping in 2016

#299

Post by greglindahl »

cookiecaper wrote: post #298
wumpus wrote: post #290

Earlier quoted context omitted.

I don't see how that's true at all. Running a meatspace telephone book through a sheet-fed scanner and OCR isn't wildly different from scraping a website.

It's different because you don't contact another party's server to do it. The CFAA makes it illegal to "exceed authorized access" to networked computers. "Authorized access" is whatever the server's owner says it is. That's why the copyright status of factual accumulations isn't a protection for internet scraping. If Feist v. Rural occurred now and Rural, like most companies, kept their information in a database onli…

You've made a large number of authoritative-sounding comments on this story... and this one, like many of the others, is a guess.

Re: Web Scraping in 2016

#300

Post by sjwright »

greglindahl wrote: post #293
sjwright wrote: post #287

Earlier quoted context omitted.

And if you have some way to identify yourself as a potential competitor to google and not some jackass trying to scrape email addresses or spam comments forms, I'm all ears.

A majority of the websites that blekko, a google competitor, contacted to ask for robots.txt access ignored us.

I agree, it's a difficult conundrum. It sucks.
Post reply on HN