Craigslist, anyone?
I actually wrote a CL crawler in Ruby - https://github.com/marcamillion/craigslist-ruby-crawler
I used it to crawl for freelance web dev gigs, but it can be re-purposed to do anything.
I Don’t Need No Stinking API: Web Scraping For Fun and Profit
151–160 of 176 posts
Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit
#152What I wish I could do is capture Flash audio (or any audio) streams with my Mac. All I want is to listen to the audio-only content with an audio player when I'm out driving or jogging, etc. Audio-only content that has to be played off a web page usually runs into the contradiction that if I'm in a position to click buttons on my web browser (not driving, for example), I'm in a position to do my REAL work and have no…
Wireshark & RTMPDump may get you most of the way there.
Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit
#153I love HTML scraping. But Javascript???...The juiciest data sets these days are increasingly in JS. For the love of me i can't get around scraping JS :( I do know that Selenium can be used for this...but am yet to see a decent example for the same. Does anyone have any good resources/examples on JS scraping that they could share?? I would be eternally grateful.
If you are using Python, you can also use pyv8 to evaluate Javascript code.
There is also Ghost.py[0]
[0]http://jeanphix.me/Ghost.py/
If you are planning to use phantomjs, import sh and it's commandline all the way to payday :D
Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit
#154The issue with web scraping is that it relies on the scraper to keep up with changes made to the site. If a site owner changes the layout or implements a new feature, the programs depending on the scraper immediately fail. This is much less likely to happen when working with official APIs.
This is indeed painful. I was scraping the Pirate Bay last year for a school project, their HTML would occasionally change in subtle ways that would break my scraper for hours or days until I noticed it.
Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit
#155There are some recent federal cases (Weev http://www.wired.com/opinion/2012/11/att-ipad-hacker-when-em..., Aaron Swartzhttp://www.wired.com/threatlevel/2012/09/aaron-swartz-felony..., and a prosecution of scalpers http://www.wired.com/threatlevel/2010/07/ticketmaster/) that view scraping as a felony hacking offense. The feds think that an attempt to evade CAPTCHAS, IP and MAC blocks is a felony worthy of years in prison.
In fact, the feds might think that clearing your cookies or switching browsers to get another 10 free articles from the NYTimes is also felony hacking.
Which is to say, be careful what you admit to in this forum AND how you characterize what you are doing in your private conversations and e-mails.
Weev now faces a decade or more in prison because he drummed up publicity by sending emails to journalists that used the verb "stole".
Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit
#156Earlier quoted context omitted.
Headless: Xvfb on Linux. (Virtual Framebuffer, let's you run apps that require a GUI) You can use one of the many options that include Webkit (like phantom.js, the capybara-webkit gem, or Selenium if you want a real browser like Firefox to do the work)
PhantomJS doesn't need Xvfb anymore it can run headless without this dependency.
well, it wasn't due to anything in phantomjs actually -- it was because qt introduced project lighthouse & their qt platform abstraction. project lighthouse was a fork that got integrated into qt 4.8 that phantomjs includes. (you can see the entire qt source tree if you git-clone phantomjs)
Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit
#157There are some recent federal cases (Weev http://www.wired.com/opinion/2012/11/att-ipad-hacker-when-em... , Aaron Swartz http://www.wired.com/threatlevel/2012/09/aaron-swartz-felony... , and a prosecution of scalpers http://www.wired.com/threatlevel/2010/07/ticketmaster/ ) that view scraping as a felony hacking offense. The feds think that an attempt to evade CAPTCHAS, IP and MAC blocks is a felony worthy of years in…
Very good point, I've added the following disclaimer:
While scraping can sometimes be used as a legitimate way to
access all kinds of data on the internet, it’s also important
to consider the legal implications. As was pointed out in the
comments on HN[1], there are many cases where scraping data
may be considered illegal, or open you to the possibility of
being sued. Similar to using a firearm, some uses of web
scraping techniques can be used for utility or sport, while
others can land you in jail. I am not a lawyer, but you
should be smart about how you use it.
Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit
#158Earlier quoted context omitted.
They are also using title, description, some snippets from the page and taking a cached version of the site and images you can view without having to visit the site itself. They are also using this data as a product to sell advertising against. If there wasn't so much benefit for most of all sites to be in search engine indexes you would thinking at least some would object to this scraping. There would be lots of oth…
Google is even moving into the territory of scraping content to display. Relevant wikipedia snippets are now being displayed on the search page as a side bar. While Wiki probably doesn't care...there are plenty of other sites that would not like Google to scrape the content and display it on the search page.
Yeah, Wikipedia is creative commons so that should be okay?
You are right though I wonder if they have the rights to sports results and weather that they are pulling.
They have even convinced us all to go mark up our page to help them pull stuff like ratings and reviews out.
Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit
#159To all HN: All this being said, how do we prevent our sites from being scraped in this way? What can you not get around, and what are the potential uses for an 'unscrapeable' site to your mind.
if you don't want something to be scrapped, don't publish it on the internet. scrapping prevention reminds me of blocking right-click and other ridiculous solutions back in the day. hey, if I can view it, it means the data reached my end point.
Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit
#160Earlier quoted context omitted.
Another proof that spending 15 minutes on research can save you days in development and production.
And your comment is another proof that people tend to assume everyone else is an idiot.
Seems like one a situation where the language in one cultural context would be insulting, but in another, is merely a literal statement.