61–70 of 234 posts
Earlier quoted context omitted.
What page were you trying to hit? We'll check it out
Pages buried in here: https://fannin4.wcjc.edu/ The course catalog is public, so no login is needed. I want to scrape various data related to courses, to populate forms automatically and such.
the reason i ever have to write a scraper is because of pagination. while this looks awesome, i'll have to stick to scraping until that is solved. :(
Earlier quoted context omitted.
Thanks... yes, public data from governments is a great use case. Often a lot of apps built using scrapers will wind up driving up traffic/ sales a the source site so it's okay. We want to do responsible web scraping, so will respect webmasters robots.txt files to make sure it's legal.
I love the execution, but I also see inherent problems. Robots.txt is just a convention to advise crawlers. I'm confident most sites explicitly state this is against their terms of service. You will encounter terms along the lines of: "Unauthorized uses of the Site also include, without limitation, those listed below. You agree not to do any of the following, unless otherwise previously authorized by us in writing: U…
I built something very similar last year, but sadly never got around to polishing and launching it: http://exfiltrate.org/ (There's a prototype of an API generator hidden in a menu somewhere but it's nowhere near production ready)
Can't wait to play around with this tonight.
Suggestion. Allow one to select images.
Are you familiar with ScraperWiki? I'm wondering how your work fits in with it. Edit: looks like they've moved away from that space, but have an old version available at: https://classic.scraperwiki.com/
The software is abandoned, but their algorithms are described in a paper:
http://people.csail.mit.edu/dfhuynh/research/papers/uist2006-augmenting-web-sites.pdfThe Simile group at MIT did something similar back around 2006. Automatic identification of collections in web pages (repeated structures), detection of fields by doing tree comparisons between the repeated structures, and fetching of subsequent pages. The software is abandoned, but their algorithms are described in a paper: http://people.csail.mit.edu/dfhuynh/research/papers/uist2006-augmenting-web-sites.pdf
This is excellent. Even it if doesn't work for scraping all sites, it simplifies the average use case so much that it's not even funny. Feature proposal: deal with pagination.