Do any of you who scrape fear retaliation from the sites you scrape? Maybe you are violating a ToS or scraping copyrighted text, and they cut off your IP. Thoughts?
Ask YC: What do you scrape? How do you scrape?
41–47 of 47 posts
Re: Ask YC: What do you scrape? How do you scrape?
#42Do any of you who scrape fear retaliation from the sites you scrape? Maybe you are violating a ToS or scraping copyrighted text, and they cut off your IP. Thoughts?
I think you have to take into consideration the TOS, copyright and also robots.txt. If you ignore these then its well within the site owners rights to do something about it - blocking you or further. We always look at the robots.txt file first and use that as our benchmark in terms of what they (the site) wish robots/crawlers to look at
Re: Ask YC: What do you scrape? How do you scrape?
#43Scraping EU public procurement contracts from the "Tenders Europa Daily" database ( http://ted.europa.eu/ ). There's more than a million documents with each document requiring up to two requests. Been at it for several weeks with a multithreaded scraper and we're almost through. Using Solvent (simile.mit.edu/solvent/) to generate xpath expressions and HtmlAgilityPack (www.codeplex.com/htmlagilitypack) to run the xpat…
Be careful here. The content is actually copyrighted. Whilst you can scrape it their T&Cs expressly forbid it. They sell licenses to access this information - the license is NOT expensive and they provide direct access to all the data in XML.
http://ted.europa.eu/Exec?DataFlow=ShowPage.dfl&Template...
Quote: "Reproduction is authorised provided the source is acknowledged. However, to prevent disruptions in service to our normal users from bulk downloads of TED data, we reserve the right to check for, and block, attempts to download excessive quantities of documents, particularly using automated or robot-like tools."
... they apparently chose not to exercise that right in this case, the scrape completed last night (all 18 GB of it).
Re: Ask YC: What do you scrape? How do you scrape?
#44anyone that complains about HTML scraping is a pussy. Seriously its trivial compared to what we had to do in the past. I like hpricot for ruby.
Re: Ask YC: What do you scrape? How do you scrape?
#45I've written a lot of this sort of program over the last 18 months. This is something that people are in need of all the time. I would say that there isn't yet a tool which does this to the level that customers want.
I use Mechanize, both in its Ruby and Python forms (I prefer Ruby) and plain old regular expressions to get the information that I want. Often times I will use a divide and conquer strategy by removing part of the web page (for example, the ) and successively paring it down to what I really want.
Javascript can be a problem. What I normally do is actually read the Javascript on the page, and then recreate that behavior in my Ruby code. Often times this means simply setting some form values (usually hidden) and then submitting the form.
Re: Ask YC: What do you scrape? How do you scrape?
#46I have a similar DSL built in Ruby that can be run by by either Mechanize or Watir. I highly recommend Watir if you need to scrape ajax.
Re: Ask YC: What do you scrape? How do you scrape?
#47http://gatherer.wizards.com with BeautifulSoup, the only parser I've found that can deal with this @$%^! HTML.