I made this https://www.drupal.org/project/example_web_scraper and produced the underlying code many years ago. The idea is to map xpath queries to your data model and use some reusable infrastructure to simply apply it. It was very good, imho (for what it was). (I'm writing this comment since I don't see any other comments with the words map or model :/ )
Ask HN: What are best tools for web scraping?
221–230 of 243 posts
Re: Ask HN: What are best tools for web scraping?
#222Nokogiri
Re: Ask HN: What are best tools for web scraping?
#223No one has mentioned it so I will: consider Lynx, the text-mode web-browser. Being command-line you can automate with Bash or even Python. I have used it quite happily to crawl largeish static sites (10,000+ web pages per site). Do a `man lynx` the options of interest are -crawl, -traversal, and -dump. Pro tip - use in conjunction with HTML TIDY prior to the parsing phase (see below).
I have also used custom written Python crawlers in a lot of cases.
The other thing I would emphasize is that a web scraper has multiple parts, such as crawling (downloading pages) and then actually parsing the page for data. The systems I've set up in the past typically are structured like this:
1. crawl - download pages to file system 2. clean then parse (extract data) 3. ingest extracted data into database 4. query - run adhoc queries on database
One of the trickiest things in my experience is managing updates. So when new articles/content are added to the site you only want to have to get and add that to your database, rather than crawl the whole site again. Also detecting updated content can be tricky. The brute force approach of course is just to crawl the whole site again and rebuild the database - not ideal though!
Of course, this all depends really on what you are trying to do!
Re: Ask HN: What are best tools for web scraping?
#224Earlier quoted context omitted.
I’m crawling around 80-120M per month and the price for me fits my needs. But I suggest that you contact them if you have special needs or requirements. Also you have to consider the amount of work, time and money that you will save by not maintaining your own system to avoid blocks and bans from the websites you are trying to crawl. With them you just call an API endpoint and you don't have to care about all that
Thanks. One follow up questions, in JS heavy pages, how operations like infinite scrolling etc are exposed/executed?
I can't tell, I haven't scraped any page with infinite scrolling yet
Re: Ask HN: What are best tools for web scraping?
#225One of the important avenues to scrape AJAX heavy and phantomjs avoiding websites is using the google chrome extension support. They can mirror the dom and send it to an external server for processing where we can use python lxml to xpath to appropriate nodes. This worked for me to scrape Google, before we hit the capatcha. If anyone is interested, i can share code i wrote to scrape websites ! If you can scrape findt…
> This worked for me to scrape Google, before we hit the capatcha. If Google wanted to give back something to the community, it would offer cheap automated searches (current prices are absurd). Another thing - more depth after the first 1000 results. Sometimes you want to know the next result. We shouldn't need to do all these stupid things to batch query a search engine, it should be open. That makes it all the more…
Agree 100% too.
As for 'federated search engine' - it's not 'federated' per se but check out Gigablast search engine. Open source (source on GitHub) and a TOTALLY AWESOME piece of software written by one guy. You can do good searches at the Gigablast site[1], or set up your own search engine. Gigablast also offers an API (I may be wrong but I think DuckDuckGo uses that API for some tasks).
Re: Ask HN: What are best tools for web scraping?
#226I agree with others, with curl and the likes you will hit insurmountable roadblocks sooner or later. It's better to go full headless browser from the start.
I use a python->selenium->chrome stack. The Page Object Model [0] has been a revelation for me. My scripts went from being a mess of spaghetti code to something that's a pleasure to write and maintain.
[0] https://www.guru99.com/page-object-model-pom-page-factory-in...
Re: Ask HN: What are best tools for web scraping?
#227i wrote a tool: PhantomJsCloud.com
it's getting a little long in the tooth, but I will be updating it soon to use a Chrome based renderer. If you have any suggestions, you can leave it here or PM me :)
Re: Ask HN: What are best tools for web scraping?
#228I would recommend using Headless Chrome along with a library like puppeteer[0]. You get the advantage of using a real browser with which you run pages' javascript, load custom extensions, etc. [0]: https://github.com/GoogleChrome/puppeteer
Simple and straight forward, +1
Re: Ask HN: What are best tools for web scraping?
#229I've had a surprising amount of success with the HTML Agility Pack in .net, if you have a decent understanding of HTML it's pretty usable.
same. I'm a .NET person and i do web scraping stuff on the side, HTML Agility Pack has been easy to pick up.
Re: Ask HN: What are best tools for web scraping?
#230For non-coders, import.io is great. However, they used to have a generous free plan that has since went away (you are limited to 500 records now). Still a great product, problem is they don't have a small plan (starts at $299/month and goes up to $9,999).
I was looking at services in this area a few weeks ago to automate a small need I had and ran across these guys. They offer a free 5,000 monthly request basic plan. I gave it a try, worked fine (I ended up building my own solution for greater control). It's just for scraping open graph (with some fall-back capability) tags though. https://www.opengraph.io/
I use Grepsr. Really recommend, they have a Chrome extension that works like Kimono. Really easy for non technical people. If you have someone in Marketing or whatever that needs some data, maybe the only thing that they need to know is to use CSS Selectors and so on.