For non developers dexi.io is great.
Ask HN: What are best tools for web scraping?
31–40 of 243 posts
Re: Ask HN: What are best tools for web scraping?
#32I am currently building a platform for renting scrapers.
It's not released yet but stay tuned.
Re: Ask HN: What are best tools for web scraping?
#33`clj-http`, `enlive`, `cheshire` in case of `clojure` worked fine for me
Re: Ask HN: What are best tools for web scraping?
#34I use Colly[0][1] which is a young but decent scraping framework for Golang.
[0] http://go-colly.org/ [1] https://github.com/gocolly/colly
Re: Ask HN: What are best tools for web scraping?
#35[deleted]
Re: Ask HN: What are best tools for web scraping?
#36If you are a programmer, scrapy[0] will be a good bet. It can handle robots.txt, request throttling by ip, request throttling by domain, proxies and all other common nitty-gritties of crawling. The only drawback is handling pure javascript sites. We have to manually dig into the api or add a headless browser invocation within the scrapy handler. Scrapy also has the ability to pause and restart crawls [1], run the cra…
Would you still recommend Scrapy if the task wasn't specifically crawling?
Re: Ask HN: What are best tools for web scraping?
#37beautifulsoup
Using this as well with Requests to automate eBay/gumtree/craigslist. Works very well
Any details on this anywhere, or is it not for public consumption? I'm just getting started in Python and want to do something with Gumtree and eBay as an idea to help me in a different sphere.
Re: Ask HN: What are best tools for web scraping?
#38I use R since that is the language I use mostly httr and rvest. Edit I missed typing rvest thanks for the comments you use the two together. https://cran.r-project.org/web/packages/httr/vignettes/quick...
Rvest is also another nice option in R.
Re: Ask HN: What are best tools for web scraping?
#39I've actually wrote about this! General tips that I've found from doing more than a few projects [0], and then an overview of Python libraries I use [1]. If you don't want to clock on the links, requests and BeautifulSoup / lxml is all you need 90% of the time. Throw gevent in there and you can get a lot of scraping done in not as much time as you think it would take. And as long as we're talking about web scraping,…
> BeautifulSoup / lxml
When should one use one or the other, would you say?
Re: Ask HN: What are best tools for web scraping?
#40I had great experience with www.apify.com.