regex (runs)
Ask HN: What are best tools for web scraping?
211–220 of 243 posts
Re: Ask HN: What are best tools for web scraping?
#212Re: Ask HN: What are best tools for web scraping?
#213I use Node and either puppeteer[0] or plain Curl[1]. IMO Curl is years ahead of any Node.js request lib. For proxies I use (shameless plug!) https://gimmeproxy.com . [0] https://github.com/GoogleChrome/puppeteer [1] https://github.com/JCMais/node-libcurl
Really nice concept.
Re: Ask HN: What are best tools for web scraping?
#214Anyone who suggests a tool that can't understand JavaScript doesn't know what they are talking about You should be using Headless Chrome or Headless Firefox with a library that can control them in a user-friendly manner
There are a great many sites that degrade gracefully when JS support is not available. It makes absolutely no sense to waste the resources required to run a full headless browser when simple HTTP requests will retrieve the same information faster, more efficiently, and in a way that's easier to parallelize.
Yes, but a great many sites don't, and for those, you need Selenium + browser, full stop.
Re: Ask HN: What are best tools for web scraping?
#215Earlier quoted context omitted.
There are a great many sites that degrade gracefully when JS support is not available. It makes absolutely no sense to waste the resources required to run a full headless browser when simple HTTP requests will retrieve the same information faster, more efficiently, and in a way that's easier to parallelize.
A lot of times you can also watch the api calls JS pages (or apps) make and retrieve nice structured json data. I personally avoid executing js unless it's necessary, as it adds more complexity, and is noticeably more brittle.
Using an undocumented API, however, carries significant risk for production operations.
Re: Ask HN: What are best tools for web scraping?
#216I am really surprised nobody mentioned pyspider. It is simple, has a web dashboard and can handle JS pages. It can store data to a database of your choice. It can handle scheduling, recrawling. I have used it to crawl Google Play. 5$ Digital Ocean VPS with pyspider installed on it could handle millions of pages crawled, processed and saved to a database.
Re: Ask HN: What are best tools for web scraping?
#217Earlier quoted context omitted.
A lot of times you can also watch the api calls JS pages (or apps) make and retrieve nice structured json data. I personally avoid executing js unless it's necessary, as it adds more complexity, and is noticeably more brittle.
Using an undocumented API, however, carries significant risk for production operations.
If you're web scraping then you've already decided that this risk is worthwhile, it's already an undocumented API.
Re: Ask HN: What are best tools for web scraping?
#218I've just finished my research on web scraping for my company (took me about 7 days). I started with import.io and scrapinghub.com for point and click scraping to see if I could do it without writing codes. Ultimately, UI point and click scraping is for none-technical. There are many data you would find it hard to scrape. For example, lazada.com.my stores the product's SKU inside an attribute that looks like which I couldn't get. import.io's pricing is also something. I need to pay $999 a month for accessing API data is just too high.
So I decided to use scrapy, the core of scrapinghub.com.
I haven't written much python before but scrapy was very easy to learn. I wrote 2 spiders and run on scrapinghub (their serverless cloud). Scrapinghub support jobs scheduling and many other things at a cost. I prefer scrapinghub because in my team we don't have DevOps. It also supports Crawlera to prevent IP banning, Portia for point and click (still in beta, it was still hard to use), and Splash for SPA websites but it's buggy and the github repo is not under active maintenance.
For DOM query I use BeautifulSoup4. I love it. It's jQuery for python.
For SPA websites I wrote a scrapy middleware which uses puppeteer. The puppeteer is deployed on Amazon Lambda (1m free request first 365 days, more than enough for scraping) using this https://github.com/sambaiz/puppeteer-lambda-starter-kit
I am planning to use Amazon RDS to store scraped data.
Re: Ask HN: What are best tools for web scraping?
#219If you are a programmer, scrapy[0] will be a good bet. It can handle robots.txt, request throttling by ip, request throttling by domain, proxies and all other common nitty-gritties of crawling. The only drawback is handling pure javascript sites. We have to manually dig into the api or add a headless browser invocation within the scrapy handler. Scrapy also has the ability to pause and restart crawls [1], run the cra…
I've recently made a little project with scrapy (for crawling) and BeautifulSoup (for parsing html) and it works out great. One more thing to add to the above list are pipelines, they make downloading files quite easy.
I made a little BTC price ticker on an OLED with and arduino. I used BeautifulSoup to get the data. Went from knowing nothing about web scraping to getting the thing working pretty quick. Very easy to use.
Re: Ask HN: What are best tools for web scraping?
#220golang
I signed up for proxycrawl, used the javascript api to access a SPA website written in React and it just show a blank page. https://api.proxycrawl.com/?token=aDcC1lB-NZ5_r4vMSN-L3A&url... (I don't mind my token is exposed)