I’ve been using puppeteer to scrape and it’s been fantastic. Since it’s a headless browser, it can handle SPA just as well as server side loaded traditional websites. It’s also incredibly easy to use with async/await.
I assume this puppeteer:
Ask HN: What are best tools for web scraping?
81–90 of 243 posts
Re: Ask HN: What are best tools for web scraping?
#82A good host xD Preferably one that doesn't mind giving you a bunch of IPs, and if they do, don't charge a fortune for them. Then you can worry about what software you're gonna use.
Which hosts have you used, or would you recommend?
OVH
You can get upto 256 IPs per server and _not_ pay monthly fees -- just a $3 upfront setup charge.
You're welcome xD
Re: Ask HN: What are best tools for web scraping?
#83I recently stumbled across http://go-colly.org/, that looks well thought out and simple to use. It seems like a slimmed down Go version of Scrapy.
Re: Ask HN: What are best tools for web scraping?
#84If you know Java, then my go to library is Jsoup https://jsoup.org/
It lets you use jQuery-like selectors to extract data.
Like this: Elements newsHeadlines = doc.select("#mp-itn b a");
Re: Ask HN: What are best tools for web scraping?
#85Earlier quoted context omitted.
Yes. It beats building up your own crawler that handles all the edge cases. That said, before you reach the limits of scrapy, you will more likely be restricted by preventive measures put in place by twitter(or any other large website) to limit any one user hogging too much resources. Services like cloudflare or similar are aware of all the usual proxy servers and such and will immediately block such requests.
So how to do it ? You have to become google/bing ?
One approach, that is commonly mentioned in this thread is to simulate a behavior of a normal user as much as possible. For instance rendering the full page (including JS, CSS, ...) which is far more resource intensive than just downloading the HTML page.
However if you're crawling big platforms, there are often ways in that can scale and be undetected for very long periods of time. Those include forgotten API endpoints that were build for some new application that was dismissed after a time, mobile interface that taps into different endpoints, obscure platform specific applications (e.g. playstation or some old version of android). Older and larger the platform is, the more probable is that they have many entry points they don't police at all or at least very lightly.
One of the most important rules of scrapping is to be patient. Everyone is anxious to get going as soon as they can, however once you start pounding on a website, consequently draining their resources, they will take measures against you and the whole task will get way more complicated. Would you have the patience and make sure you're staying within some limits (hard to guess from the outside), you will be eventually able to amass large datasets.
Re: Ask HN: What are best tools for web scraping?
#86The best tool for web scraping, for me, is something easy to deploy and redeploy; and something that doesn't rely on three working programs--eliminating selenium sounds great.
For those reasons I like https://github.com/knq/chromedp
Re: Ask HN: What are best tools for web scraping?
#87For someone on a Javascript stack, I highly recommend combining a requester (e.g., "request" or "axios") with Cheerio, a server-side jQuery clone. Having a familiar, well-known interface for selection helps a lot.
We use this stack at WrapAPI (https://wrapapi.com), which we highly recommend as a tool to turn webpages into APIs. It doesn't completely do all the scraping (you still need to write a script), but it does make turning a HTML page into a JSON structure much easier.
Re: Ask HN: What are best tools for web scraping?
#88I did a little web scraping project a few years ago using:
* cURL
* regex
Re: Ask HN: What are best tools for web scraping?
#89If you use PHP, Simple HTML DOM[0] is an awesome and simple scraping library. [0] http://simplehtmldom.sourceforge.net/
I also have used Simple HTML Dom
One thing I haven't worked on yet is waiting for stuff to load if that is a problem. Otherwise you try to limit hitting a site either using sleep/CRON
Re: Ask HN: What are best tools for web scraping?
#90If you use PHP, Simple HTML DOM[0] is an awesome and simple scraping library. [0] http://simplehtmldom.sourceforge.net/
I also have used Simple HTML Dom
One thing I haven't worked on yet is waiting for stuff to load if that is a problem. Otherwise you try to limit hitting a site either using sleep/CRON
What's also interesting is session tokens, one site I was able to hunt down the generated token bread crumb which JS produced, but it wasn't valid. Still had to visit the site, interesting.