I wrote some blog post about Java web scraping here :
https://ksah.in/introduction-to-web-scraping-with-java/
As others said, phantomJS (and now headless Chrome) are good tools to deal with heavy js websites
Ask HN: What are best tools for web scraping?
71–80 of 243 posts
Re: Ask HN: What are best tools for web scraping?
#72I just tried puppeteer yesterday for the first time. It seems to work very well. My only complaint is that it is very new and does now have a plethora of examples.
I previously have used WWW::Mechanize in the Perl world, but single page applications with Javascript really require something with a browser engine.
Re: Ask HN: What are best tools for web scraping?
#73I've actually wrote about this! General tips that I've found from doing more than a few projects [0], and then an overview of Python libraries I use [1]. If you don't want to clock on the links, requests and BeautifulSoup / lxml is all you need 90% of the time. Throw gevent in there and you can get a lot of scraping done in not as much time as you think it would take. And as long as we're talking about web scraping,…
> BeautifulSoup / lxml When should one use one or the other, would you say?
BeautifulSoup has a friendly API, but it is slow. It has a lxml backend, however.
If you're familiar with writing XPath queries, lxml is great.
Re: Ask HN: What are best tools for web scraping?
#74I maintain about 8 crawlers and I use only vanilla Python
I have a function to help me search :
def find_r(value, ind, array,stop_word):
indice = ind
for i in array:
indice = value.find(i,indice)+1
end = value.find(stop_word,indice)
return value[indice: end], end
resulting_text , end_index = find_r(string, start_index, [""], "Re: Ask HN: What are best tools for web scraping?
#75If you need to interpret javascript, or otherwise simulate regular browsing as closely as possible, you may consider running a browser inside a container and controlling it with selenium. I have found it’s necessary to run inside the container if you do not have a desktop environment. This is better suited for specific use cases rather than mass collection because it is slower to run a full browsing stack than to only operate at the HTTP layer. I have found that alternatives like phantomJS are hard to debug. Consider opening VNC on the container for debugging. Containers like this that I know of are SeleniumHQ and elgalu/selenium.
Re: Ask HN: What are best tools for web scraping?
#76This is perhaps the fastest way to screenscrape a dynamically executed website. 1. First go get and run this code, which allows immediate gathering of all text nodes from the DOM: https://github.com/prettydiff/getNodesByType/blob/master/get... 2. Extract the text content from the text nodes and ignore nodes that contain only white space: let text = document.getNodesByType(3), a = 0, b = text.length, output = []; do {…
And how do you do #1? Node, I presume?
Re: Ask HN: What are best tools for web scraping?
#77`clj-http`, `enlive`, `cheshire` in case of `clojure` worked fine for me
and 'hickory' [https://github.com/davidsantiago/hickory] to work with the site data however you want.
Re: Ask HN: What are best tools for web scraping?
#78I’ve been using puppeteer to scrape and it’s been fantastic. Since it’s a headless browser, it can handle SPA just as well as server side loaded traditional websites. It’s also incredibly easy to use with async/await.
Re: Ask HN: What are best tools for web scraping?
#79If you need to scrape content from complex JS apps (eg. React) where it doesn't pay to reverse engineer their backend API (or worse, it's encrypted/obfuscated) you may want to look at CasperJS.
It's a very easy to use frontend to PhantomJS. You can code your interactions in JS or CoffeeScript and scrape virtually anything with a few lines of code.
If you need crawling, just pair a CasperJS script with any spider library like the ones mentioned around here.
Re: Ask HN: What are best tools for web scraping?
#80I just tried puppeteer yesterday for the first time. It seems to work very well. My only complaint is that it is very new and does now have a plethora of examples. I previously have used WWW::Mechanize in the Perl world, but single page applications with Javascript really require something with a browser engine.
[deleted]