Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: WrapAPI v2 – Build APIs, scrapers, bots on any website

wrapapi.com

Post reply on HN
Print view View original

21–30 of 62 posts

Re: Show HN: WrapAPI v2 – Build APIs, scrapers, bots on any website

#21

Post by supermdguy »

zackify wrote: post #15

Doesn't work at all with JS. This is a big thing on many sites now. Also, since that is the case, you could build this in a few hours using something like: https://github.com/bda-research/node-crawler . Yes, it would have no gui, so you lose that.

I've had issues with web scraping with content generated from JS, and I just ran it through PhantomJS, then extracted the rendered HTML.

Re: Show HN: WrapAPI v2 – Build APIs, scrapers, bots on any website

#22

Post by RandomBookmarks »

zackify wrote: post #15

Doesn't work at all with JS. This is a big thing on many sites now. Also, since that is the case, you could build this in a few hours using something like: https://github.com/bda-research/node-crawler . Yes, it would have no gui, so you lose that.

If JS a problem for you, try Kantu. It works with screenshots and uses OCR for scraping. The beauty is that it works with any kind of site. But clearly, the speed can not match a node.js or perl based scraper (mechanize etc), so it is not suitable for high volumes.

Re: Show HN: WrapAPI v2 – Build APIs, scrapers, bots on any website

#24

Post by gardnr »

RandomBookmarks wrote: post #22
zackify wrote: post #15

Doesn't work at all with JS. This is a big thing on many sites now. Also, since that is the case, you could build this in a few hours using something like: https://github.com/bda-research/node-crawler . Yes, it would have no gui, so you lose that.

If JS a problem for you, try Kantu. It works with screenshots and uses OCR for scraping. The beauty is that it works with any kind of site. But clearly, the speed can not match a node.js or perl based scraper (mechanize etc), so it is not suitable for high volumes.

Do you find it better than Phantom?

Just reading about Kantu now. It reminds me of http://www.sikuli.org/

Re: Show HN: WrapAPI v2 – Build APIs, scrapers, bots on any website

#28

Post by wrapapi »

zackify wrote: post #15

Doesn't work at all with JS. This is a big thing on many sites now. Also, since that is the case, you could build this in a few hours using something like: https://github.com/bda-research/node-crawler . Yes, it would have no gui, so you lose that.

For sites that load data using AJAX, we recommend you take a look at our Chrome extension (https://wrapapi.com/#/chromePlugin). Our philosophy isn't to run a full headless browser (similar to Phantom), but rather make it really easy to find the AJAX requests that actually load the data you need.
Post reply on HN