Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: WrapAPI v2 – Build APIs, scrapers, bots on any website

wrapapi.com

Post reply on HN
Print view View original

31–40 of 62 posts

Re: Show HN: WrapAPI v2 – Build APIs, scrapers, bots on any website

#31

Post by RandomBookmarks »

gardnr wrote: post #24
RandomBookmarks wrote: post #22

Earlier quoted context omitted.

If JS a problem for you, try Kantu. It works with screenshots and uses OCR for scraping. The beauty is that it works with any kind of site. But clearly, the speed can not match a node.js or perl based scraper (mechanize etc), so it is not suitable for high volumes.

Do you find it better than Phantom? Just reading about Kantu now. It reminds me of http://www.sikuli.org/

Yeah, the concept is the same as Sikuli, but all inside Chromium (and the OCR is better).

>Do you find it better than Phantom?

It depends. Once you have a working script, web scraping with Phantom is much faster and much more resource efficient. But since Kantu works visually, you do not have to touch any page source code. That makes it much easier/faster to create the automation in the first place, especially for complex sites with date controls, drag & drop and other Javascript.

Re: Show HN: WrapAPI v2 – Build APIs, scrapers, bots on any website

#32

Post by wrapapi »

notwhoyouthink wrote: post #23

Glitchy on Chrome 57. When I load a page to build, I'm unable to scroll more than 1/4 of the page then it jumps back to the top.

Let us know what page you're trying at peter@wrapapi.com, and I'll take a look and fix it ASAP.

Re: Show HN: WrapAPI v2 – Build APIs, scrapers, bots on any website

#33

Post by wrapapi »

danvoell wrote: post #10

How does this compare to Kimono?

We're quite inspired by Kimono, and aim to be just as easy to use while handling use cases beyond scraping (e.g., fully automated form-filling, POST requests, etc.). One of the big feature requests we've been getting has been RSS feeds though, so we're definitely trying to get to full feature parity!

Re: Show HN: WrapAPI v2 – Build APIs, scrapers, bots on any website

#34

Post by wrapapi »

krmmalik wrote: post #25

Im asking since the term API is mentioned. Is this designed for technical or non technical people? Im a non coder but really could do with the scraper so would this work for me?

This is designed for at least semi-technical people, but it's really not that hard to give it a try for simple sites. Try watching the video, and shoot me an email at peter@wrapapi.com if you run into any issues!

Re: Show HN: WrapAPI v2 – Build APIs, scrapers, bots on any website

#35

Post by webninja »

This tool is really well thought-out and useful! I made a working API in less than 1 hr. This tool has a much better design & implementation than Kimono and easier than using Python 3 + Beautiful Soup 4 which is how I made my previous web scrapers. This tool also works for POSTing to web forms.

Re: Show HN: WrapAPI v2 – Build APIs, scrapers, bots on any website

#37

Post by webninja »

salimmadjd wrote: post #3

What if the content you want to scrape->API is behind a login gate? Is there an option of authentication?

Yes, you can get the content you want even if it's behind a login page! Expect to create 2 APIs, one for logging in and another for getting the content. An example is provided on their homepage: https://wrapapi.com/v2#/caseStudies/cj

Re: Show HN: WrapAPI v2 – Build APIs, scrapers, bots on any website

#38

Post by cookiecaper »

iambpentameter wrote: post #20

Does this violate any current US law?

The software itself probably wouldn't, but the use of it for anything anyone cares about probably would. The CFAA, etc., make unwanted scraping illegal and this has been tested repeatedly in court.

The company that runs this software as a service needs to be very careful. 3Taps was similar and got destroyed for relaying data scraped from Craigslist.

Contacting the server after its operator has expressed its wish for you to stop is a violation of the CFAA (in that you are "exceeding authorized access" and/or gaining "unauthorized access" to a protected computer system). If it's found that the site's ToS is binding upon you, which it typically would be, you don't really even need separate notice to be held liable.

Storing a copy of a web page in RAM creates a copy that is eligible for copyright protection, and it is likely that any implied license to read that page will be invalidated by the access revocation.

IANAL.

Re: Show HN: WrapAPI v2 – Build APIs, scrapers, bots on any website

#39

Post by wrapapi »

webninja wrote: post #35

This tool is really well thought-out and useful! I made a working API in less than 1 hr. This tool has a much better design & implementation than Kimono and easier than using Python 3 + Beautiful Soup 4 which is how I made my previous web scrapers. This tool also works for POSTing to web forms.

Thanks webninja! The POSTing part is one of the biggest things we were trying to get right while not making it any harder to use than Kimono. Is there anything that was confusing when trying to learn it for the first time? If so, we're still trying to make it easier =)
Post reply on HN