An open source API for web scraping

_jomo wrote: post #4

Title should probably contain 'Show HN:' ? Very interesting though. Just tried scraping twitter and it works great: https://falkor-api.herokuapp.com/api/query?url=https://twitt... Edit: works great as long as there are no quotes, hashtags, or links in the tweets. Is it possible to include sub-elements? So basically this is a DOM API in JSON. Simple, but I like it. Any plans to add JSONP support?

Hey. Thanks. Yeah I will add a ton of features over the next few days. JSONP should be an easy one. Feel free to add an issue in Github and I'll get it done for you.

Only really started hacking around on the idea the other day so early stages. Want to add filters so you can say "grab me only the text" or "grab me just the class names". Obviously another step would be to grab multiple elements in one request.

Re: An open source API for web scraping

#6

Post by Jake232 » Wed, Jun 24, 2015, 7:55 PM UTC

Cool idea. This could easily be extended to support something like a proxy pool; that way you can rate limit / rotate proxies for X domain globally at this server level. That way it's across all your projects, rather than having to do it on a per project basis.

Adding xPath support as well as CSS selectors would be a good addition.

Re: An open source API for web scraping

#7

Post by owainlewis » Wed, Jun 24, 2015, 7:59 PM UTC

Jake232 wrote: post #6

Cool idea. This could easily be extended to support something like a proxy pool; that way you can rate limit / rotate proxies for X domain globally at this server level. That way it's across all your projects, rather than having to do it on a per project basis. Adding xPath support as well as CSS selectors would be a good addition.

Will definitely do something with caching and rate limiting when I get some time. These queries are quite expensive so definitely needs a bit of work in those areas.

Re: An open source API for web scraping

#8

Post by curiously » Wed, Jun 24, 2015, 9:58 PM UTC

Pretty interesting. Wrote a web scraping api you can paste in to your browser and download results last year but took it down to work on another project. You can take look at what a url could look like.

https://web.archive.org/web/20140420162639/http://scrape.ly/

For example if you wanted the profile of authors of today's stories

    http://scrape.ly/s/{http://news.combination.com}
    {'ueoma87'}*{'next':'Next Page'}{'karma':'331', 
    'username':'ueoma87'}

Would've returned all the profiles of each story's author today and yesterday and so on.

Re: An open source API for web scraping

#9

Post by getriver » Thu, Jun 25, 2015, 6:10 AM UTC

A better error message would be helpful. For example I tried to do: https://falkor-api.herokuapp.com/api/query?url=https://kodin..., all I got was "Request failed"

Re: An open source API for web scraping

#10

Post by owainlewis » Thu, Jun 25, 2015, 7:49 AM UTC

curiously wrote: post #8

Pretty interesting. Wrote a web scraping api you can paste in to your browser and download results last year but took it down to work on another project. You can take look at what a url could look like. https://web.archive.org/web/20140420162639/http://scrape.ly/ For example if you wanted the profile of authors of today's stories http://scrape.ly/s/{http://news.combination.com} {'ueoma87'}*{'next':'Next Page'}{'karma…

Thanks. This looks really interesting. I may well borrow some ideas ; )