Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building an API in 60 seconds, without any server setup

api.blockspring.com

Post reply on HN
Print view View original

41–50 of 72 posts

Re: Building an API in 60 seconds, without any server setup

#41

Post by dheera »

I'd be more interested in seeing an infrastructure for scraping where the API functions are fixed, but the actual scraping functions are dynamically loaded so that the API user doesn't have to maintain it or re-pull/re-fork/re-compile when the website design changes.

Bonus points if one can make an ORM out of it, e.g.

    for article in get_api('reddit.com').todayilearned.filter('new').limit(100):
        ... do something ...
Where a call to get_api() dynamically fetches the latest scraping functions, in case reddit's page design has changed.

Triple bonus points if the system can be designed in a de-centralized fashion to defend against ToSes that try to disrciminate between human eyes and machine eyes.

Re: Building an API in 60 seconds, without any server setup

#42

Post by whackedspinach »

hackerews wrote: post #39
superuser2 wrote: post #29

I'm not sure I understand what this is. Free hosted PaaS for single files, essentially? I know nothing about where the code is running? Is it performant? Is it reliable? How are they covering server costs / at what point would I be asked to pay, and how much?

Hey superuser, Great questions. Let me answer them one at a time! Where is the code running? Every script run happens in its own sandbox on our machines. We use docker, it's awesome. People are asking for dedicated machines specifically for their company, and others are asking to hook into their own machines. Once we figure out how people/companies want to use the site, we'll put up some options for where code will b…

Paul,

How are you securing the Docker containers since Docker hasn't yet added user namespaces?

Re: Building an API in 60 seconds, without any server setup

#43

Post by donpinkus »

dheera wrote: post #41

I'd be more interested in seeing an infrastructure for scraping where the API functions are fixed, but the actual scraping functions are dynamically loaded so that the API user doesn't have to maintain it or re-pull/re-fork/re-compile when the website design changes. Bonus points if one can make an ORM out of it, e.g. for article in get_api('reddit.com').todayilearned.filter('new').limit(100): ... do something ... Wh…

Hey dheera that's super interesting. Since it's about web scraping specifically, I bet the guys over at Kimonolabs.com would be the experts on that!

Or if someone on here knows how to do this, and wants to throw up an API on Blockspring for it, we'd all love it :)

Re: Building an API in 60 seconds, without any server setup

#44

Post by jacke »

Check this out: https://github.com/hmarr/codecube 1. It's open source 2. Put this on heroku and use it privately. 3. Highly stable(because it's write using golang, you can run millions of request, and it's still work fine) ??? NONPROFIT

But project no longer in developing, so i guess maybe someone wants to reanimate codecube? On weekends for example, contact me: iamjacke AT gmail.com

Re: Building an API in 60 seconds, without any server setup

#45

Post by lgas »

hackerews wrote: post #36
innguest wrote: post #35

Earlier quoted context omitted.

Another vote for Haskell.

Language support is super easy to add. Any list of libraries you'd really want with Haskell?

If it's easy, Clojure with support for arbitrary libraries via leiningen would be great :)

Re: Building an API in 60 seconds, without any server setup

#48

Post by hackerews »

whackedspinach wrote: post #42
hackerews wrote: post #39

Earlier quoted context omitted.

Hey superuser, Great questions. Let me answer them one at a time! Where is the code running? Every script run happens in its own sandbox on our machines. We use docker, it's awesome. People are asking for dedicated machines specifically for their company, and others are asking to hook into their own machines. Once we figure out how people/companies want to use the site, we'll put up some options for where code will b…

Paul, How are you securing the Docker containers since Docker hasn't yet added user namespaces?

Containers open and close with every script run and then they're deleted. Never reopened.

Re: Building an API in 60 seconds, without any server setup

#50

Post by whackedspinach »

hackerews wrote: post #48
whackedspinach wrote: post #42

Earlier quoted context omitted.

Paul, How are you securing the Docker containers since Docker hasn't yet added user namespaces?

Containers open and close with every script run and then they're deleted. Never reopened.

Do you run the scripts as a specific user inside the container then? I was under the impression that running untrusted code in Docker as UID 0 was not yet safe.
Post reply on HN