Publish your Python function to PiCloud and invoke via REST

Publish your Python function to PiCloud and invoke via REST
blog.picloud.com

We're running on Heroku, where workers are expensive (at least, unless you scale them dynamically). We looked at SimpleWorker, which is basically PiCloud for Ruby. But latency for enqueue-ing a job was way too high, and not having access to database models and such made it inconvenient for that purpose. (Long story short, we ended up with Resque like everyone else.) Does PiCloud have the enqueue latency problem beat? (Like under 100ms RTT?)

But these services don't make sense to me for big data or scientific computing either. If you have tons of data, you're probably going to end up with something like MapReduce with some specialized HBase-like backend. And most "scientific computing" I've seen seems to be either MATLAB (if performance is no issue at all) or custom C++ code linking to specific linear algebra libraries (if performance is important). That said, I'm not an expert in either of these use cases, so maybe I'm wrong? Maybe this is for mobile or to fix some brokenness in GAE?

Put another way, what is the use case where you (a) don't care about high enqueue latency, (b) don't care about running within your own infrastructure and with your own code (e.g., for database access), (c) want tons of compute cycles in Python without having to manage the infrastructure, but (d) don't want specific control over the actual infrastructure and data persistence?

From the get-go, we've targeted users with serious compute-intensive needs. These users from both academia and industry generally do what we call "scientific computing." A couple examples are oil companies doing immense amounts of geophysics simulations, and bioinformatics laboratories in universities doing comparative genomics; we'll be posting a guest blog entry in the next week on successfully reducing the time of sequence alignment from 20-25 hours to less than an hour. Other examples include neuroscience, astronomy, seismology, weather analysis, and protein folding.

As time has gone on, we have noticed two sizable class of users who we consider to be non-scientific. The first are the risk analysis divisions of financial firms who run Monte Carlo simulations in their never-ending endeavors to determine their portfolio's risk profile. The second are web companies doing all sorts of background processing: video encoding, feed aggregation, social data scraping, and analytics of all sorts.

The REST feature is an experiment on many levels. In the short term, it helps our existing users utilize PiCloud without Python, which is very important in companies with large polyglot codebases (read: Java). In the medium term, we want to see whether there's adequate interest from web and mobile companies who require serious scalability, and thus are interested by how we entirely divorce their algorithms from their servers. In the long term, we want to see how far we can take the idea of "publishing" a function; there are many users out there who would love to share their algorithms in working form but lack the programming know how and computing hardware to do it.

To answer your other questions: We're releasing a BigData a la MapReduce solution for our users in the coming months. Also, Python has a strong scientific computing community primarily because it interoperates very well with existing C/C++/Fortran code via extensions. In most of our use cases, Python is being used as a wrapper for C code, which is doing the heavy lifting. PiCloud, unlike many of the other background processing services, allows users to deploy more than just Python or Ruby code. We used to allow this through a "package upload" system, but we've recently just replaced it (in beta) with the ability to fully customize a file system environment (apt-get away!).

Our users are able to access their databases, often times without any code modifications, from PiCloud, which means their Django models work just fine. If their database resides on EC2 us-east, then the performance hit is minimal.

We just did a quick non-scientific check of our enqueue time from an EC2 us-east server to PiCloud. A cold start (first job) takes 900ms (It could take longer depending on what needs to get transferred), but once the connection has been made, enqueue takes around 86ms, and as the system heats up it reduces to 50ms.