Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Faast.js – Serverless Batch Computing Made Simple

faastjs.org

Post reply on HN
Print view View original

11–20 of 33 posts

Re: Show HN: Faast.js – Serverless Batch Computing Made Simple

#11

Post by BrandiATMuhkuh »

Love what you did!

We resently were exactly in a situation where we had to do heavy processing of ~4000 items each running between 1-10minutes. To speed the process up we ran it on lambda. That means our process went down from 10h++ on a single core computer to about 15min running it on 4000 lambdas.

Your library would have saved us quite some work as it would take away a lot of Aws config, deploy, etc....

Btw: I'm thinking of building a similar library for multi core/webworkers for node.js. currently a lot of boilerplate is required on node.js to make a loop run parallel on all cores.

Re: Show HN: Faast.js – Serverless Batch Computing Made Simple

#13

Post by asadlionpk »

This can be great for scrapping jobs!

There are IP-based rate limiters on sites (linkedIn, facebook, etc), but each lambda has a new public IP so by using faast.js, I can stay under the radar.

Plus you can essentially spawn a headless chrome (puppeteer) to do advanced stuff.

Re: Show HN: Faast.js – Serverless Batch Computing Made Simple

#14

Post by achou »

BrandiATMuhkuh wrote: post #11

Love what you did! We resently were exactly in a situation where we had to do heavy processing of ~4000 items each running between 1-10minutes. To speed the process up we ran it on lambda. That means our process went down from 10h++ on a single core computer to about 15min running it on 4000 lambdas. Your library would have saved us quite some work as it would take away a lot of Aws config, deploy, etc.... Btw: I'm t…

Very cool. What kind of data was it, if you don't mind sharing?

Faast.js can be used with multi-core, just use the "local" mode and run it on a large box. I'm billing this as a way to test locally before running in the cloud, but it's actually a completely viable way to run parallel processes on one machine, with the option to run on serverless with a one line change.

Re: Show HN: Faast.js – Serverless Batch Computing Made Simple

#16

Post by achou »

gregmac wrote: post #8

From what I can tell, it's the invocation model and deployment that is unique here? You invoke faast from your local machine (or build server, or cron job, whatever), and in turn it deploys some functions to a serverless platform and runs them, then tears them all down when complete. Eg, from the site, this code runs locally: import { faast } from "faastjs"; import * as funcs from "./functions"; (async () => { const…

You're basically correct, and thanks for the suggestion to add documentation about deployment in production.

One special case is if your functions return a lot of data; outbound data charges can get expensive fast, and you'll be limited in getting responses by your network link. So you can run the coordinator code on, say, EC2 in the same region and then the link to Lambda is super fast and you won't have any outbound data costs.

Re: Show HN: Faast.js – Serverless Batch Computing Made Simple

#17

Post by linuxdude314 »

achou wrote: post #15
mring33621 wrote: post #12

This is neat, but would be more useful if it could deploy cloud functions made in language {x} and provide local js proxies for them.

Good idea. Any specific example you have in mind?

Python is a good place to start.

Re: Show HN: Faast.js – Serverless Batch Computing Made Simple

#18

Post by sourc3 »

This is very neat! Last year I had to essentially do this on GCP and relied on a very similar implementation. Everyone was surprised to see JS being used for data processing but it worked wonderfully.

One thing I want to ask is the retries, how do you handle that currently? I ran into multiple cases where functions would fail for transient reasons.

Re: Show HN: Faast.js – Serverless Batch Computing Made Simple

#19

Post by achou »

asadlionpk wrote: post #13

This can be great for scrapping jobs! There are IP-based rate limiters on sites (linkedIn, facebook, etc), but each lambda has a new public IP so by using faast.js, I can stay under the radar. Plus you can essentially spawn a headless chrome (puppeteer) to do advanced stuff.

Indeed, I've put together a simple example of using puppeteer with faast.js in this repo: https://github.com/faastjs/examples/tree/master/aws-puppetee...

Re: Show HN: Faast.js – Serverless Batch Computing Made Simple

#20

Post by achou »

sourc3 wrote: post #18

This is very neat! Last year I had to essentially do this on GCP and relied on a very similar implementation. Everyone was surprised to see JS being used for data processing but it worked wonderfully. One thing I want to ask is the retries, how do you handle that currently? I ran into multiple cases where functions would fail for transient reasons.

Functions need to be idempotent, so you have to assume they will be retried. Faast.js will proactively do retries in some cases where it thinks a function is slow, to reduce tail latency.

If a function fails to execute for transient reasons and exceeds the retry maximum (a config setting you can change), then it will reject the return value promise. You can catch that and handle with another attempt, or report an error, or just ignore it and report less accurate or complete results.

Post reply on HN