Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Faast.js – Serverless Batch Computing Made Simple

faastjs.org

Post reply on HN
Print view View original

21–30 of 33 posts

Re: Show HN: Faast.js – Serverless Batch Computing Made Simple

#21

Post by penagwin »

achou wrote: post #16
gregmac wrote: post #8

From what I can tell, it's the invocation model and deployment that is unique here? You invoke faast from your local machine (or build server, or cron job, whatever), and in turn it deploys some functions to a serverless platform and runs them, then tears them all down when complete. Eg, from the site, this code runs locally: import { faast } from "faastjs"; import * as funcs from "./functions"; (async () => { const…

You're basically correct, and thanks for the suggestion to add documentation about deployment in production. One special case is if your functions return a lot of data; outbound data charges can get expensive fast, and you'll be limited in getting responses by your network link. So you can run the coordinator code on, say, EC2 in the same region and then the link to Lambda is super fast and you won't have any outboun…

This is how I interpreted it's usage too. We've all started an instance on DO/AWS/GCP/ETC for some batch job were we wanted 32 cores or whatnot. This lets you use lambda's for the scaling instead of the cores directly. How efficient this is performance wise I have no clue.

Re: Show HN: Faast.js – Serverless Batch Computing Made Simple

#22

Post by teej »

gregmac wrote: post #8

From what I can tell, it's the invocation model and deployment that is unique here? You invoke faast from your local machine (or build server, or cron job, whatever), and in turn it deploys some functions to a serverless platform and runs them, then tears them all down when complete. Eg, from the site, this code runs locally: import { faast } from "faastjs"; import * as funcs from "./functions"; (async () => { const…

To serve as a data point, I effectively built an in-house version of this a few years ago built on top of AWS Lambda, all in Python. The "entry point" code or orchestration code was hosted normally on an EC2 instance. More specifically we were using Airflow, so our Airflow server would kick off a Python program that would then orchestrate a couple thousand Lambdas.

Re: Show HN: Faast.js – Serverless Batch Computing Made Simple

#25

Post by mring33621 »

achou wrote: post #15
mring33621 wrote: post #12

This is neat, but would be more useful if it could deploy cloud functions made in language {x} and provide local js proxies for them.

Good idea. Any specific example you have in mind?

Honestly, I would want java. Probably would have to provide a mapping spec file (like IDL) to help generate the mediation code between the local proxies and the deployed functions.

Re: Show HN: Faast.js – Serverless Batch Computing Made Simple

#26

Post by BrandiATMuhkuh »

achou wrote: post #14
BrandiATMuhkuh wrote: post #11

Love what you did! We resently were exactly in a situation where we had to do heavy processing of ~4000 items each running between 1-10minutes. To speed the process up we ran it on lambda. That means our process went down from 10h++ on a single core computer to about 15min running it on 4000 lambdas. Your library would have saved us quite some work as it would take away a lot of Aws config, deploy, etc.... Btw: I'm t…

Very cool. What kind of data was it, if you don't mind sharing? Faast.js can be used with multi-core, just use the "local" mode and run it on a large box. I'm billing this as a way to test locally before running in the cloud, but it's actually a completely viable way to run parallel processes on one machine, with the option to run on serverless with a one line change.

Wow that's awesome. I'll have a look at it ASAP. We have actually just converted our lambda code to run on a multi core machine + much wiser algorithms to massively speed up the process.

I have not deeply look into your library yet. But how do you deal with de/serialising? We use https://www.npmjs.com/package/class-transformer to correctly de/serialise ts-objects.

Also, do you create a new webworker per function call or do you create only as many workers as threads/cores on the machine and run the functions inside those? Starting a webworker can be very expensive if the serialised data is large .

Ps: each lambda function ran a special parsing of complex mathematics-excercises. We are an ed-tech company ;)

Re: Show HN: Faast.js – Serverless Batch Computing Made Simple

#27

Post by achou »

BrandiATMuhkuh wrote: post #26
achou wrote: post #14

Earlier quoted context omitted.

Very cool. What kind of data was it, if you don't mind sharing? Faast.js can be used with multi-core, just use the "local" mode and run it on a large box. I'm billing this as a way to test locally before running in the cloud, but it's actually a completely viable way to run parallel processes on one machine, with the option to run on serverless with a one line change.

Wow that's awesome. I'll have a look at it ASAP. We have actually just converted our lambda code to run on a multi core machine + much wiser algorithms to massively speed up the process. I have not deeply look into your library yet. But how do you deal with de/serialising? We use https://www.npmjs.com/package/class-transformer to correctly de/serialise ts-objects. Also, do you create a new webworker per function call…

The serialization/deserialization is just JSON for now, though I plan on adding some configurability and perhaps changing the implementation at some point. There is some runtime checking to make sure the arguments are correctly serializable.

In local mode, a process is created up to the concurrency limit you specify, and each process is reused for subsequent calls (mimicking how Lambda reuses containers, allowing you to use the same caching behavior you'd use on Lambda). I'm not currently using webworkers, but that's something I could see a new mode for easily. For larger data, I would recommend storing arguments and return values directly in cloud storage like S3, or on local disk in local mode.

I would be interested to learn how your experiment with faast.js goes!

Re: Show HN: Faast.js – Serverless Batch Computing Made Simple

#28

Post by netofeverythin3 »

gregmac wrote: post #8

From what I can tell, it's the invocation model and deployment that is unique here? You invoke faast from your local machine (or build server, or cron job, whatever), and in turn it deploys some functions to a serverless platform and runs them, then tears them all down when complete. Eg, from the site, this code runs locally: import { faast } from "faastjs"; import * as funcs from "./functions"; (async () => { const…

Very cool. Worth checking out a similar project Durable Functions. However those orchestrations can run in serverless and can scale to zero during the “waiting for other tasks” step.

https://docs.microsoft.com/en-us/azure/azure-functions/durab...

Disclaimer - product manager for Azure durable functions

Re: Show HN: Faast.js – Serverless Batch Computing Made Simple

#29

Post by achou »

netofeverythin3 wrote: post #28
gregmac wrote: post #8

From what I can tell, it's the invocation model and deployment that is unique here? You invoke faast from your local machine (or build server, or cron job, whatever), and in turn it deploys some functions to a serverless platform and runs them, then tears them all down when complete. Eg, from the site, this code runs locally: import { faast } from "faastjs"; import * as funcs from "./functions"; (async () => { const…

Very cool. Worth checking out a similar project Durable Functions. However those orchestrations can run in serverless and can scale to zero during the “waiting for other tasks” step. https://docs.microsoft.com/en-us/azure/azure-functions/durab... Disclaimer - product manager for Azure durable functions

I'd love to add Azure support but I'm not super familiar with it. Would be great to chat about it sometime.
Post reply on HN