Ask HN: Is it a good idea to use AWS Lambda for ETL Pipelines?

#1

Post by 91dylwyn » Mon, Jan 17, 2022, 6:13 PM UTC

We are working on an OSS for ETL (Typhoon data orchestrator) and we are thinking about using AWS Lambda as the backend. The benefit is that it has a built in scheduler and we don't need to build our own. We can also spawn multiple Lambdas to scale concurrency.

Do you think this would be a sensible option? Would the 15 minutes running time be a deal breaker?

Re: Ask HN: Is it a good idea to use AWS Lambda for ETL Pipelines?

#2

Post by ignorantguy » Mon, Jan 17, 2022, 6:25 PM UTC

We use lambda in our production environment for ETL. It works really well. In our case when we were using it, we modeled the code to account for 15 minute window. Hope that helps.

Re: Ask HN: Is it a good idea to use AWS Lambda for ETL Pipelines?

#3

Post by 91dylwyn » Mon, Jan 17, 2022, 6:34 PM UTC

ignorantguy wrote: post #2

We use lambda in our production environment for ETL. It works really well. In our case when we were using it, we modeled the code to account for 15 minute window. Hope that helps.

Are you finding any pain points? How did you manage the 15 minute window in case of large amounts of data running over time?

How do you manage the DAGs on it? For example creating dependencies between tasks and monitoring and rerunning tasks?

Re: Ask HN: Is it a good idea to use AWS Lambda for ETL Pipelines?

#4

Post by vlahmot » Thu, Jan 20, 2022, 7:11 AM UTC

You can always use step functions or some janky recursive calls to work around the 15 min timeout.

I would recommend using docker containers and writing it in a way that you can swap out lambda/ecs/k8s/etc as the backend infrastructure. That way you you can punt on doing any fancy work to try and break the jobs up across lambdas under that 15 min constraint.