Show HN: Run Puppeteer on AWS Lambda
github.com
Show HN: Run Puppeteer on AWS Lambda
1–10 of 25 posts
Re: Show HN: Run Puppeteer on AWS Lambda
#2This is fantastic.
- I'm just getting started with Lambda so pardon if this is ignorant, but what's the cold start time of Chromium? Or can you warm start it somehow?
- Since scraping often depends on state, wouldn't you hit a timeout doing longer scraping joba?
Re: Show HN: Run Puppeteer on AWS Lambda
#3This is fantastic. - I'm just getting started with Lambda so pardon if this is ignorant, but what's the cold start time of Chromium? Or can you warm start it somehow? - Since scraping often depends on state, wouldn't you hit a timeout doing longer scraping joba?
Thanks!
So usually with Lambda, you want your jobs to be as atomic/quick as possible, as Lambda is stateless and has a maximum duration of 15 minutes.
As for the warm up times, the decompression of Chromium with Brotli takes about 700ms on a 1.5GB Lambda (this is faster than Gzip/Zip). Launching Chromium itself and opening a new tab takes another 400ms or so. If you keep your Lambdas warm (by registering a scheduled ClowdWatch event every 15 minutes for instance) your startup time will effective be those 400ms.
Re: Show HN: Run Puppeteer on AWS Lambda
#4What is the advantage of running automated tests as lambda ?
Typically when automation tests are run, they are long running processes and lambda execution may not be suitable.
The cold start times of lambda is another challenge.
What is a good practice/model for running suites ? One test per lambda or, one spec per lambda ? Still inclined to have ec2 instances created and destroyed via devops tools like terraform to run automation.
Thoughts please.
Re: Show HN: Run Puppeteer on AWS Lambda
#5What is the advantage of running automated tests as lambda ? Typically when automation tests are run, they are long running processes and lambda execution may not be suitable. The cold start times of lambda is another challenge. What is a good practice/model for running suites ? One test per lambda or, one spec per lambda ? Still inclined to have ec2 instances created and destroyed via devops tools like terraform to…
Well, puppeteer can be used for more than test suites (think screenshots, PDF rendering, proxified APIs, ...).
But for running long automated tests, I'd probably look into alternatives like Fargate, where the billing model is per-second with a one minute minimum. Terraform + EC2 spot instances works too, obviously. :)
Re: Show HN: Run Puppeteer on AWS Lambda
#6What is the advantage of running automated tests as lambda ? Typically when automation tests are run, they are long running processes and lambda execution may not be suitable. The cold start times of lambda is another challenge. What is a good practice/model for running suites ? One test per lambda or, one spec per lambda ? Still inclined to have ec2 instances created and destroyed via devops tools like terraform to…
Well, puppeteer can be used for more than test suites (think screenshots, PDF rendering, proxified APIs, ...). But for running long automated tests, I'd probably look into alternatives like Fargate, where the billing model is per-second with a one minute minimum. Terraform + EC2 spot instances works too, obviously. :)
Thanks. The example you’ve mentioned will be useful.
Re: Show HN: Run Puppeteer on AWS Lambda
#7What is the advantage of running automated tests as lambda ? Typically when automation tests are run, they are long running processes and lambda execution may not be suitable. The cold start times of lambda is another challenge. What is a good practice/model for running suites ? One test per lambda or, one spec per lambda ? Still inclined to have ec2 instances created and destroyed via devops tools like terraform to…
Lambda have a maximum running time of 15min. Even if you have « Cold Start » it won’t more than a minute for Chrome to be up and running.
Meaning you’ll get at least 10 to 14 minutes of Headless testing.
As for recommandations on how to do so , unless your testing is super long you should just run the entire thing in one function . Otherwise decouple your testing based on the various modules of your app ( i.e one module per function )
Re: Show HN: Run Puppeteer on AWS Lambda
#8What is the advantage of running automated tests as lambda ? Typically when automation tests are run, they are long running processes and lambda execution may not be suitable. The cold start times of lambda is another challenge. What is a good practice/model for running suites ? One test per lambda or, one spec per lambda ? Still inclined to have ec2 instances created and destroyed via devops tools like terraform to…
Lambda have a maximum running time of 15min. Even if you have « Cold Start » it won’t more than a minute for Chrome to be up and running. Meaning you’ll get at least 10 to 14 minutes of Headless testing. As for recommandations on how to do so , unless your testing is super long you should just run the entire thing in one function . Otherwise decouple your testing based on the various modules of your app ( i.e one mod…
Understand the 15 min and <1min coldstart. The question was more towards test suites with 100’s of tests for really large products and you cannot break them. The scenarios won’t make sense. The feedback from these tests will not be done in 15 mins.
Re: Show HN: Run Puppeteer on AWS Lambda
#9This is fantastic. - I'm just getting started with Lambda so pardon if this is ignorant, but what's the cold start time of Chromium? Or can you warm start it somehow? - Since scraping often depends on state, wouldn't you hit a timeout doing longer scraping joba?
Thanks! So usually with Lambda, you want your jobs to be as atomic/quick as possible, as Lambda is stateless and has a maximum duration of 15 minutes. As for the warm up times, the decompression of Chromium with Brotli takes about 700ms on a 1.5GB Lambda (this is faster than Gzip/Zip). Launching Chromium itself and opening a new tab takes another 400ms or so. If you keep your Lambdas warm (by registering a scheduled…
Presumably you can also keep Chromium running and keep a pool of tabs to reuse. Doing this it would pretty fast I imagine.
Depending on your use case you can also disable security and open as many iframes as you want in a single tab. Not sure how this compares to multiple tabs though.
Of course you'll run into cold start again when lambda has to scale.
Re: Show HN: Run Puppeteer on AWS Lambda
#10Earlier quoted context omitted.
Thanks! So usually with Lambda, you want your jobs to be as atomic/quick as possible, as Lambda is stateless and has a maximum duration of 15 minutes. As for the warm up times, the decompression of Chromium with Brotli takes about 700ms on a 1.5GB Lambda (this is faster than Gzip/Zip). Launching Chromium itself and opening a new tab takes another 400ms or so. If you keep your Lambdas warm (by registering a scheduled…
Presumably you can also keep Chromium running and keep a pool of tabs to reuse. Doing this it would pretty fast I imagine. Depending on your use case you can also disable security and open as many iframes as you want in a single tab. Not sure how this compares to multiple tabs though. Of course you'll run into cold start again when lambda has to scale.
Yes, this is a good solution but only if you don't have any sort of session data that you want flushed out after it runs. One could argue you could use browser contexts (incognito tabs) to have ephemeral sessions, but unfortunately that feature doesn't work in --single-process mode (which AWS Lambda requires).