Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Run Puppeteer on AWS Lambda

github.com

Post reply on HN
Print view View original

11–20 of 25 posts

Re: Show HN: Run Puppeteer on AWS Lambda

#11

Post by jpambrun »

alixaxel wrote: post #3
nailer wrote: post #2

This is fantastic. - I'm just getting started with Lambda so pardon if this is ignorant, but what's the cold start time of Chromium? Or can you warm start it somehow? - Since scraping often depends on state, wouldn't you hit a timeout doing longer scraping joba?

Thanks! So usually with Lambda, you want your jobs to be as atomic/quick as possible, as Lambda is stateless and has a maximum duration of 15 minutes. As for the warm up times, the decompression of Chromium with Brotli takes about 700ms on a 1.5GB Lambda (this is faster than Gzip/Zip). Launching Chromium itself and opening a new tab takes another 400ms or so. If you keep your Lambdas warm (by registering a scheduled…

If you keep your Lambda warm, shouldn't you just use something like browserless (https://github.com/joelgriffith/browserless)?

Re: Show HN: Run Puppeteer on AWS Lambda

#12

Post by pouta »

alixaxel wrote: post #10
Touche wrote: post #9

Earlier quoted context omitted.

Presumably you can also keep Chromium running and keep a pool of tabs to reuse. Doing this it would pretty fast I imagine. Depending on your use case you can also disable security and open as many iframes as you want in a single tab. Not sure how this compares to multiple tabs though. Of course you'll run into cold start again when lambda has to scale.

Yes, this is a good solution but only if you don't have any sort of session data that you want flushed out after it runs. One could argue you could use browser contexts (incognito tabs) to have ephemeral sessions, but unfortunately that feature doesn't work in --single-process mode (which AWS Lambda requires).

I'm using incognito mode to parse some pages that for some reason I can't using the normal context.

I have been considering moving my pool of chromium workers to lambda functions so we can avoid api slowdowns due to a high number of parsings at the same time.

Are there any other side effects of running chromium headless in a lambda function?

Re: Show HN: Run Puppeteer on AWS Lambda

#13

Post by defied »

sriram_iyengar wrote: post #4

What is the advantage of running automated tests as lambda ? Typically when automation tests are run, they are long running processes and lambda execution may not be suitable. The cold start times of lambda is another challenge. What is a good practice/model for running suites ? One test per lambda or, one spec per lambda ? Still inclined to have ec2 instances created and destroyed via devops tools like terraform to…

There are quite some disadvantages to use AWS Lambda to do Automated Testing (test time is capped to 15min, cold start waiting time, ...).

The advantage is that you can run a lot of tests concurrently at a relatively cheap cost.

The company [1] I work for offers VMs that are created/destroyed automatically after each test. There's no cold start, and no time limit. Plus you can choose to run headless like Puppeteer or test in an actual OS like Win/Mac.

[1] https://testingbot.com

Re: Show HN: Run Puppeteer on AWS Lambda

#14

Post by hayd »

sriram_iyengar wrote: post #4

What is the advantage of running automated tests as lambda ? Typically when automation tests are run, they are long running processes and lambda execution may not be suitable. The cold start times of lambda is another challenge. What is a good practice/model for running suites ? One test per lambda or, one spec per lambda ? Still inclined to have ec2 instances created and destroyed via devops tools like terraform to…

parallelization

Re: Show HN: Run Puppeteer on AWS Lambda

#15

Post by e1g »

jpambrun wrote: post #11
alixaxel wrote: post #3

Earlier quoted context omitted.

Thanks! So usually with Lambda, you want your jobs to be as atomic/quick as possible, as Lambda is stateless and has a maximum duration of 15 minutes. As for the warm up times, the decompression of Chromium with Brotli takes about 700ms on a 1.5GB Lambda (this is faster than Gzip/Zip). Launching Chromium itself and opening a new tab takes another 400ms or so. If you keep your Lambdas warm (by registering a scheduled…

If you keep your Lambda warm, shouldn't you just use something like browserless ( https://github.com/joelgriffith/browserless )?

I run browserless Docker container on-prem and it works very well for us. Fire&forget, +1.

Re: Show HN: Run Puppeteer on AWS Lambda

#16

Post by whoisjuan »

sriram_iyengar wrote: post #4

What is the advantage of running automated tests as lambda ? Typically when automation tests are run, they are long running processes and lambda execution may not be suitable. The cold start times of lambda is another challenge. What is a good practice/model for running suites ? One test per lambda or, one spec per lambda ? Still inclined to have ec2 instances created and destroyed via devops tools like terraform to…

I think is more cost efficient and scalable than having an EC2 instance being created and destroyed every time you run a test.

Re: Show HN: Run Puppeteer on AWS Lambda

#17

Post by tnolet »

I run a ton of Puppeteer jobs (300k in the last month), currently on EC2 and Digital Ocean VM's, mostly due to the subtle difficulties of running Puppeteer on Lambda.

Will certainly have a look at this project and contribute where possible.

My main concerns are not so much cold start time, as for my use case this is not really a huge issue, but mainly the performance of Chrome on AWS Lambda boxes. The rendering, navigation etc. needs to be snappy.

Re: Show HN: Run Puppeteer on AWS Lambda

#18

Post by tnolet »

sriram_iyengar wrote: post #4

What is the advantage of running automated tests as lambda ? Typically when automation tests are run, they are long running processes and lambda execution may not be suitable. The cold start times of lambda is another challenge. What is a good practice/model for running suites ? One test per lambda or, one spec per lambda ? Still inclined to have ec2 instances created and destroyed via devops tools like terraform to…

My SaaS uses Puppeteer for website monitoring. Shameless plug: https://checklyhq.com/product/transaction-monitoring/

It's been quite difficult to get it right, with Puppeteer being young etc. but it chugs along nicely now at around 10k-15k runs per day spread over 4 regions.

Re: Show HN: Run Puppeteer on AWS Lambda

#19

Post by thesandlord »

tnolet wrote: post #17

I run a ton of Puppeteer jobs (300k in the last month), currently on EC2 and Digital Ocean VM's, mostly due to the subtle difficulties of running Puppeteer on Lambda. Will certainly have a look at this project and contribute where possible. My main concerns are not so much cold start time, as for my use case this is not really a huge issue, but mainly the performance of Chrome on AWS Lambda boxes. The rendering, navi…

Google App Engine and Google Cloud Functions got native support for Puppeteer a few months ago as well. Let me know what you think if you try it out.

https://news.ycombinator.com/item?id=17795626

(I work for Google Cloud)

Re: Show HN: Run Puppeteer on AWS Lambda

#20

Post by alixaxel »

thesandlord wrote: post #19
tnolet wrote: post #17

I run a ton of Puppeteer jobs (300k in the last month), currently on EC2 and Digital Ocean VM's, mostly due to the subtle difficulties of running Puppeteer on Lambda. Will certainly have a look at this project and contribute where possible. My main concerns are not so much cold start time, as for my use case this is not really a huge issue, but mainly the performance of Chrome on AWS Lambda boxes. The rendering, navi…

Google App Engine and Google Cloud Functions got native support for Puppeteer a few months ago as well. Let me know what you think if you try it out. https://news.ycombinator.com/item?id=17795626 (I work for Google Cloud)

The performance of puppeteer is super bad on CGF (you can read more about it here https://github.com/GoogleChrome/puppeteer/issues/3120). It would actually be great to have someone really improve this situation instead of dismissing it as a weird IO problem.
Post reply on HN