Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS Lambda and .zip is a recipe for serverless success

medium.com

Post reply on HN
Print view View original

11–20 of 117 posts

Re: AWS Lambda and .zip is a recipe for serverless success

#11

Post by skybrian »

idbehold wrote: post #5

You know what would be better than a .zip? A .tar.gz file. Zip files cannot be unzipped without having the whole thing in memory. What's funny is that the Lambda environments (at least for Node) do not have the `zip` executable available which means that to zip/unzip something inside Lambda I need one written in javascript which has to be included in the zip file I upload to AWS instead of being able to use the nativ…

You have it backwards. Zip files have a table of contents letting you unzip individual files without decompressing the whole thing, and you don't need to load it all into memory.

Re: AWS Lambda and .zip is a recipe for serverless success

#12

Post by FLUX-YOU »

idbehold wrote: post #5

You know what would be better than a .zip? A .tar.gz file. Zip files cannot be unzipped without having the whole thing in memory. What's funny is that the Lambda environments (at least for Node) do not have the `zip` executable available which means that to zip/unzip something inside Lambda I need one written in javascript which has to be included in the zip file I upload to AWS instead of being able to use the nativ…

.tar.gz sucks on Windows though (even 7-zip is a two step process).

Re: AWS Lambda and .zip is a recipe for serverless success

#13

Post by devonkim »

_xnmw wrote: post #4
mirceal wrote: post #2

Serverless has its upsides but it also has downsides. In an ideal, pure, world it’s awesome. The problem is that your lambda function usually needs to use some sort of external resources. (It is pretty useless if you don’t interact with anything) Now you have to worry about and understand how to do access control on the said resources, how to collect the logs, how to collect metrics relevant to your code, how do you…

Right, and one of biggest downsides to serverless: reusing database connections/pools between requests. Right now the only "standard" solution offered by serverless's biggest advocates is pretty much a hack: rely on AWS's preservation of memory across some warm invocations. Basically keep it in a global variable and if you get lucky it'll still be there the next time your functions runs. [1] http://blog.rowanudell.co…

I think a mixture of persistent proxies could work well with lambda functions for this use case. You can run lambdas in your VPCs with assigned security groups and IAM roles just like an EC2 instance and therefore lambda functions should be able to proxy through an intermediate server that performs the connection pooling on behalf of calling functions. Not sure what the standard solution for something like this would be though because every time I've seen connection pooling it's for use locally. Perhaps some nginx or haproxy hacking could make it work.

Re: AWS Lambda and .zip is a recipe for serverless success

#14

Post by matt_wulfeck »

devonkim wrote: post #13
_xnmw wrote: post #4

Earlier quoted context omitted.

Right, and one of biggest downsides to serverless: reusing database connections/pools between requests. Right now the only "standard" solution offered by serverless's biggest advocates is pretty much a hack: rely on AWS's preservation of memory across some warm invocations. Basically keep it in a global variable and if you get lucky it'll still be there the next time your functions runs. [1] http://blog.rowanudell.co…

I think a mixture of persistent proxies could work well with lambda functions for this use case. You can run lambdas in your VPCs with assigned security groups and IAM roles just like an EC2 instance and therefore lambda functions should be able to proxy through an intermediate server that performs the connection pooling on behalf of calling functions. Not sure what the standard solution for something like this would…

In this scenario you’re still paying the cold-connection cost to your proxy, and that’s just keeping a pool of tcp connections open, not necessarily doing anything smart/optimized in the sql driver.

Re: AWS Lambda and .zip is a recipe for serverless success

#15

Post by fulafel »

skybrian wrote: post #11
idbehold wrote: post #5

You know what would be better than a .zip? A .tar.gz file. Zip files cannot be unzipped without having the whole thing in memory. What's funny is that the Lambda environments (at least for Node) do not have the `zip` executable available which means that to zip/unzip something inside Lambda I need one written in javascript which has to be included in the zip file I upload to AWS instead of being able to use the nativ…

You have it backwards. Zip files have a table of contents letting you unzip individual files without decompressing the whole thing, and you don't need to load it all into memory.

But this does not work for streaming since the directory is at the end. So you have to temporarily store the whole zip file.

Re: AWS Lambda and .zip is a recipe for serverless success

#16

Post by sheetjs »

skybrian wrote: post #11
idbehold wrote: post #5

You know what would be better than a .zip? A .tar.gz file. Zip files cannot be unzipped without having the whole thing in memory. What's funny is that the Lambda environments (at least for Node) do not have the `zip` executable available which means that to zip/unzip something inside Lambda I need one written in javascript which has to be included in the zip file I upload to AWS instead of being able to use the nativ…

You have it backwards. Zip files have a table of contents letting you unzip individual files without decompressing the whole thing, and you don't need to load it all into memory.

The table of contents in a zip file is at the end, not the beginning, so you need random access to the full file if you want to properly jump to the right location. Some zip files use the length feature of local file headers so you can jump ahead in the file without processing the data, but many skip the length so you have to work through the DEFLATE algorithm to find the end of the first file (to find where the second file starts)

Re: AWS Lambda and .zip is a recipe for serverless success

#17

Post by dev1n »

fapjacks wrote: post #9

Cloud commoditized datacenters. People then tightly coupled their applications to cloud providers like AWS. After some time, containers came along and commoditized the cloud providers. Those applications no longer needed to be tightly coupled to a specific cloud provider. This reduced both costs and the risk of being invested in a single cloud provider. Serverless (AWS Lambda) is just the cloud's way of trying to "de…

API Gateway is the literal manifestation of the worst product I have ever had the misfortune of configuring. This is mostly due to AWS's incompetence at delivering any semblance of working documentation around the product. Looking forward to their new "open source" documentation.

Re: AWS Lambda and .zip is a recipe for serverless success

#19

Post by skybrian »

fulafel wrote: post #15
skybrian wrote: post #11

Earlier quoted context omitted.

You have it backwards. Zip files have a table of contents letting you unzip individual files without decompressing the whole thing, and you don't need to load it all into memory.

But this does not work for streaming since the directory is at the end. So you have to temporarily store the whole zip file.

Yes, you need random-access file I/O.

But if the zip file isn't stored locally, sending the entire thing over the network to access one subfile is very inefficient. Instead you'd want a network protocol where you request the files you actually want and the remote server just sends those.

Re: AWS Lambda and .zip is a recipe for serverless success

#20

Post by marcc »

alexcroox wrote: post #3

I actually find zipping and uploading manually a PITA. Started using the Serverless framework [1] recently and now I just run `serverless deploy --stage dev --aws-profile profilename` from my repo (which is an npm script). [1] https://serverless.com/

Uploading manually is a PITA. That’s because you really should automate it using CI/CD if it’s not your local dev environment. If you are manually uploading your code for a dev environment it would be worth exploring a better local, offline framework to avoid this. Don’t upload manually.
Post reply on HN