Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A shared file system for lambda functions

aws.amazon.com

Post reply on HN
Print view View original

101–110 of 113 posts

Re: A shared file system for lambda functions

#101

Post by twoodfin »

cldellow wrote: post #100
twoodfin wrote: post #97

Earlier quoted context omitted.

I’m genuinely curious about the factors that made it OK to have an unencrypted protocol into 2018. Is the AWS infrastructure already encrypting at another layer? Nobody worries about attacks on data that’s only transiting AWS?

I think Google's internal network was unencrypted in 2013 [1], allowing the NSA to tap it. After the Snowden leaks showed that they were being tapped, Google moved to encrypting data on their internal links. I wouldn't be surprised if Amazon made a similar decision. [1]: https://blog.encrypt.me/2013/11/05/ssl-added-and-removed-her...

Right, but this is in the context of AWS, where Amazon has got thousands of customers running their own code in its data center. I'd (naively?) assumed that any novel protocols to ship data back and forth between hosts in that environment would have been built on TLS from day 1.

Re: A shared file system for lambda functions

#102

Post by staticassertion »

twoodfin wrote: post #97
shoo wrote: post #95

I worked with EFS but not lambda in 2017-2018 when migrating an app to AWS - an app which included a bunch of random application code that assumed it could read or write into a network file system. Having EFS as a migration target to replace on prem CIFS was relatively pleasant, which removed the need to rewrite a bunch of the application code. S3 would have been a reasonable replacement but that would have required…

I’m genuinely curious about the factors that made it OK to have an unencrypted protocol into 2018. Is the AWS infrastructure already encrypting at another layer? Nobody worries about attacks on data that’s only transiting AWS?

AWS has stated that internal traffic, at least within a VPC, is "tamper proof". I don't recall any specific details about that, whether it is the case across regions, accounts, etc.

Re: A shared file system for lambda functions

#103

Post by thecampfire »

Konohamaru wrote: post #66

This is misdirection by naming, like how University of Phoenix is similar to Arizona State University, Phoenix. "Lambda functions" sounds like anonymous functions, but is actually referring to a proprietary interface to AWS Lambda™. They named it this way so that readers confuse AWS Lambda™ for programming lambdas.

And that's why Lambda functions is capitalized.

Re: A shared file system for lambda functions

#104

Post by jupp0r »

scarface74 wrote: post #16
jupp0r wrote: post #13

This is a horrible idea. This gives lambda functions shared mutable state to interfere with each other, with very brittle semantics compared to most databases (even terrible ones).

I need this specifically as a part of a state machine. Most of the steps involve a Lambda loading and unloading csv data between S3, Redshift, and Aurora where no local storage is needed. The last step where we had to download the files locally and compress multiple files together was done manually via a script because they were greater than 512Mb. We were just about to put the script in Fargate (Serverless Docker) a…

Why do you need a distributed file system to solve that problem? It sounds like all you need is some scratch space?

Re: A shared file system for lambda functions

#105

Post by jupp0r »

hansef wrote: post #19
scarface74 wrote: post #16

Earlier quoted context omitted.

I need this specifically as a part of a state machine. Most of the steps involve a Lambda loading and unloading csv data between S3, Redshift, and Aurora where no local storage is needed. The last step where we had to download the files locally and compress multiple files together was done manually via a script because they were greater than 512Mb. We were just about to put the script in Fargate (Serverless Docker) a…

This - if you have to fetch data from or output data to outside of the AWS ecosystem, the 512mb /tmp limit pushes you into the additional (relative) complexity of having to run on Fargate pretty quickly. Just had to deal with this for a content ingest job involving pulling a couple GB of data from an FTP server, processing it and pushing it into an RDS database on an hourly basis. Would have been super simple if the…

Where do you need a distributed file system in this use case? Sounds like all you need is some local scratch space?

Re: A shared file system for lambda functions

#106

Post by scarface74 »

jupp0r wrote: post #105
hansef wrote: post #19

Earlier quoted context omitted.

This - if you have to fetch data from or output data to outside of the AWS ecosystem, the 512mb /tmp limit pushes you into the additional (relative) complexity of having to run on Fargate pretty quickly. Just had to deal with this for a content ingest job involving pulling a couple GB of data from an FTP server, processing it and pushing it into an RDS database on an hourly basis. Would have been super simple if the…

Where do you need a distributed file system in this use case? Sounds like all you need is some local scratch space?

Lambda only gives you 512MB pf local “scratch space”. If it had to provision and de provision gigs of space at each invocation, it would probably cause longer start times and shut down times.

Re: A shared file system for lambda functions

#107

Post by philsnow »

inshadows wrote: post #59
philsnow wrote: post #25

sharp knife to hand people -- because EFS is just NFS, it uses NFS for security / isolation. Everything that can mount a given volume needs to agree on what unix users are what, and you need to make sure to completely lock down root access, otherwise you can't enforce any kind of data isolation. If your use case can deal with one EFS volume per isolation boundary, you can use IAM to control who can mount what volume,…

Wrong. NFSv4 with Kerberos maps UIDs to Kerberos principals. Kerberos further allows you to apply (authenticated) encryption to transfers.

I stand corrected. I kind of knew that since I've been struggling with identity and access control on a home network, and have been trying to muddle through so far without setting up kerberos. I know there are people who have no trouble setting this stuff up, but I'm not one of them.

Re: A shared file system for lambda functions

#108

Post by philsnow »

geertj wrote: post #81
philsnow wrote: post #25

sharp knife to hand people -- because EFS is just NFS, it uses NFS for security / isolation. Everything that can mount a given volume needs to agree on what unix users are what, and you need to make sure to completely lock down root access, otherwise you can't enforce any kind of data isolation. If your use case can deal with one EFS volume per isolation boundary, you can use IAM to control who can mount what volume,…

(PM-T on the EFS team) The EFS/Lambda integration uses EFS Access Points, which allow you to enforce a specific POSIX identity and directory for NFS operations. You can also use IAM policies to require that specific IAM roles/users use a specific access point.

Ah, didn't know that part, that's very good to learn. Thanks!

edit: ah, EFS access points are general and you can mount them from EC2 instances ? MUCH better.

Re: A shared file system for lambda functions

#109

Post by shoo »

staticassertion wrote: post #102
twoodfin wrote: post #97

Earlier quoted context omitted.

I’m genuinely curious about the factors that made it OK to have an unencrypted protocol into 2018. Is the AWS infrastructure already encrypting at another layer? Nobody worries about attacks on data that’s only transiting AWS?

AWS has stated that internal traffic, at least within a VPC, is "tamper proof". I don't recall any specific details about that, whether it is the case across regions, accounts, etc.

back when i was working on this (for a large org) they regarded anything in AWS as fundamentally lower trust than their own on-premises corporate networks, so there was a security requirement to do encryption in transit for all TCP connections -- this was for an application within the VPC with no public traffic

Re: A shared file system for lambda functions

#110

Post by staticassertion »

shoo wrote: post #109
staticassertion wrote: post #102

Earlier quoted context omitted.

AWS has stated that internal traffic, at least within a VPC, is "tamper proof". I don't recall any specific details about that, whether it is the case across regions, accounts, etc.

back when i was working on this (for a large org) they regarded anything in AWS as fundamentally lower trust than their own on-premises corporate networks, so there was a security requirement to do encryption in transit for all TCP connections -- this was for an application within the VPC with no public traffic

It's gonna depend on your threat model ultimately. Hard to imagine running on AWS if you don't trust them though.
Post reply on HN