OT: I teach computational methods and even as much as I dislike teaching/conflating it with web dev, I have included "let's build a web app" because students like building and deploying a thing, and because Heroku has a free tier. I've considered the possibility of having students do things on AWS (beyond web dev), including Lambda, and just expensing the costs. It seems feasible to quickly set up every student with…
Serverless Map/Reduce
101–110 of 161 posts
Re: Serverless Map/Reduce
#102The article counts characters in documents stored on S3 - which makes sense since S3 is great for storing documents and can handle unlimited concurrency, priced per usage. But what's the solution for structured data? DynamoDB is the obvious main candidate, but it's billed by hour and high concurrency is very expensive, requiring complicated temporary increases and decreases of concurrency that are hard to predict. Is…
Re: Serverless Map/Reduce
#103The article counts characters in documents stored on S3 - which makes sense since S3 is great for storing documents and can handle unlimited concurrency, priced per usage. But what's the solution for structured data? DynamoDB is the obvious main candidate, but it's billed by hour and high concurrency is very expensive, requiring complicated temporary increases and decreases of concurrency that are hard to predict. Is…
If you're dealing with a TON (5+ TB) of data I recommend heading in to RDS, BQ, or redshift.
Re: Serverless Map/Reduce
#104I do not agree with term serverless. Amazon Lambda is a service, therefore there is a server involved. It's like saying deathless meat, because someone else killed the animal you are consuming.
With AWS API gateway + lambda + dynamodb, I maintain exactly zero servers. Zero physical servers. Zero server OSes. Zero sysadmin. It's perfectly reasonable IMO for that to be called "serverless".
It's perfectly reasonable IMO for that to be called "farmless chickens".
Re: Serverless Map/Reduce
#105I do not agree with term serverless. Amazon Lambda is a service, therefore there is a server involved. It's like saying deathless meat, because someone else killed the animal you are consuming.
With AWS API gateway + lambda + dynamodb, I maintain exactly zero servers. Zero physical servers. Zero server OSes. Zero sysadmin. It's perfectly reasonable IMO for that to be called "serverless".
It's perfectly reasonable IMO for that to be called "factory-less car".
Re: Serverless Map/Reduce
#106The article counts characters in documents stored on S3 - which makes sense since S3 is great for storing documents and can handle unlimited concurrency, priced per usage. But what's the solution for structured data? DynamoDB is the obvious main candidate, but it's billed by hour and high concurrency is very expensive, requiring complicated temporary increases and decreases of concurrency that are hard to predict. Is…
If you're doing any sort of table scan op then DDB perf/cost will be less than stellar. If you have an index / range key it works really (like really) well -- even in massively parallel situations. If you're dealing with a TON (5+ TB) of data I recommend heading in to RDS, BQ, or redshift.
It seems the only storage services compatible with variable unlimited bursts of concurrency are S3 and SimpleDB. S3 comes with many problems for handling structured data (no update of records only replace, locking, listing items is slow/costly, etc.). SimpleDB is no longer being iterated, is limited to 10gb per domain, and looks like it's being slowly phased out.
It seems like massively parallel lambdas depend on few fetches of large blobs of data - which is basically batch-processing EMR-style, or better suited to redshift. Not something that opens the door for novel use-cases.
I would have really liked for dynamodb to be more of a service than a vm. I wish its concurrency was unlimited and you paid for usage rather than time. Basically DynamoDB with SimpleDB pricing.
Re: Serverless Map/Reduce
#107Earlier quoted context omitted.
With AWS API gateway + lambda + dynamodb, I maintain exactly zero servers. Zero physical servers. Zero server OSes. Zero sysadmin. It's perfectly reasonable IMO for that to be called "serverless".
With my supermarket, each time I buy a chicken I maintain zero farms. Zero physical farms. Zero farmers. It's perfectly reasonable IMO for that to be called "farmless chickens".
Re: Serverless Map/Reduce
#108OT: I teach computational methods and even as much as I dislike teaching/conflating it with web dev, I have included "let's build a web app" because students like building and deploying a thing, and because Heroku has a free tier. I've considered the possibility of having students do things on AWS (beyond web dev), including Lambda, and just expensing the costs. It seems feasible to quickly set up every student with…
Check this and the github education pack out: https://aws.amazon.com/education/awseducate/
I'm willing just to expense things on my account for the sake of simplicity but will see if AWS has a more streamlined process for class/per-student approval. Thanks!
Re: Serverless Map/Reduce
#109Re: Serverless Map/Reduce
#110Earlier quoted context omitted.
If you're doing any sort of table scan op then DDB perf/cost will be less than stellar. If you have an index / range key it works really (like really) well -- even in massively parallel situations. If you're dealing with a TON (5+ TB) of data I recommend heading in to RDS, BQ, or redshift.
It's less the total size of the data I'm worried about and more the concurrency. For example, say I had a process that retrieved 1000 tiny records (using index query) and ran some cpu-intensive calculation on them, and I wanted to run 1000 of those processes simultaneously to reduce into a final result. This would require tuning dynamo to thousands of concurrent reads (and maybe writes, depending on the process), the…
If you only need one index, then just name your s3 document by the compound index value and call it a day.
Otherwise, just use RDS for everything.