Live data from Hacker News

Serverless Map/Reduce

tothestars.io

101–110 of 161 posts

Re: Serverless Map/Reduce

#101
post #50

OT: I teach computational methods and even as much as I dislike teaching/conflating it with web dev, I have included "let's build a web app" because students like building and deploying a thing, and because Heroku has a free tier. I've considered the possibility of having students do things on AWS (beyond web dev), including Lambda, and just expensing the costs. It seems feasible to quickly set up every student with…

Check this and the github education pack out: https://aws.amazon.com/education/awseducate/

Re: Serverless Map/Reduce

#102

The article counts characters in documents stored on S3 - which makes sense since S3 is great for storing documents and can handle unlimited concurrency, priced per usage. But what's the solution for structured data? DynamoDB is the obvious main candidate, but it's billed by hour and high concurrency is very expensive, requiring complicated temporary increases and decreases of concurrency that are hard to predict. Is…

[deleted]

Re: Serverless Map/Reduce

#103

The article counts characters in documents stored on S3 - which makes sense since S3 is great for storing documents and can handle unlimited concurrency, priced per usage. But what's the solution for structured data? DynamoDB is the obvious main candidate, but it's billed by hour and high concurrency is very expensive, requiring complicated temporary increases and decreases of concurrency that are hard to predict. Is…

If you're doing any sort of table scan op then DDB perf/cost will be less than stellar. If you have an index / range key it works really (like really) well -- even in massively parallel situations.

If you're dealing with a TON (5+ TB) of data I recommend heading in to RDS, BQ, or redshift.

Re: Serverless Map/Reduce

#104

I do not agree with term serverless. Amazon Lambda is a service, therefore there is a server involved. It's like saying deathless meat, because someone else killed the animal you are consuming.

With AWS API gateway + lambda + dynamodb, I maintain exactly zero servers. Zero physical servers. Zero server OSes. Zero sysadmin. It's perfectly reasonable IMO for that to be called "serverless".

With my supermarket, each time I buy a chicken I maintain zero farms. Zero physical farms. Zero farmers.

It's perfectly reasonable IMO for that to be called "farmless chickens".

Re: Serverless Map/Reduce

#105

I do not agree with term serverless. Amazon Lambda is a service, therefore there is a server involved. It's like saying deathless meat, because someone else killed the animal you are consuming.

With AWS API gateway + lambda + dynamodb, I maintain exactly zero servers. Zero physical servers. Zero server OSes. Zero sysadmin. It's perfectly reasonable IMO for that to be called "serverless".

With my car dealer, I maintain exactly zero factories. Zero physical factories. Zero factory workers.

It's perfectly reasonable IMO for that to be called "factory-less car".

Re: Serverless Map/Reduce

#106
post #103

The article counts characters in documents stored on S3 - which makes sense since S3 is great for storing documents and can handle unlimited concurrency, priced per usage. But what's the solution for structured data? DynamoDB is the obvious main candidate, but it's billed by hour and high concurrency is very expensive, requiring complicated temporary increases and decreases of concurrency that are hard to predict. Is…

If you're doing any sort of table scan op then DDB perf/cost will be less than stellar. If you have an index / range key it works really (like really) well -- even in massively parallel situations. If you're dealing with a TON (5+ TB) of data I recommend heading in to RDS, BQ, or redshift.

It's less the total size of the data I'm worried about and more the concurrency. For example, say I had a process that retrieved 1000 tiny records (using index query) and ran some cpu-intensive calculation on them, and I wanted to run 1000 of those processes simultaneously to reduce into a final result. This would require tuning dynamo to thousands of concurrent reads (and maybe writes, depending on the process), then scaling it back down after the operation because it is very costly and priced by hour. This makes it complicated and expensive on dynamo.

It seems the only storage services compatible with variable unlimited bursts of concurrency are S3 and SimpleDB. S3 comes with many problems for handling structured data (no update of records only replace, locking, listing items is slow/costly, etc.). SimpleDB is no longer being iterated, is limited to 10gb per domain, and looks like it's being slowly phased out.

It seems like massively parallel lambdas depend on few fetches of large blobs of data - which is basically batch-processing EMR-style, or better suited to redshift. Not something that opens the door for novel use-cases.

I would have really liked for dynamodb to be more of a service than a vm. I wish its concurrency was unlimited and you paid for usage rather than time. Basically DynamoDB with SimpleDB pricing.

Re: Serverless Map/Reduce

#107

Earlier quoted context omitted.

With AWS API gateway + lambda + dynamodb, I maintain exactly zero servers. Zero physical servers. Zero server OSes. Zero sysadmin. It's perfectly reasonable IMO for that to be called "serverless".

With my supermarket, each time I buy a chicken I maintain zero farms. Zero physical farms. Zero farmers. It's perfectly reasonable IMO for that to be called "farmless chickens".

It's just a term. Don't worry about it.

Re: Serverless Map/Reduce

#108
post #101
post #50

OT: I teach computational methods and even as much as I dislike teaching/conflating it with web dev, I have included "let's build a web app" because students like building and deploying a thing, and because Heroku has a free tier. I've considered the possibility of having students do things on AWS (beyond web dev), including Lambda, and just expensing the costs. It seems feasible to quickly set up every student with…

Check this and the github education pack out: https://aws.amazon.com/education/awseducate/

Oh, very nice! I hadn't heard that AWS had a program. I've used Github's education pack, once as an educator to get a free org with 20 private accounts, and once where I asked students to signup. The former situation was too cumbersome (I had to create a repo for each student, and students constantly confused their own personal account with their org account), and in the latter, only a few students out of 20 got approved by the time the course ended -- I think someone told me the process was backlogged.

I'm willing just to expense things on my account for the sake of simplicity but will see if AWS has a more streamlined process for class/per-student approval. Thanks!

Re: Serverless Map/Reduce

#110
post #103

Earlier quoted context omitted.

If you're doing any sort of table scan op then DDB perf/cost will be less than stellar. If you have an index / range key it works really (like really) well -- even in massively parallel situations. If you're dealing with a TON (5+ TB) of data I recommend heading in to RDS, BQ, or redshift.

It's less the total size of the data I'm worried about and more the concurrency. For example, say I had a process that retrieved 1000 tiny records (using index query) and ran some cpu-intensive calculation on them, and I wanted to run 1000 of those processes simultaneously to reduce into a final result. This would require tuning dynamo to thousands of concurrent reads (and maybe writes, depending on the process), the…

Just use RDS and S^3 for the blobs. RDS can do tens of thousands of index lookups a second.

If you only need one index, then just name your s3 document by the compound index value and call it a day.

Otherwise, just use RDS for everything.

Post reply on HN