Serverless Map/Reduce

Post by ec109685 » Sat, Nov 05, 2016, 2:38 AM UTC

kpil wrote: post #57

badminton1 wrote: post #52

I do not agree with term serverless. Amazon Lambda is a service, therefore there is a server involved. It's like saying deathless meat, because someone else killed the animal you are consuming.

Exactly. The cloud is just someone else's computer. It's a genuine business case, as running a lean and secure server farm is no child's play, but it's also oversold so much that my eyes hurts. Actual serverless computing would be implemented as p2p. Possibly interesting in a typical EvilCorpesque large corporation. Besides, Seti@home (Boinc) have been doing serverless computing for almost 20 years and it wasn't the…

Post by thinkloop » Sat, Nov 05, 2016, 3:34 AM UTC

ec109685 wrote: post #110

thinkloop wrote: post #106

Earlier quoted context omitted.

It's less the total size of the data I'm worried about and more the concurrency. For example, say I had a process that retrieved 1000 tiny records (using index query) and ran some cpu-intensive calculation on them, and I wanted to run 1000 of those processes simultaneously to reduce into a final result. This would require tuning dynamo to thousands of concurrent reads (and maybe writes, depending on the process), the…

Just use RDS and S^3 for the blobs. RDS can do tens of thousands of index lookups a second. If you only need one index, then just name your s3 document by the compound index value and call it a day. Otherwise, just use RDS for everything.

Post by badminton1 » Sat, Nov 05, 2016, 4:11 AM UTC

mypalmike wrote: post #99

badminton1 wrote: post #93

Earlier quoted context omitted.

Yes, I also think that's a much more accurate term that better represents the concept.

Great. We now know what color to paint the bike shed. Hoorah, progress!

Post by badminton1 » Sat, Nov 05, 2016, 4:18 AM UTC

ec109685 wrote: post #112

badminton1 wrote: post #105

Earlier quoted context omitted.

With my car dealer, I maintain exactly zero factories. Zero physical factories. Zero factory workers. It's perfectly reasonable IMO for that to be called "factory-less car".

If originally you had to precure time at a factory to produce your car and now you can go to a dealer and buy a car, I would call that new way factory-less — the factory has been taken out of your personal car buying equation.

Post by ma2rten » Sat, Nov 05, 2016, 4:40 AM UTC

jedberg wrote: post #26

teej wrote: post #25

Earlier quoted context omitted.

Some are, some aren't. For example, my experience has been that it's harder to get the 100 S3 bucket-per-account limit raised.

That used to be a hard technological limitation but they fixed it a few months back. You should be able to get more now (but you have to have a good justification for it).