Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Serverless Map/Reduce

tothestars.io

Post reply on HN
Print view View original

41–50 of 161 posts

Re: Serverless Map/Reduce

#41

Post by vidarh »

ceejayoz wrote: post #22
ronack wrote: post #20

Earlier quoted context omitted.

We had it raised to several thousand concurrent requests with no questions asked. YMMV based on age of account, amount of spend, etc.

Yeah, most of the limits are there just to prevent "whoops, we fired up 500 $2/hour servers and left them running all month, can you refund us?" sort of newbie situations. Very few are technological and AWS has been happy to bump them up for me in each occasion I've had a need.

I've had plenty of cases where it's taken time, and required escalation and justification to get even relatively modest instance limits raised, depending on the data centre and instance types.

It's not technological - it's about ensuring they have a reasonable handle on capacity and what they happen to have plenty of.

It's not a problem as long as you're not asking for astronomical limits, but people need to be aware of it and request increases in advance of starting to hit limits rather than assume it'll be near immediate to get them lifted.

Re: Serverless Map/Reduce

#42

Post by sharpy »

chrisseaton wrote: post #36
jedberg wrote: post #26

Earlier quoted context omitted.

That used to be a hard technological limitation but they fixed it a few months back. You should be able to get more now (but you have to have a good justification for it).

100 seems an odd number to be a technological limitation, doesn't it? It's not a power of two or anything so it's not the size of an integer. How could technology have been limiting it?

S3 bucket names are globally scoped - ie. they are literally scarce resources.

Re: Serverless Map/Reduce

#43

Post by benballjr »

kuschku wrote: post #29
luhn wrote: post #13

Earlier quoted context omitted.

I'm not saying "serverless" is a good term, I'm saying that it's the term. It's won. You can argue all you want that it's a terribly misleading/incorrect term, but people aren't going to stop using it. So let's move on.

> It's won. That’s the linguistic descriptivist position. We can also require all papers and journals and conferences to use "Function as a Service" everywhere, and force all professors to teach "Function as a Service", and require all official publications to use "Function as a Service", by defining an authoritative dictionary, which gets its authority by law. Then wait a few months, and the term "serverless" will b…

It really has more to do with what the vendors creating the "serverless" tools perpetuating at the industry level. You can blame the technical marketing teams and tech media that also pick up the terminology (whoops, our bad).

Re: Serverless Map/Reduce

#44

Post by nostrademons »

etrain wrote: post #19
hueving wrote: post #17

Earlier quoted context omitted.

>Theoretically, you've taken something that would take 70 hours and had it run in 250ms without having to set up any additional infrastructure. And you've spent the cost of building out that 8 server infrastructure in one batch.

One of the nice things about lambda is the billing is super granular - you get billed at 100ms intervals. Assuming 70 hours at $0.000000834/100ms [1] The whole job costs $2.10. [1] - https://aws.amazon.com/lambda/pricing/

By comparison (since I was curious), 70 hours on an m3.medium spot instance will run around $0.70. On an on-demand, it's about $5.39. EMR will cost you about $7.00 on top of the EC2 costs.

If you can peg the CPU and don't mind getting interrupted, spot instances are still a fair bit cheaper. But Lambda looks pretty attractive for any other use-case where the statelessness of Lambda doesn't bite you.

Re: Serverless Map/Reduce

#45

Post by dnackoul »

Does anyone have experience building mobile backends in Lambda? I was looking at an API Gateway / Lambda / Amazon RDS stack for building a central data store and was wondering what people's experience with that setup is?

Re: Serverless Map/Reduce

#46

Post by [deleted] »

nostrademons wrote: post #44
etrain wrote: post #19

Earlier quoted context omitted.

One of the nice things about lambda is the billing is super granular - you get billed at 100ms intervals. Assuming 70 hours at $0.000000834/100ms [1] The whole job costs $2.10. [1] - https://aws.amazon.com/lambda/pricing/

By comparison (since I was curious), 70 hours on an m3.medium spot instance will run around $0.70. On an on-demand, it's about $5.39. EMR will cost you about $7.00 on top of the EC2 costs. If you can peg the CPU and don't mind getting interrupted, spot instances are still a fair bit cheaper. But Lambda looks pretty attractive for any other use-case where the statelessness of Lambda doesn't bite you.

[deleted]

Re: Serverless Map/Reduce

#47

Post by pjc50 »

kuschku wrote: post #29
luhn wrote: post #13

Earlier quoted context omitted.

I'm not saying "serverless" is a good term, I'm saying that it's the term. It's won. You can argue all you want that it's a terribly misleading/incorrect term, but people aren't going to stop using it. So let's move on.

> It's won. That’s the linguistic descriptivist position. We can also require all papers and journals and conferences to use "Function as a Service" everywhere, and force all professors to teach "Function as a Service", and require all official publications to use "Function as a Service", by defining an authoritative dictionary, which gets its authority by law. Then wait a few months, and the term "serverless" will b…

Is this a parody? Prescriptivism in law is completely unthinkable to UK/US native English speakers.

Re: Serverless Map/Reduce

#48

Post by res0nat0r »

kuschku wrote: post #29
luhn wrote: post #13

Earlier quoted context omitted.

I'm not saying "serverless" is a good term, I'm saying that it's the term. It's won. You can argue all you want that it's a terribly misleading/incorrect term, but people aren't going to stop using it. So let's move on.

> It's won. That’s the linguistic descriptivist position. We can also require all papers and journals and conferences to use "Function as a Service" everywhere, and force all professors to teach "Function as a Service", and require all official publications to use "Function as a Service", by defining an authoritative dictionary, which gets its authority by law. Then wait a few months, and the term "serverless" will b…

Seems like we are getting in the weeds over silly semantics.

When you hear "serverless", just think of it as a service where you don't have to worry about running or maintaining your own OS on a server weather that is virtual or bare metal. That's it, it's as simple as that.

Re: Serverless Map/Reduce

#50

Post by danso »

OT: I teach computational methods and even as much as I dislike teaching/conflating it with web dev, I have included "let's build a web app" because students like building and deploying a thing, and because Heroku has a free tier.

I've considered the possibility of having students do things on AWS (beyond web dev), including Lambda, and just expensing the costs. It seems feasible to quickly set up every student with controlled access via IAM...but is there a way to set up rate-limiting, ideally through a policy? That is, shut an IAM down if a student accidentally invokes a million processes? Or, for that matter, limiting the storage capacity of a S3 bucket?

Post reply on HN