Live data from Hacker News

Serverless Map/Reduce

tothestars.io

81–90 of 161 posts

Re: Serverless Map/Reduce

#81
post #32

Is there any AWS Lambda equivalent that could be deployed on bare metal?

Docker

That's a horrible and flat out wrong answer. Docker simply provides a transferable containerized execution environment for processes with a simple push pull workflow.

Lambda is a fully elastic infrastructure you can build, deploy, and execute functions.

Simply having docker-engine and docker-cli installed on my machine doesn't give me anything close to lambda.

Re: Serverless Map/Reduce

#82
post #15

This is a screenshot of my google search from 2 days ago: http://i.imgur.com/BNAcSsn.png I've been using Lambda quite a bit, I think it's SO amazingly useful. Tasks that are highly parallelized and CPU intensive can literally be infinitely scaled out. I find it weird that their poster child use case is still always a reactive event like watching S3 and formatting images. There are so many use cases for directly invok…

So google compute setup I did a while back with preemptible instances + a celery queue + some autoscale based on load...

The guts to make all that work was 50 or so lines of config. I think my auto scale script was 20 lines or so of Python.

I guess the biggest downside was spinning up the new server took about 2 minutes, so for big load spikes it took a bit for it to level out... but with GCE per minute billing, all the napkin math I did says it is a fair bit cheaper per cpu unit to do it this way.

In conclusion, someone good with devops + a normal work queue system could do this years ago. I guess it's cool to lower the barrier to entry, but not it does not seem like a game changer. It totally IS cool to scale out backend work to a job farm. Just seems like not the only way to do it.

Re: Serverless Map/Reduce

#83
post #14
post #2

If it doesn't run on a server, then what does this plumbing-work run on? Clickbait name?

Call it Function as a Service if you prefer.

How about "worker queue" ? because it is exactly what is it. Obfuscating the details is silly. There is nothing new about the technology. It is only marketed as new (which is reflected in the pricing).

Re: Serverless Map/Reduce

#84
post #15

This is a screenshot of my google search from 2 days ago: http://i.imgur.com/BNAcSsn.png I've been using Lambda quite a bit, I think it's SO amazingly useful. Tasks that are highly parallelized and CPU intensive can literally be infinitely scaled out. I find it weird that their poster child use case is still always a reactive event like watching S3 and formatting images. There are so many use cases for directly invok…

if it's one of you will not be able to get the limit if you do this even at a rate of 1 req/sec you will be paying close to 1.4 million USD per month just for compute.

Re: Serverless Map/Reduce

#85
post #32

Is there any AWS Lambda equivalent that could be deployed on bare metal?

I've recently come across funktion[1] which is designed to run on kubernetes (which can run on baremetal). Also there is OpenWhisk[2] which is open sourced by IBM. Probably alot more but those are the ones I know off the top of my head.

[1] - https://github.com/fabric8io/funktion.

[2] - https://developer.ibm.com/openwhisk/

Re: Serverless Map/Reduce

#86

Earlier quoted context omitted.

Even that's tricky, because it's also Backend as a Service BaaS/MBaaS at work, various frameworks and other solutions, etc.

It can power a front end. It's like a micro container as a service.

As long as you don't mind 3s startup latency, or 200ms hot path latency?

Re: Serverless Map/Reduce

#87
post #63
post #47

Earlier quoted context omitted.

Is this a parody? Prescriptivism in law is completely unthinkable to UK/US native English speakers.

Well, in France, or Germany (where I’m from), I grew up right after such a change had happened. I’m not sure why English speakers resist it so much, it simplifies the situation so much, and isn’t an issue anyway.

It's a philosophical red line, really. It's simply not something that belongs to the state, or even to the country. It would be like trying to legislate that the sky was green.

Similarly the metric system - which has actual advantages that mere linguistic change does not, and legal support - has been accepted only very grudgingly. ( https://en.wikipedia.org/wiki/Metric_Martyrs )

(Doesn't linguistic reform occasionally cause discontinuities in literature? What happens if you want to use a metaphor or turn of phrase that's been forcibly obsoleted?

Like, I might occasionally refer to something as being "tuppence-ha'penny", despite that coin being abolished over thirty years ago.

Re: Serverless Map/Reduce

#88
post #80

I do not agree with term serverless. Amazon Lambda is a service, therefore there is a server involved. It's like saying deathless meat, because someone else killed the animal you are consuming.

Fight the good fight, you're not the only one standing up against this ridiculous moniker. Never back down. Keep calling out the bullshit every time and everywhere this term is promoted. People ask "what's the big deal?" well the big deal is ,calling a service "serverless" is both a lie and misleading for marketing purposes.

Like when people say that integration tests are unit tests. That gets me going every single time (a unit test is a test on a unit of code, e.g: a function or method).

If you have seen "the interview", this is like when the guy says "on the line", and gets corrected each time.

Re: Serverless Map/Reduce

#89
The article counts characters in documents stored on S3 - which makes sense since S3 is great for storing documents and can handle unlimited concurrency, priced per usage.

But what's the solution for structured data? DynamoDB is the obvious main candidate, but it's billed by hour and high concurrency is very expensive, requiring complicated temporary increases and decreases of concurrency that are hard to predict.

Is there a good solution for running massively parallel lamdas on stuctured data?

Re: Serverless Map/Reduce

#90
post #87
post #63

Earlier quoted context omitted.

Well, in France, or Germany (where I’m from), I grew up right after such a change had happened. I’m not sure why English speakers resist it so much, it simplifies the situation so much, and isn’t an issue anyway.

It's a philosophical red line, really. It's simply not something that belongs to the state, or even to the country. It would be like trying to legislate that the sky was green. Similarly the metric system - which has actual advantages that mere linguistic change does not, and legal support - has been accepted only very grudgingly. ( https://en.wikipedia.org/wiki/Metric_Martyrs ) (Doesn't linguistic reform occasionall…

> What happens if you want to use a metaphor or turn of phrase that's been forcibly obsoleted?

The same that happens when you try to read any old literature. Like the Illias, which talks about "wine-colored sea" (it was dark blue – the brightness was meant, not hue), or shakespear, etc.

> It's simply not something that belongs to the state, or even to the country.

That likely differs here because the German language was unified in the first place by an academic, and not through consensus.

Post reply on HN