Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Serverless Revolution Has Stalled

infoq.com

Post reply on HN
Print view View original

281–290 of 670 posts

Re: The Serverless Revolution Has Stalled

#281

Post by Supermancho »

joana035 wrote: post #264
mark242 wrote: post #247

Earlier quoted context omitted.

Yes, this exactly. The fact that I can guarantee that my code is running, and have entire massive teams at Amazon whose literal job title is to keep my code running, it is such a freeing feeling. I might be able to delete rows from Dynamo from my code messing up, but Dynamo will never go offline.

Are you really sure?

Tracking exactly what individual db entries ran and didn't run when there's one of these "serverless" "painless" interruptions to an ETL, ensures that you have an endless number of ever-changing status columns and re-run processes.

Are you really sure that it all re-ran? No. You have to query your latest batch - which means you have to have arbitrary boundaries for when failures might have occurred like say, last 24 hours. If was a weekend, just query the whole database looking for any entry that's missing the "final" status success. Try not to run anything while you're doing this, unless your system is tolerant to the DB being locked for a full-table scan.

This is all very fragile and change-averse. Who would choose to run an ETL like this? I have worked for large companies who all end up with this rickety system. Every one, every time. God forbid you want to run tests with a mock set of serverless AWS services.

Re: The Serverless Revolution Has Stalled

#282

Post by franciscop »

konjin wrote: post #245
Demiurge wrote: post #227

Earlier quoted context omitted.

How is owning an AWS account easier than a cheap dedicates server? You should have a code repo and backups in either case. At least some people mention "but I can scale this lambda x1000" and that's one advantage... But you can do all those tasks on a hetzner server in same amount of type, just python scripts.

Because you need two servers. One to run the application. One to run the application while the first is being upgraded. Now you need three servers, one to run the application, one as backup and one as load balancer. 99.99% uptime means you can be down no more than an hour a year. Which is really easy to overshoot when you're dicking around with a dist upgrade on three linux boxes. So that's why you end up with a serv…

99.99% of the websites out there don't need 99.99% uptime (from a business point of view)

Re: The Serverless Revolution Has Stalled

#283

Post by lstroud »

fizixer wrote: post #145
whalesalad wrote: post #93

Earlier quoted context omitted.

I felt your pain immediately and decided to write my own mini-framework to accomplish this. What I have now is a loosely coupled, serverless, frontend+backend monorepo that wraps AWS SAM and CloudFormation. At the end of the day it is just a handful of scripts and some foundational conventions. I just (this morning!) started to put together notes and docs for myself on how I can generalize and open source this to mak…

> ... stack is vue/python/s3/lambda/dynamodb/stripe Did you chose python for backend dev? Any framework like flask or django? Also no preference of single-language for both backend and front-end, by using node.js backend? Just trying to get into web-dev.

I really like Chalice for serverless Python projects.

Re: The Serverless Revolution Has Stalled

#284

Post by ethanwillis »

shotgunsparkle wrote: post #254
Demiurge wrote: post #227

Earlier quoted context omitted.

How is owning an AWS account easier than a cheap dedicates server? You should have a code repo and backups in either case. At least some people mention "but I can scale this lambda x1000" and that's one advantage... But you can do all those tasks on a hetzner server in same amount of type, just python scripts.

On the flip side, serverless can scale to 0. Makes it very cost effective for one off functions

It doesn't actually scale to 0. The scale to 0 argument is actually just accounting trickery. Servers are still waiting around burning cycles doing nothing waiting to run a serverless function. All that has changed is that Amazon has spread the cost of low utilization across all customers. Which gets baked into your cost as well.

Re: The Serverless Revolution Has Stalled

#285

Post by Silhouette »

brianwawok wrote: post #273
k__ wrote: post #86

Earlier quoted context omitted.

The problem is getting K8s to the point that it can go almost anywhere.

Eh? I was able to go from a docker image to live and 0 downtime deploys in a few hours?

But how many hours/days/years did it take you to learn how all the parts move before you became capable of setting something like that up within a few hours?

Anecdotally, stories of businesses abandoning entire cloud automation projects because they wasted weeks and never had anything to show from it don't seem unusual, so evidently a considerable amount of knowledge and skill is required to get value out of these tools.

Re: The Serverless Revolution Has Stalled

#286

Post by anamexis »

mark242 wrote: post #222

Here's why I love serverless. I cannot tell you the number of times I have implemented "upload your photo and it'll get resized to (profile avatar size from design specs)". It's ridiculous, and it's one of those things that everyone burns time implementing their fun hook into Imagemagick. Now I have one lambda that gets pointed at a new record stream from an S3 bucket, and I'm done. I cannot tell you the number of ti…

I don't know, it sounds like you've just implemented all of those things n+1 times.

Re: The Serverless Revolution Has Stalled

#287

Post by ClumsyPilot »

konjin wrote: post #245
Demiurge wrote: post #227

Earlier quoted context omitted.

How is owning an AWS account easier than a cheap dedicates server? You should have a code repo and backups in either case. At least some people mention "but I can scale this lambda x1000" and that's one advantage... But you can do all those tasks on a hetzner server in same amount of type, just python scripts.

Because you need two servers. One to run the application. One to run the application while the first is being upgraded. Now you need three servers, one to run the application, one as backup and one as load balancer. 99.99% uptime means you can be down no more than an hour a year. Which is really easy to overshoot when you're dicking around with a dist upgrade on three linux boxes. So that's why you end up with a serv…

Basically nothing I have ever used had 99.99% uptime except maybe the phone and emergency services. Why do you need this level of uptime unless life and limb depends on you?

My bank has regular scheduled downtime, Steam/Dota has regular downtime, entire Microsoft datacenters had issues and had around a day of downtime, Lime, Uber, my mobile carrier - all had downtime or bugs or glitches that were equivalent to downtime.

If you really need crazy uptime, then you must make sure you have no bugs, because bugs cause more downtime than any hardware failures.

That means an approach to software development that isn't relying on flavour of the month js library.

Re: The Serverless Revolution Has Stalled

#288

Post by ErikAugust »

justinram11 wrote: post #220

As someone on a two-man-team who runs a lot of little "utility" functions in AWS Lambda with the Serverless Framework[1] to support our DevOps / Build processes, it's been one of the most productive tools in my toolkit (after the initial learning curve, of course). It allows me to stand up a practically maintenance-free endpoint in a matter of hours (usually to glue separate services together): * Want to run a quick…

They are good for message queue handlers.

But not good for production REST API endpoints in my experience. In the context of AWS Lambda, you are talking a 29 second timeout in API Gateway. API Gateway is fairly expensive. Also, is connection pooling a solved problem?

Many times you can do better with an EC2 instance running PM2 and Express, or whatever. Like, what does Serverless Framework do locally anyway? Express? Why not just run Express?

Re: The Serverless Revolution Has Stalled

#289

Post by rogerkirkness »

mark242 wrote: post #222

Here's why I love serverless. I cannot tell you the number of times I have implemented "upload your photo and it'll get resized to (profile avatar size from design specs)". It's ridiculous, and it's one of those things that everyone burns time implementing their fun hook into Imagemagick. Now I have one lambda that gets pointed at a new record stream from an S3 bucket, and I'm done. I cannot tell you the number of ti…

It would be nice if someone made a machine that would run all three of those discrete, generalized computing tasks on a single general purpose "thing". That way, you could infinitely scale your work up or down within available memory based on what needed to be done. It would even intelligently free something from memory if it wasn't being used.

Servers.

Re: The Serverless Revolution Has Stalled

#290

Post by justinram11 »

Supermancho wrote: post #281
joana035 wrote: post #264

Earlier quoted context omitted.

Are you really sure?

Tracking exactly what individual db entries ran and didn't run when there's one of these "serverless" "painless" interruptions to an ETL, ensures that you have an endless number of ever-changing status columns and re-run processes. Are you really sure that it all re-ran? No. You have to query your latest batch - which means you have to have arbitrary boundaries for when failures might have occurred like say, last 24…

I generally agree that if you're setting up multi-step or "enterprise grade" ETL processes that it's a job better handled by something like Airflow (we've done a few smaller ones with Step Functions which hasn't been terrible, but Airflow is still better).

Our use of AWS Lambda for ETL is usually just one-off small processes that don't have dependencies.

As soon as you start getting into "This ETL depends on X, Y, and Z being run first" I think you're out of "small utility" territory.

Post reply on HN