Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Leaving serverless led to performance improvement and a simplified architecture

unkey.com

Post reply on HN
Print view View original

171–180 of 272 posts

Re: Leaving serverless led to performance improvement and a simplified architecture

#171

Post by tacticus »

dprotaso wrote: post #126
jjice wrote: post #100

Like the article says, I think serverless has it's place, but I don't think it's for most applications. I can't see myself _ever_ using serverless services as a core part of my application for pretty much any startup, if I can avoid it. The infrastructure overhead is actually worse, IMO. Everything is so platform specific and it's much stranger to test and develop against locally. Each platform has a different way to…

That’s why Knative (Serverless on Kubernetes) accepts containers. It’s the standard packaging format that lets you lift and shift apps to many different platforms.

Lambda shifting to this model would be such a nice future. though even the lambda variants that can run containers have some painful issues.

Re: Leaving serverless led to performance improvement and a simplified architecture

#172

Post by sunrunner »

themafia wrote: post #167
reactordev wrote: post #145

Earlier quoted context omitted.

Uploads to an S3 bucket can trigger a lambda… don’t complicate things. The upload trigger can tell the system about the upload and the client can continue on their day. Uploader on the client uses presigned url. S3 triggers lambda. Lambda function takes file path and tells background workers about it either via queue, mq, rest, gRPC, or doing the lift in workflow etl functions. Easy peasy. /s

> Easy peasy. /s It actually is though. I don't need to build a custom upload client, I don't need to manage restart behavior, I get automatic restarts if any of the background workers fail, I have a dead letter queue built in to catch unusual failures, I can tie it all together with a common API that's a first class component of the system. Working in the cloud forces you to address the hard problems first. If you a…

> Working in the cloud forces you to address the hard problems first.

It also forces you to address all the non-existent problems first, the ones you just wish you had like all the larger companies that genuinely have to deal with thousands of file upload per second.

And don't forget all the new infrastructure you added to do the job of just receiving the file in your app server and putting it into the place it was going to go anyway but via separate components that all always seem to end up with individual repositories, separate deployment pipelines, and that can't be effectively tested in isolation without going into their target environment.

And all the additional monitoring you need on each of the individual components that were added, particularly on those helpful background workers to make sure they're actually getting triggered (you won't know they're failing if they never got called in the first place due to misconfiguration).

And you're now likely locked into your upload system being directly coupled to your cloud vendor. Oh wait, you used Minio to provide a backend-agnostic intermediate layer? Great, that's another layer that needs managing.

Is a content delivery network better suited to handling concurrent file uploads from millions of concurrent users than your app server? I'd honestly hope so, that's what it's designed for. Was it necessary? I'd like to see the numbers first.

At the end of the day, every system design decision is a trade off and almost always involves some kind of additional complexity for some benefit. It might be worth the cost, but a lot of these system designs don't need this many moving parts to achieve the same results and this only serves to add complexity without solving a direct problem.

If you're actually that company, good for you and genuinely congratulations on the business success. The problem is that companies that don't currently and may never need that are being sold system designs that, while technically more than capable, are over-designed for the problem they're solving.

Re: Leaving serverless led to performance improvement and a simplified architecture

#173

Post by andersmurphy »

dalberto wrote: post #162
mattmanser wrote: post #140

Earlier quoted context omitted.

Docker is much like microservices. Appropriate for a subset of apps and yet touted as being 'the norm' when it shouldn't be. There are drawbacks to using docker, such as security patching and operational overhead. And if you're blindly putting it into every project, how are you mitigating the risks it introduces? Worse, the big reason it was useful, managing dependency hell, has largely been solved by making develope…

Hard disagree. I've used Docker predominantly in monoliths, and it has served me well. Before that I used VMs (via Vagrant). Docker certainly makes microservices more tenable because of the lower overhead, but the core tenets of reproducibility and isolation are useful regardless of architecture.

Depends on the language. Java or Go you really don't need docker.

Re: Leaving serverless led to performance improvement and a simplified architecture

#174

Post by daxfohl »

marcosdumay wrote: post #153
brap wrote: post #142

Earlier quoted context omitted.

I don’t know if this is a good rule of thumb, I think it really depends on what you use the dependencies for, how often you need them, etc. Consider for example a single DB dependency. Should the server be close to the DB or the client? It depends. How often does the client need the server? How often does the server need the DB? Which usecases are expected to be fast and which can be sacrificed as slow? What can be c…

Oh, it's a very good rule of thumb. It's probably not universal, but it's really close to it. the problem is that nobody designs the dependencies flexible enough to let them run without fine-control. And the main application always wants to change the way it uses the dependencies, so it always needs further flexibility. You can build an exception to the rule if you explicitly try. But I'm not sure one appears natural…

Well, I guess one can take one more step back and say this is all merely an example of "premature optimization is the root of all evil". Unless you know a-priori that you have some very hard latency requirements, start with something simple and low-maintenance. If low-latency requirements come in later, then design that holistically, not just looking at your component. Make sure you're measuring the right things; OOTB metrics often miss the e2e experience. And IME most latency issues come from unexpected places; I know I've spent weeks optimizing services to get an extra percent or two out of them, only to realize there's a config setting that reduced latency by half.

So generally, simplicity is your friend when it comes to latencies (among other things). Fewer things to cause long-tail spikes, more simple things you can try out that don't break the whole system, whereas if you start with a highly-optimized thing up-front, fixing some unexpected long-tail issue may require a complete rewrite.

Also, check with your PM or end users as to whether latency is even important. If the call to your service is generally followed up to a call to some ten-second process, users aren't going to notice the 20ms improvement to your own thing.

Re: Leaving serverless led to performance improvement and a simplified architecture

#175

Post by themafia »

sunrunner wrote: post #172
themafia wrote: post #167

Earlier quoted context omitted.

> Easy peasy. /s It actually is though. I don't need to build a custom upload client, I don't need to manage restart behavior, I get automatic restarts if any of the background workers fail, I have a dead letter queue built in to catch unusual failures, I can tie it all together with a common API that's a first class component of the system. Working in the cloud forces you to address the hard problems first. If you a…

> Working in the cloud forces you to address the hard problems first. It also forces you to address all the non-existent problems first, the ones you just wish you had like all the larger companies that genuinely have to deal with thousands of file upload per second. And don't forget all the new infrastructure you added to do the job of just receiving the file in your app server and putting it into the place it was g…

> the ones you just wish you had

You will have these problems. Not as often as the larger companies but to imagine that they simply don't exist is the opposite of sound engineering.

> if they never got called in the first place due to misconfiguration

Centralized logging is built into all these platforms. Debugging these issues is one of the things that becomes absurdly easy.

> likely locked into your upload system

The protocol provided by S3 is available through dozens of vendors.

> Was it necessary?

It only matters if it is of equivalent or lessor cost.

> every system design decision is a trade off

Yet you explicitly ignore these.

> are being sold system designs

No, I just read the documentation, and then built it. That's one of those "trade offs" you're willingly ignoring.

Re: Leaving serverless led to performance improvement and a simplified architecture

#176

Post by reactordev »

Dylan16807 wrote: post #170
stuartjohnson12 wrote: post #155

Earlier quoted context omitted.

> Uploads to an S3 bucket can trigger a lambda… don’t complicate things. I read this and was getting ready to angrily start beating my keyboard. The best satire is hard to detect.

I don't really get the joke. S3 triggering a lambda doesn't sound meaningfully more complicated than using a lambda by itself. What am I missing?

Solving a serverless limitation with more serverless so you can continue doing serverless when you can’t FormUpload a simple 101mb zip file as an application/octet-stream. Doubling down on it for a triple beat.

Re: Leaving serverless led to performance improvement and a simplified architecture

#177

Post by MrDarcy »

reactordev wrote: post #145
akdev1l wrote: post #87

Earlier quoted context omitted.

The way to work around this issue is to provide a presigned S3 url Have the users upload to s3 directly and then they can either POST you what they uploaded or you can find some other means of correlating the input (eg: files in s3 are prefixed with the request id or something) I agree this is annoying and maybe I’ve been in AWS ecosystem for too long. However having an API that accepts an unbounded amount of data is…

Uploads to an S3 bucket can trigger a lambda… don’t complicate things. The upload trigger can tell the system about the upload and the client can continue on their day. Uploader on the client uses presigned url. S3 triggers lambda. Lambda function takes file path and tells background workers about it either via queue, mq, rest, gRPC, or doing the lift in workflow etl functions. Easy peasy. /s

If you don’t do it this way you fail the system design interview.

Re: Leaving serverless led to performance improvement and a simplified architecture

#178

Post by danpalmer »

30ms p99 for a cache read! Serverless might have been a problem, but I'm not sure it was the problem. In my experience a p99 of 2ms is more typical - 30ms is the sort of time I'd expect for p99 on a database query in production serving.

You don't need process-local caches to get the sort of performance they're looking for, and there are good reasons why most teams avoid stateful processing, it's much harder to get right and has bad failure modes.

Re: Leaving serverless led to performance improvement and a simplified architecture

#179

Post by Dylan16807 »

reactordev wrote: post #176
Dylan16807 wrote: post #170

Earlier quoted context omitted.

I don't really get the joke. S3 triggering a lambda doesn't sound meaningfully more complicated than using a lambda by itself. What am I missing?

Solving a serverless limitation with more serverless so you can continue doing serverless when you can’t FormUpload a simple 101mb zip file as an application/octet-stream. Doubling down on it for a triple beat.

I wouldn't really call it "more" severless to rearrange the order a bit. Which makes it "solving a serverless limitation so you can continue doing severless". And that's just a deliberately awkward way of saying "solving a serverless limitation" because if you can solve it easily why would you not continue? Spite?

So I still don't see how it's notably worse than the idea of using serverless at all.

Re: Leaving serverless led to performance improvement and a simplified architecture

#180

Post by torginus »

evantbyrne wrote: post #84
randomtoast wrote: post #55

Earlier quoted context omitted.

If you're looking for a middle ground between VMs and serverless, ECS Fargate is a good option. Because a container is always running, you won't experience any cold start times.

ECS is good, just expensive and still requires more devops than it should. Docker Swarm is an easy way to run production container services on VMs. I built a free golang tool called Rove that provisions fresh Ubuntu VMs in one command and diffs updates. It's also easy-enough to use Swarm directly.

Honestly I didn't have a good experience with ECS (Fargate) - I remember I had to write a ton of CF deployment scripts+bash scripts, setting up a private AWS docker registry, having a terrible time debugging while my CF deployment always failed, deploys taking forever, finding out that AWS is too miserly to pay Docker to use the official repo so they are stuck on the free tier, meaning sometimes deploys would fail due to Dockerhub kicking the AWS docker agent out etc. It had limitations like not being able to attach a block volume to the docker instance, so overall I remember spending a week setting up the IaC for a simple-ass CRUD app on Fargate ECS.

Setting up the required roles and permissions was also a nightmare. The deployment round trip time was also awful.

The 2 good experiences I had with AWS was when we had a super smart devops guy who set up the whole docker pipeline on top of actual instances, so we could deploy our docker compose straight to a server in under 1 minute (this wasn't a scaled app), and had everything working.

Lambda is also pretty cool, you can just zip everything up and do a deploy from aws cli without much scripting and pretty straightforward IaC.

Post reply on HN