How we built a serverless architecture with AWS • Hacker News Board

Re: How we built a serverless architecture with AWS

#51

Post by remify » Fri, Jul 12, 2019, 9:39 AM UTC

closeparen wrote: post #29

nisten wrote: post #20

Earlier quoted context omitted.

I used to hate aws for how expensive their bandwidth and storage was, until I started actually using it last year. I think their new serverless stack is about to leave a lot of devops out of a job. You can setup a a CI/CD pipeline in about half an hour with amplify, at the previous company I remember it taking a good 3 weeks to get CircleCi up and running properly. And then moving a microservice over to it is basical…

>You can setup a a CI/CD pipeline in about half an hour with amplify, at the previous company I remember it taking a good 3 weeks to get CircleCi up and running properly. >And then moving a microservice over to it is basically 1 command, a few options, mostly just copy over the config from your old express backend with a few changes, and you're done. It's insane. As a an engineer at a decent sized tech company, this…

Except now you don't need an infrastructure team. That's the whole point of serverless architecture: to be able to scale without having huge team scale as well.

Re: How we built a serverless architecture with AWS

#52

Post by bborud » Fri, Jul 12, 2019, 9:46 AM UTC

I like the idea of serverless architectures, but I still wouldn't use it for anything that is important.

- Using a serverless architecture almost always implied getting married to your provider. You can run your code in only one place. You have given up all bargain power. When the relationship ends you have to build your system over again.

- It isn't really serverless; they're just not your servers.

- They are only efficient for the workloads the architecture is designed for. Stray outside the parameters and things start to become expensive, slow or both.

- If you use serverless architectures you have to make damn sure the people who built it stick around, because the only value you are left with if your provider folds or increases prices on you is inside the heads of the people who built the solution.

I have already seen friends getting burnt by this. Typically people build a prototype or a technology demo, it gets funding, the CTO insists that it isn't important to do anything about the serverless bits and just go with them (there is no pause to make good on "we'll fix it later" once money gets involved), then get jerked around by the service provider because they can't provide the support needed. Then they slam head first into the costs of actual production traffic which, somehow, even though it requires only basic arithmetic skills, none of them had been able to calculate before the huge bills started rolling in.

Re: How we built a serverless architecture with AWS

#53

Post by reilly3000 » Fri, Jul 12, 2019, 9:52 AM UTC

bzbz wrote: post #46

xsmasher wrote: post #37

Earlier quoted context omitted.

Do you mean you don't want it to handle 1k concurrent requests (you want some to be rejected or queued instead?) or do you mean that the concurrent execution causes some other problem? (honest question, not snark)

I think they mean there’s a 1k concurrent request limit that they hit. Though the alternative would be dedicated servers and load balancers, no?

Right, I'm referring to AWS limits. I was running a benchmark yesterday against a logging endpoint I made with a similar architecture to the article. One function is attached to a public ALB endpoint and does some validation then writes the event to SQS; this was taking 100-200ms with 128Mb of RAM. A second function was attached to the SQS queue; its job was to pull events and write them out to an external service (Stackdriver, which sinks to BigQuery). This function was taking 800-1200ms at 128Mb RAM, or 300-500ms at 512Mb (expensive!).

While running some load testing with Artillery I found that I was often getting 429 errors on my front-end endpoint. When pushing 500+ RPS, the 2nd function was taking up over 50% of the concurrent execution limit and new events coming into the front-end would get throttled and in this case thrown out. That also means that any future Lambdas in the same AWS account would exacerbate this problem. Our traffic is spiky and can easily hit 500+ RPS on occasion, so this really wasn't acceptable.

My solution was to refactor the 2nd function into a Fargate task that polls the SQS queue instead. It was easily able to handle any workload I threw at it, and also able to run 24/7 for a fraction of the cost of the Lambda. Each invocation of the Lambda was authenticating with the GCP SDK before passing the event and the Lambda has to stay executing while the 2 stages of network requests were completed.

I'm happy to report I haven't been able to muster a test that breaks anything since I started using Fargate!

https://docs.aws.amazon.com/lambda/latest/dg/concurrent-exec...

Re: How we built a serverless architecture with AWS

#54

Post by lentil » Fri, Jul 12, 2019, 12:16 PM UTC

reilly3000 wrote: post #53

bzbz wrote: post #46

Earlier quoted context omitted.

I think they mean there’s a 1k concurrent request limit that they hit. Though the alternative would be dedicated servers and load balancers, no?

Right, I'm referring to AWS limits. I was running a benchmark yesterday against a logging endpoint I made with a similar architecture to the article. One function is attached to a public ALB endpoint and does some validation then writes the event to SQS; this was taking 100-200ms with 128Mb of RAM. A second function was attached to the SQS queue; its job was to pull events and write them out to an external service (S…

> the 2nd function was taking up over 50% of the concurrent execution limit and new events coming into the front-end would get throttled and in this case thrown out.

It sounds like you already found a great solution for your particular case. But it's also worth mentioning that you can apply per-function concurrency limits, which can be another way to prevent a particular function from consuming too much of the overall concurrency. For anyone who's lambda workload is cheaper than a 27/7 task, that could be a good option.

> Each invocation of the Lambda was authenticating with the GCP SDK before passing the event

I'm curious whether you tried moving the authentication outside of the handler function so it could be reused for multiple events? I've found that can make a huge difference for some use cases.

Re: How we built a serverless architecture with AWS

#55

Post by vageli » Fri, Jul 12, 2019, 1:40 PM UTC

Charles_t wrote: post #8

This is not going to scale. Lambdas are hella slow. The cold starts will kill you.

> This is not going to scale. Lambdas are hella slow. The cold starts will kill you.

Frameworks like Zappa handle this out of this box by setting up executions to run via cloudwatch event cron.

Re: How we built a serverless architecture with AWS

#56

Post by closeparen » Fri, Jul 12, 2019, 2:54 PM UTC

remify wrote: post #51

closeparen wrote: post #29

Earlier quoted context omitted.

>You can setup a a CI/CD pipeline in about half an hour with amplify, at the previous company I remember it taking a good 3 weeks to get CircleCi up and running properly. >And then moving a microservice over to it is basically 1 command, a few options, mostly just copy over the config from your old express backend with a few changes, and you're done. It's insane. As a an engineer at a decent sized tech company, this…

Except now you don't need an infrastructure team. That's the whole point of serverless architecture: to be able to scale without having huge team scale as well.

You haven’t needed an infrastructure team since PHP shared hosting, and certainly not since Heroku or Elastic Beanstalk, except that people kept wanting greater complexity at lower cost. There is nothing new about “serverless” there.

Re: How we built a serverless architecture with AWS

#57

Post by notthisshitaga » Fri, Jul 12, 2019, 3:34 PM UTC

bborud wrote: post #52

I like the idea of serverless architectures, but I still wouldn't use it for anything that is important. - Using a serverless architecture almost always implied getting married to your provider. You can run your code in only one place. You have given up all bargain power. When the relationship ends you have to build your system over again. - It isn't really serverless; they're just not your servers. - They are only e…

"- Using a serverless architecture almost always implied getting married to your provider. You can run your code in only one place. You have given up all bargain power. When the relationship ends you have to build your system over again."

I think you can use a format that is mostly provider-independent. Any movement will change just the integration. Also, this kind of applies for anything large. You get married to the API anyway and movement will always be painful to some degree.

"- It isn't really serverless; they're just not your servers."

You mean they are not on prem? 'Cuz they can be. You mean that the name is bad? It really isn't as bad as people seem to imply. When done right, you don't worry about the servers. You mean that you have no control over the execution environment? Well, if spectre and meltdown have taught us anything is that really you lose control at some point anyway.

Re: How we built a serverless architecture with AWS

#58

Post by Jedi72 » Fri, Jul 12, 2019, 4:34 PM UTC

dmix wrote: post #47

Jedi72 wrote: post #43

Earlier quoted context omitted.

SSR is the future? What has PHP been doing for 15+ years?

I'm talking about Next.js/Nuxt.js style JS front-ends replacing exactly that plus JS heavy frontends like Angular and SPA react apps which was the last decade's modus operandi. The way SSR hooks React/Vue into these JS apps "hydrating" them after loading prerendered component based views...to make them interactive without losing any performance compared to static HTML, is unique and extremely powerful, which most peo…

Yo dawg I heard you like job security, so we put a program in your program, now you you can render while you render

More seriously, I do understand the difference, but disagree with the whole approach in 95% of cases

Re: How we built a serverless architecture with AWS

#59

Post by akishinevsky » Fri, Jul 12, 2019, 5:17 PM UTC

holoduke wrote: post #40

I don't understand what you really gain with this setup. I mean, this extreme vendor lock-in situation is so short term. The absolute wrong strategy if you ask me. I would be curious to see this company in 5 years from now.

Let me ask you this question differently: lets say you are exploring a new market opportunity for the an exciting product you want to build. Would you rather spend cycles building what AWS has done already and thus delay speed to market or would you rather use what has already been done by AWS managed services and build what nobody has done before? Surely, this architecture will evolve over time, but it will only evolve as the startup quickly discovers what the market needs.

Re: How we built a serverless architecture with AWS

#60

Post by akishinevsky » Fri, Jul 12, 2019, 5:27 PM UTC

remify wrote: post #51

closeparen wrote: post #29

Earlier quoted context omitted.

>You can setup a a CI/CD pipeline in about half an hour with amplify, at the previous company I remember it taking a good 3 weeks to get CircleCi up and running properly. >And then moving a microservice over to it is basically 1 command, a few options, mostly just copy over the config from your old express backend with a few changes, and you're done. It's insane. As a an engineer at a decent sized tech company, this…

Except now you don't need an infrastructure team. That's the whole point of serverless architecture: to be able to scale without having huge team scale as well.

Exactly. We were able to support millions of new devices without infrastructure involvement.