81–90 of 207 posts
We also use it in prod with for events, and customers who subscribe to message queues and webhooks etc.
We have our backend processing for both https://doarama.com and https://ayvri.com done in lambda. All of https://ayvri.com is built using serverless. (we'll be migrating from doarama to ayvri in the coming weeks) For our processing we handled 200k uploads in one hour on ayvri when we where building scenes for the Wings for Life event (the World's largest organized run). When just relying on serverless triggering an e…
Why was the cost high when relying on triggering from S3? Isn't lambda charged per invocation?
That time includes the amount of time to spin-up the lambda.
Ours is a long-running lambda, and we benefit from some local cache when they are running as well.
In order to handle the load, we had to extend the number of lambda's available on our account as well, so we are talking about 1000s of seconds being eaten up every second.
Cloudwatch logs have been a big game changer versus on-disk logs for us. Getting into larger clusters (and larger log files), figuring out what’s happening in log files became somewhat arduous. https://github.com/jorgebastida/awslogs for viewing / tailing / searching is a lot easier. It’s also fairly straightforward to get logs streamed through to ELK hosted within AWS, if you’re interested in that angle.
Do you guys find CloudWatch logs too slow? Even on very small apps I find searching the past few days takes minutes (5-10m), but I'm using "structured" JSON logs as well.
Yes, at my last company we needed to generate Open Graph thumbnails that composited several images together. We decided to use a serverless architecture that basically shelled out to an ImageMagick command and then pushed the thumbnail to S3 which was served via a CDN. The main problem we ran in to was a lack of processing power, but the new options from AWS solved our issues. I'd definitely do it again.
I think this gets at the core of why I think "100% serverless" isn't the right move for many projects. If you decide you're never going to boot a machine and manage it yourself, you're locked into the exact set of choices your cloud had made available for you. When your project has a need that's not covered, you're stuck. I'm not talking about vendor lock in here, purely about the reduced flexibility within a given v…
Just the opposite of serious for me. Example: there's a website for all the weather radar stations in Canada that lets you see the last two hours of 10-minute radar snapshots. I wanted a dump of them to try some ML algorithms, but even after requests and emails there simply wasn't one. So I set up a lambda to run every hour, load the website for all 31 weather stations and save all 6 images from the last hour to S3.…
The initial learning curve for running your own system is a bit higher, but I think it's well worth learning. Once you have the system up and running you start finding all sorts of uses for it. So if you need it for just one thing it might not be worth it, but as you start doing more things with it I think it really pays off.
We've shipped something simple and non-mission-critical in production (URL rewriting for ad placements). It has been pretty much set-and-forget. Last anyone had to even look at it was almost 3 years ago, and afaik it's still working (our ad sales team would be complaining loudly if it weren't). For something peripheral like that, it's nice not to have to run servers for it or devote any energy to keeping it running.…
Regarding the tooling: I hated debugging Lambdas via CloudWatch logs when using them for Hustle, it drove me to drink. I eventually got upset enough that I made a tool to stream the cloudwatch logs to the terminal and colorize, indent + nicely format json output: https://github.com/TylerBrock/saw
Have you tried anything like iopipe for helping debug lambdas?
I've been working with Lambda a lot more lately and it is not so bad... but also not great. I'm saying this cause I found it hard to have a git-first (or git ops) workflow that is good in AWS: it looks like everything is made to be changed manually. CloudFormation is a slow with some resources (if you need CloudFront it will take tens of minutes) and CodePipeline has a pretty terrible UX and user experience. CodePipeline is cheap and it works for sure, but it's not a good system for pipelines as restarting, terminating steps and getting the output of steps just don't work in a decent way (I want to see the output in the steps, not jump to CloudWatch). Pretty much every other system outside of AWS is better than that, but the integration with Lambda and APIGateway is not as good unfortunately. If you know of a better system for CI/CD with AWS Lambda outside of CodePipeline, I'd be interested to try it.
In a similar way, most of the serverless frameworks I've tried are written for a workflow that is executed from CLI which is great to start and attractive for developers but not good enough for a company that aims at full reproducibility of setups and "hands off" operations. Source code change should trigger changes in the Lambda/API Gateway setup all the time and it would be great if devs don't have to trigger changes manually.
Apart from those steps, I think Lambda is definitely promising and I see the company I'm working for right now using it more and more. The developer experience is still lacking IMO but I'm confident we'll get there at some point.
A few years ago we tried the Kinesis -> Lambda thing, but it failed during a large traffic event. This was due to the way the Kinesis poller works, and was made worse by the fact that we had two lambdas running off the same Kinesis stream.
The main issue was you can't control the Kinesis poller - meaning we ran into resource contention during this high traffic event and the iterator age fell behind quite drastically. So we abandoned it in favour of EMR + Flink + Apache Beam.
Other then that though, the S3 -> Lambda stuff works perfectly, and has been running for 3+ years with no issues.
I have spent the last year and a half building a completely serverless production service on Lambda, API Gateway, and DynamoDB (along with the standard auxiliary services like CW, SNS, Route53, S3, CF, X-Ray, etc.). It was a lot of work establishing new patterns for many of the operational aspects, particularly custom CW metrics and A/B deployments with Lambda traffic shifting, but in the end everything is set up nic…
Your setup, although appealing from an AWS ecosystem perspective, sounds like a bit expensive to me on a first reading. Of course everything depends on the specifics but Lambdas and DynamoDB are expensive at scale. I wonder how it compares cost-wise to a more traditional solution.
Just the opposite of serious for me. Example: there's a website for all the weather radar stations in Canada that lets you see the last two hours of 10-minute radar snapshots. I wanted a dump of them to try some ML algorithms, but even after requests and emails there simply wasn't one. So I set up a lambda to run every hour, load the website for all 31 weather stations and save all 6 images from the last hour to S3.…
As someobe new to serverless, is your code available to study as a pseudo tutorial?