61–70 of 112 posts
https://blog.torproject.org/changes-tor-exit-list-service
While some attackers have access to compromised home routers and IoT devices, most script kiddies do not, so they use Tor (because they don't want to get caught).
A few thoughts:
1. Build a “live status” dashboard for your processing system so you can see the profile on the IPs or customers being affected, so you have more visibility into the issue when it happens
2. Can you just put the system behind cloudfront and process CF log files instead of having to invoke lambdas?
3. Identify the affected customer and put their lambda results in a separate queue, to help limit blast radius if it happens again from other customers
4. Double check there wasn’t throttling because downstream services were throttling lambda (you mentioned SQS?)
5. Make your lambda even faster (if it’s 100ms make it 10ms or less); can you move your logic out of lambda? Can you instead of inserting into SQS log to cloudwatch and then read its data? Or ensure no cross region lambda SQS invocations, etc
6. Consider replacing lambda (that cannot batch process invocations) with app servers that can absorb and then batch send many requests to SQS (let them absorb and coalesce a large amount of requests into aggregate SQS messages at the edge); going a step further eg even uploading aggregated data files to S3 for later processing then put a single message on the SQS queue pointing to that file
One simple way to stop a lot of shenanigans such as this is to block Tor exit nodes from connecting to your services. https://blog.torproject.org/changes-tor-exit-list-service While some attackers have access to compromised home routers and IoT devices, most script kiddies do not, so they use Tor (because they don't want to get caught).
Advantage comes with those who attack, not defend. Attackers will always find new ways to attack and eventually break-in. Be ready for this. As history shows, crackers (DDoS'ers) always win if the price for your head is high enough (money or fame).
Earlier quoted context omitted.
We are indeed. Neither of us enjoy DevOps so we pay a premium to not have to manage servers. It brings a huge mental health benefit (we are a two person company), we're profitable and our monthly Lambda cost isn't that significant. Honestly, the biggest inefficiency (in terms of cost) is our use of SQS/RDS, which we are ditching soon.
Well, it's your money... I'd be happy to help make this more efficient though (at no charge of course). Offhand I'd say "web server which accumulates data and uploads it to S3 every N requests or M seconds" would probably get you what you need at a tiny fraction of the cost of "lambda which posts to SQS". Create an AMI and toss it at an autoscaling group and you really won't need to worry about scaling issues either.
Fathom has a $34/month plan which handles up to 400k pageviews per month. 400k Lambda invocations is 16 cents, if they're doing it efficiently. Transfer pricing seems negligible also at only ~2KB a request.
I'm pleasantly surprised, actually, but of course when someone throws millions of improper requests at you every second.. I guess that's when the game changes ;-)
Hey as a (way smaller) competitor of yours, I honestly feel you. I understand you may not want my advice. But anyways, here it goes :) I have dealt with similar issues too at my full-time job. We ingest data in the billions range too, and our costs are a lot lower than what you describe. * On the public endpoint, can you move the first point of contact closer to the edge? You want to block requests before they even r…
It's not just a theoretical case, I saw this for many customers with Panelbear. Many JS frameworks use the history API in ways you may not be considering.
That's why I would recommend de-duplicating events on the client-side before sending them to your ingestion pipeline.
But that is DDoS matters aside.
I don't fully get how you can DDOS this service as it seems you need to give your credit card to sign up. Can't they block the API keys with issues and be done with it?
Public access point. The server accepts pageviews, so they DDoS'd that.