Someone attacked our company

As a customer of both, everything about the way Netlify operates seems half-assed.

I would say this goes beyond emotional. You only have finite resources. You have chosen to spend some money to avoid spending time and effort. That's a very reasonable call. Businesses do this all the time when they hire more staff.

I fully believe this was a DDOS attack. But, this is good advice. If you think you're getting DDOS'd, check for the number of unique IPs against your own user pool. Probably a lot more people introduce bugs into their codebase that DDOS's their own app than get repeatedly targeted by a well-funded attacker as above.

Loved this post, thanks for the detailed write-up. As a solo-SaaS-founder I have a persistent, but very mild anxiety of something like this happening. The attacks we've received have been comparably mild and easy to mitigate.

Boggles my mind who the heck thinks Fathom is valuable enough to target with such a persistent series of attacks.

Hey as a (way smaller) competitor of yours, I honestly feel you.

I understand you may not want my advice. But anyways, here it goes :)

I have dealt with similar issues too at my full-time job. We ingest data in the billions range too, and our costs are a lot lower than what you describe.

* On the public endpoint, can you move the first point of contact closer to the edge? You want to block requests before they even reach your ingestion pipeline. If not, add a few geographical Load Balancers with the sole job of accepting requests before forwarding them to your Lambda. Divert traffic at the DNS level using a geo routing policy on Route53.

* Are you also hitting DynamoDB for every request as part of your spam system? I found it quite expensive on-demand, unless you provision capacity. We used to pay tens of thousands per month for DynamoDB in the past, when a single Redis instance would have done it. Specially if it's temporary, non-critical data such as spam detection.

* Do you batch the incoming events before triggering SQS?

* How are your networking costs going? They tend to creep up in the AWS bills too.

It just sounds to me you clearly need something in-between the public endpoint and those Lambdas. But maybe I'm missing some info. Otherwise you're going to keep paying for capacity that a "traditional" server could handle, without costing you more for each request.

If you can't fight back, and don't want real requests to be lost, put a server at the battle-front, and buffer all raw HTTP requests into a queue made for this (like Kinesis). You can then process events and recover the data at your own pace. I did this in the past, and could handle 60k+ req/s with 2 instances on AWS. I used a simple Go tool to capture the traffic https://github.com/buger/goreplay

Also, at my full-time job we use Kinesis to buffer all analytics events, it's cheaper than SQS and handles billions of data points per month. This also keeps the ingest rate constant, but you seem to already do that with your workers.

Full disclosure: I'm Anthony from http://panelbear.com/ , and I just want to offer honest help.