We definitely haven’t fully embraced an event driven architecture, but we have gone all in on it where it makes sense for us. We’re processing in the order of a billion messages a day from hardware in customer’s homes, which is probably the ideal use case for event based comms. Handling devices being offline for a while becomes much simpler when the response to that can just be queuing up events and transmitting when available.
One of the key lessons we learned was that your event ingest needs to be rock solid. Put up a service, and then make it do one thing only, receive an event and throw it onto the message bus - we do this for messages from devices, but also for 3rd party services which send us webhook notifications. If ingest fails so does everything else, don’t let that happen.
The other thing I’d say is to make sure there’s a central source of truth for what services are consuming which messages. We made the mistake early on of saying services should be responsible for setting up their own subscriptions and it’s made it much more difficult to answer questions around who’s going to be impacted by changes or outages. At a minimum have a wiki page on it. Ideally manage subscriptions in Terraform or similar.
Finally, DLQs. Typically we don’t do much with them, we have logging of messages that get pushed to the DLQ and usually it’s a non-recoverable error, often around validation or accounts being disabled. They are handy in the case of an outage though as you can just push all the messages back into the queue when things recover.