Is this then comparable with an SNS fifo queue ?
AWS SNS doesn't support FIFO queues, but SQS does.
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQS...
Google pub/sub released an ordering feature
51–60 of 114 posts
Re: Google pub/sub released an ordering feature
#52Head of line blocking as a service!
Seriously, I can't shake the feeling that most people who want this are likely very, very wrong and that this "feature" will turn out to be a footgun and a trap more often than a sniper rifle...
Elaborate please, why is receiving messages in an ordered fashion a foot gun and a trap? Isn’t it totally acceptable to rely on this type of behavior if your use case demands it?
Re: Google pub/sub released an ordering feature
#53Earlier quoted context omitted.
Given cloud vendors low respect for their customers, the proper stance is to own your own infrastructure.
The company I work for had the same stance 5 years ago. We regretted it a hundred times and now we stand the gaff. Nothing better than maintaining an EC2 based Cassandra cluster instead of simple using DynamoDB, huh...
I agree with this, the best parts of the cloud is abstracting away a lot of the basic maintenance of these applications. There is lock in to some platform no matter what you do, but hopefully you can design your application so that if you do need to rearchitect you can do it in phases.
Re: Google pub/sub released an ordering feature
#54If anyone is looking for a solid alternative, check out Solace PubSub+. Been around for more than a decade and allows you to be cloud agnostic.
https://solace.com/products/event-broker/software/
(Can use in prod for free as well)
Re: Google pub/sub released an ordering feature
#55Earlier quoted context omitted.
Seriously, I can't shake the feeling that most people who want this are likely very, very wrong and that this "feature" will turn out to be a footgun and a trap more often than a sniper rifle...
Elaborate please, why is receiving messages in an ordered fashion a foot gun and a trap? Isn’t it totally acceptable to rely on this type of behavior if your use case demands it?
Because this will break in situations where you have multiple consumers or if there's any sort of redelivery/failure scenario.
Ordering only works if there's a single-threaded consumer somewhere that processes exactly one message at a time and blocks processing as soon as there is an error of any kind. The amount of scenarios that this works for is probably very small.
The footgun is that people don't realize the above constraints and assume that ordering will magically fix a lot of ordering issues (some more examples of this in the rest of this comments section).
Re: Google pub/sub released an ordering feature
#56Earlier quoted context omitted.
Does adding timestamps not handle this case?
> Does adding timestamps not handle this case? If you have one message source (a single thread or some kind of coordination), and the messages have lower frequency than the timestamp resolution, yes. The farther you get from that, the more the answer is no.
Just make the source part of the timestamp?
(1,A) Also use serial numbers instead of true timestamps.
Re: Google pub/sub released an ordering feature
#57This is definitely a surprising, if welcome, development from GCP. We used to be pretty significant users of Pub/Sub but migrated to Cloud Tasks after several discussions with our account manager indicated this wasn't the direction they wanted to go with PubSub.
The implementation here also seems to be somewhat unusual at first glass - in particular that a retry of any given tasks appears to also retry any subsequent message with the same "ordering key".
I do wonder what use cases this is targeted that wouldn't require pretty extensive work on the application side to ensure good idempotency. Does anyone have any ideas of problems this would solve in and of itself?
Re: Google pub/sub released an ordering feature
#58Whoa! This is a game changer! I was looking into both Kafka and Google Pub/Sub for a event-oriented system my team was designing. Google Pub/Sub looked very promising, but no guarantee of ordering was a deal breaker for us. I’ll consider this more strongly for the next system we build.
If you are still looking, I would recommend Solace PubSub+: https://solace.com/products/event-broker/software/
Supports zero message loss, no headaches around topic partitions, in-order messaging, support for open apis/protocols, in-memory AND persistent quality of service, support for event mesh etc
Re: Google pub/sub released an ordering feature
#59Whoa! This is a game changer! I was looking into both Kafka and Google Pub/Sub for a event-oriented system my team was designing. Google Pub/Sub looked very promising, but no guarantee of ordering was a deal breaker for us. I’ll consider this more strongly for the next system we build.
What's a use case where strict ordering is critically important?
Financial transactions (withdrawing/depositing money or placing stock orders)
Credit card/payment usecases
Re: Google pub/sub released an ordering feature
#60This kind of confuses me a bit. Why would you want to turn Pub/Sub into a queue? How badly does this affect message acknowledgment and retries? I assume just a huge hit to latency. This seems like a horrible idea for anyone expecting to use multiple subscribers or expecting to chunk multiple messages per request. Services relying on Pub/Sub should be idempotent anyway. If you need to work around that for some reason,…
> Services relying on Pub/Sub should be idempotent anyway. If you need to work around that for some reason, you are better off dumping messages from your subscriber into RabbotMQ or Redis for processing and use a Subscriber/Scheduler/Worker pattern.
Exactly. Pub/Sub is made to send messages to any consumer. To bring in order you direct messages to a specific consumer. Alright well Pub/Sub wasn't the right tool then.