Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google pub/sub released an ordering feature

cloud.google.com

Post reply on HN
Print view View original

71–80 of 114 posts

Re: Google pub/sub released an ordering feature

#71

Post by zeroxfe »

byteofbits wrote: post #57

This is definitely a surprising, if welcome, development from GCP. We used to be pretty significant users of Pub/Sub but migrated to Cloud Tasks after several discussions with our account manager indicated this wasn't the direction they wanted to go with PubSub. The implementation here also seems to be somewhat unusual at first glass - in particular that a retry of any given tasks appears to also retry any subsequent…

If you're using pubsub to propagate state, then this lets you replicate with eventual consistency. (Think db replication, instant messaging, video game scores, etc.)

Re: Google pub/sub released an ordering feature

#72

Post by jlokier »

easytiger wrote: post #46
valleyjo wrote: post #32

Earlier quoted context omitted.

That won’t work in all cases. For instance, if you get messages from devices which can be reimaged they may have clock skew in a period of time before they’re synchronized again.

But in any case you can't rely on the order of message ingress to your system to represent anything meaningful either? It would have to ensure that the key for defining order would have some hard logical ordering purpose for which time is not relevant or useful.

The order of message ingress can still be meaningful even if device clocks are skew or jump due to rebooting, reimaging, network time sync, frequency drift, etc.

A hard logical order arises from interactions. E.g. if the device receives a message, does something locally, goes through a clock change, and then sends a message dependent on one it fetched earlier, that's a logical order with out-of-order clock.

Or if a device gets a message, processes, sends something to another device, that one processes too then sends another message back to the original source, there's a logical order but with three different clocks. Even if the clocks are synchronised, there will be some drift and the messages may be processed fast enough that the drift puts their timestamps out of order.

Re: Google pub/sub released an ordering feature

#73

Post by kamalaboulhosn »

manigandham wrote: post #36
drivebycomment wrote: post #19

Earlier quoted context omitted.

Seriously, I can't shake the feeling that most people who want this are likely very, very wrong and that this "feature" will turn out to be a footgun and a trap more often than a sniper rifle...

Why? It's just exposing the partitions underneath (which all horizontally scaled event/log systems use) and letting you order by the key within each partition. This is exactly like Kafk, Pulsar, AWS Kinesis, Azure Eventhubs, and many others which are all being used by thousands of applications without issue.

Actually, it's not exactly exposing the partition underneath. The ordering key is the only unit a user has to deal with: think user ID or database row primary key. Throughput is limited to 1MB/s per ordering key, but the number of ordering keys is limited only by what can be represented in a 1KB string. One can have millions of keys if desired and they don't need to be known in advance.

The system scales automatically as needed, so there is no need to think in terms of partitions or worry about repartitioning when load increases. Order is preserved through this scaling without the user having to do anything.

Disclaimer: I work on Cloud Pub/Sub and am the primary engineer on the ordering feature.

Re: Google pub/sub released an ordering feature

#74

Post by zeroxfe »

staticassertion wrote: post #69
jerf wrote: post #61

Earlier quoted context omitted.

"What's a use case where strict ordering is critically important?" In general, as the use case grows, every use case where the developers did not make explicit and careful provision for ensuring that order is not important, with quite non-trivial effort. Even a lot of systems whose developers think they have no ordering dependencies are wrong in at least one subtle way without realizing it. If you need to megascale,…

I can understand your post, but I don't quite buy the TCP thing. I don't think anyone is using TCP for ordering, they're using it because they don't want their packet dropped. I guess all of the systems I build are just built to assume no order/ or to leverage causal ordering, because that feels much easier to reason about - enforcing ordering feels really hard, and like something that a message bus can only do some…

> I can understand your post, but I don't quite buy the TCP thing. I don't think anyone is using TCP for ordering, they're using it because they don't want their packet dropped.

Think of (almost) any modern protocol built on top of TCP, and you'll see that ordering is critical. (http, smtp, telnet/ssh, etc.)

Re: Google pub/sub released an ordering feature

#75

Post by jlokier »

drivebycomment wrote: post #19
jeffbee wrote: post #9

Head of line blocking as a service!

Seriously, I can't shake the feeling that most people who want this are likely very, very wrong and that this "feature" will turn out to be a footgun and a trap more often than a sniper rifle...

Technically I agree, but I think in most cases, the alternative is the endpoints need to handle unordered messages correctly and that won't happen either.

So which is better: The queue does it, hiding subtle footguns because doing distributed systems reliably is hard; or the queue doesn't do it and the applications hanging off have failures more often (but not often enough for people to fix) because distributed systems reliability is still hard.

Re: Google pub/sub released an ordering feature

#76

Post by kamalaboulhosn »

longcommonname wrote: post #14
hulahoof wrote: post #6

"Receiving messages in order might increase latency." It would be good to know what sort of overhead we are looking at here; very interested in this feature

From a max of 200 MB/s. To max of 1MB/s for each index.

The throughput limitation is 1MB/s per ordering key. The number of unique ordering keys allowed on a topic is limited only by what can be represented by a 1KB string, so very high throughput on a topic is still possible.

Disclaimer: I work on Cloud Pub/Sub and am the primary engineer on the ordering feature.

Re: Google pub/sub released an ordering feature

#77

Post by jeffbee »

jlokier wrote: post #75
drivebycomment wrote: post #19

Earlier quoted context omitted.

Seriously, I can't shake the feeling that most people who want this are likely very, very wrong and that this "feature" will turn out to be a footgun and a trap more often than a sniper rifle...

Technically I agree, but I think in most cases, the alternative is the endpoints need to handle unordered messages correctly and that won't happen either. So which is better: The queue does it, hiding subtle footguns because doing distributed systems reliably is hard; or the queue doesn't do it and the applications hanging off have failures more often (but not often enough for people to fix) because distributed syste…

All of the uses of Kafka I have encountered in my professional career have been deployed simply to make something asynchronous, that is, kafka exists to be a low-impedance sink of things to do later in some other process. I've not run across one that actually needed those items to be processed in order. Therefore it is my impression that ordering can be a false requirement.

Re: Google pub/sub released an ordering feature

#78

Post by heavenlyblue »

zeroxfe wrote: post #74
staticassertion wrote: post #69

Earlier quoted context omitted.

I can understand your post, but I don't quite buy the TCP thing. I don't think anyone is using TCP for ordering, they're using it because they don't want their packet dropped. I guess all of the systems I build are just built to assume no order/ or to leverage causal ordering, because that feels much easier to reason about - enforcing ordering feels really hard, and like something that a message bus can only do some…

> I can understand your post, but I don't quite buy the TCP thing. I don't think anyone is using TCP for ordering, they're using it because they don't want their packet dropped. Think of (almost) any modern protocol built on top of TCP, and you'll see that ordering is critical. (http, smtp, telnet/ssh, etc.)

There's no ordering in HTTP. If you could send a whole HTTP request as a UDP packet you'd get exactly the same protocol (obviously sans WebSockets - but you could work around that).

Re: Google pub/sub released an ordering feature

#79

Post by jedmeyers »

zaphirplane wrote: post #40
jedmeyers wrote: post #24

Earlier quoted context omitted.

If events are generated by different processes you cannot really guarantee that time is exactly the same for them, unless you do something fancy to ensure that.

Interesting. The ordering here is when the event was generated or when the event entered the queue ? I think the later and so I think the examples here don’t apply without something on top and a trade off

The queue entrypoint is not always the same process either, especially in a system like Pub/Sub.

Re: Google pub/sub released an ordering feature

#80

Post by pradn »

byteofbits wrote: post #57

This is definitely a surprising, if welcome, development from GCP. We used to be pretty significant users of Pub/Sub but migrated to Cloud Tasks after several discussions with our account manager indicated this wasn't the direction they wanted to go with PubSub. The implementation here also seems to be somewhat unusual at first glass - in particular that a retry of any given tasks appears to also retry any subsequent…

The ordering keys feature supports a large number of keys (though since the throughput limit is 1 MB/sec per key, many applications shouldn't have issues scaling up on a given key).

Imagine you have an order processing system where you have to 1) write to a database 2) write to a metrics log 3) and send an email to the customer. You can publish a message with the ordering key being the user who initiated the order. This means you are guaranteed to see message 1 before 2, which is seen before 3.

You do have to account for possible message re-deliveries. In this example, you can 1) write to the database with a order's unique ID (to prevent duplicate rows) 2) be fine with duplicates for metrics since a bit of duplication is okay (or maybe you have a job later that removes duplicates offline) 3) and be okay with sending emails to customers twice (pretty harmless). You may also keep a side-cache of processed messages to reduce the processing of duplicates, but that's a bit heavy and may not be necessary.

What Cloud Pub/Sub with ordering keys gets you in this scenario is 1) durability of published messages 2) scalability across keys 3) ordering between messages in a key 4) retries in case one step fails 5) buffering in case your subscribers are slow or down 6) a fully hosted service (no dealing with your own cluster, scales automatically) 7) global availability (no need to shard your subscription by region, simplifying your app).

Disclaimer: I work on Cloud Pub/Sub, but this explanation is my own.

Post reply on HN