Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google pub/sub released an ordering feature

cloud.google.com

Post reply on HN
Print view View original

101–110 of 114 posts

Re: Google pub/sub released an ordering feature

#101

Post by jganetsk »

ldf1111 wrote: post #43

How does this feature work? is ordering key like kinesis partition Key. Does it somehow assign an Id to each subscriber and make sure that all messages for a key are sent to a single subscriber?

Yes, the idea is each ordering key is assigned to a single subscriber client, and all messages for that key are sent in order to that client. There is effectively a cursor which is advanced by acking. When we reassign the key to another client, we replay all messages starting from the cursor.

Disclaimer: I work on Cloud Pub/Sub

Re: Google pub/sub released an ordering feature

#102

Post by hongalex »

matt_oriordan wrote: post #99

This looks like a naive implementation of adding ordering keys on one side, and waiting until all messages arrive on the other in order, without improving the underlying delivery transport to support ordering. From what I can tell, the publisher is responsible for now maintaining arbitrary ordering IDs and passing this in the publish method, see https://github.com/googleapis/nodejs-pubsub/blob/master/samp... . Given…

> the publisher is responsible for now maintaining arbitrary ordering IDs and passing this in the publish method

Ah, thanks for pointing this out. The "ordering ID" bit is actually a remnant of how to publish messages before we had officially ordering key support (and only shows up in Node, but not our other samples https://cloud.google.com/pubsub/docs/publisher#using_orderin...). We'll be fixing this sample shortly.

Disclaimer: I work on Cloud Pub/Sub. This explanation is my own.

Re: Google pub/sub released an ordering feature

#103

Post by pradn »

andrewmutz wrote: post #98
pradn wrote: post #83

Earlier quoted context omitted.

I can see how this would be useful. You could use the snapshot/seek features to replay messages from a particular point. It'd be useful retain messages for a long time so you could replay messages from any point. I could even imagine a feature for replaying back messages from only a time range. We're also aware of requests for more seamless integration with other GCP services; they may augment or provide a similar se…

That makes sense. To add some color to the use case: We'd like to use a pub sub system to store binlogs from databases (as done in projects like https://debezium.io/ ). We'd like any team to be able to bring a replica database online by starting from the beginning of time and playing back the binlogs. And then to keep playing any future binlog messages to keep the replica current. For usage like this, we could in the…

Thanks for your explanation. It makes a lot of sense. :)

Re: Google pub/sub released an ordering feature

#104

Post by camgunz »

pradn wrote: post #88
camgunz wrote: post #27

Is this just a buffer? Feels like any MQ could offer this within a given latency/timeout window. I’m pretty naive about this though, is it more involved?

Cloud Pub/Sub offers global topics/subscriptions that scale automatically and are completely managed for you (no dealing with your own cluster). With the ordering keys feature, there's no need to specify the number or ID of the ordering keys ahead of time (as one may need to do for traditional message queue partitions). In general, there's no provisioning ahead of time, so topics/subscriptions will scale as your usag…

Hey thanks for your answer.

I'm less concerned about scale or not managing my own cluster, and more about the mechanics of how this is implemented. Is it something like:

- User configures an ordering key

- Application sends a message w/ ordering key "4"

- Pub/Sub gets message w/ ordering key "4"

- Pub/Sub buffers "4" until it gets a message > "4"

- Application sends a message w/ ordering key "13"

- Pub/Sub sends "4"

- Pub/Sub buffers "13" until it gets a message > "13"

I just thought that the lack of order was in inherent tension with performance because you need a buffer to order things. Am I wrong about that (probably)?

Re: Google pub/sub released an ordering feature

#105

Post by pradn »

camgunz wrote: post #104
pradn wrote: post #88

Earlier quoted context omitted.

Cloud Pub/Sub offers global topics/subscriptions that scale automatically and are completely managed for you (no dealing with your own cluster). With the ordering keys feature, there's no need to specify the number or ID of the ordering keys ahead of time (as one may need to do for traditional message queue partitions). In general, there's no provisioning ahead of time, so topics/subscriptions will scale as your usag…

Hey thanks for your answer. I'm less concerned about scale or not managing my own cluster, and more about the mechanics of how this is implemented. Is it something like: - User configures an ordering key - Application sends a message w/ ordering key "4" - Pub/Sub gets message w/ ordering key "4" - Pub/Sub buffers "4" until it gets a message > "4" - Application sends a message w/ ordering key "13" - Pub/Sub sends "4"…

It's more like

1) Application sends a message w/ ordering key "abc" (let's call it msg1) 2) Pub/Sub gets msg1. 3) Pub/Sub acknowledges that msg1 was received. Now, you can be secure we have stored this message durably. 4) Pub/Sub sends msg1 to the subscriber ASAP if there's one available. 5) A subscriber client assigned to ordering key "abc" gets msg1. 6) The subscriber client processes msg1 and sends an ack to Pub/Sub. 7) Pub/Sub receives the ack and removes the message from storage. (It's possible to retain acked messages, but let's not talk about that for simplicity.)

Another message for the same ordering key "abc" arriving at Pub/Sub any time after step 4 will be delivered to the subscriber only after msg1. To summarize, the timestamp of a successful publish response is the timestamp by which we order messages sent to the client.

Pub/Sub may not send messages immediately if the subscriber is too slow to ack the message (perhaps each message takes a while to process) or if the subscriber is down or if there's too many messages in the backlog for the ordering key. In these cases, you can be assured that they're safely stored on disk and will be resent later (after some backoff period.)

So, to answer your question directly, we only stall the sending of ordered messages to subscribers if there's a backlog for that ordering key or if the subscriber not fast enough.

Re: Google pub/sub released an ordering feature

#106

Post by manigandham »

camgunz wrote: post #104
pradn wrote: post #88

Earlier quoted context omitted.

Cloud Pub/Sub offers global topics/subscriptions that scale automatically and are completely managed for you (no dealing with your own cluster). With the ordering keys feature, there's no need to specify the number or ID of the ordering keys ahead of time (as one may need to do for traditional message queue partitions). In general, there's no provisioning ahead of time, so topics/subscriptions will scale as your usag…

Hey thanks for your answer. I'm less concerned about scale or not managing my own cluster, and more about the mechanics of how this is implemented. Is it something like: - User configures an ordering key - Application sends a message w/ ordering key "4" - Pub/Sub gets message w/ ordering key "4" - Pub/Sub buffers "4" until it gets a message > "4" - Application sends a message w/ ordering key "13" - Pub/Sub sends "4"…

A better name would be partioning or shard key. It's not physically ordering the items based on the key, just hashing them into a partition. The physical time they are received by the service is the order they're stored and returned as.

Re: Google pub/sub released an ordering feature

#107

Post by camgunz »

pradn wrote: post #105
camgunz wrote: post #104

Earlier quoted context omitted.

Hey thanks for your answer. I'm less concerned about scale or not managing my own cluster, and more about the mechanics of how this is implemented. Is it something like: - User configures an ordering key - Application sends a message w/ ordering key "4" - Pub/Sub gets message w/ ordering key "4" - Pub/Sub buffers "4" until it gets a message > "4" - Application sends a message w/ ordering key "13" - Pub/Sub sends "4"…

It's more like 1) Application sends a message w/ ordering key "abc" (let's call it msg1) 2) Pub/Sub gets msg1. 3) Pub/Sub acknowledges that msg1 was received. Now, you can be secure we have stored this message durably. 4) Pub/Sub sends msg1 to the subscriber ASAP if there's one available. 5) A subscriber client assigned to ordering key "abc" gets msg1. 6) The subscriber client processes msg1 and sends an ack to Pub/S…

Thank you so much for this answer, really appreciate it.

Re: Google pub/sub released an ordering feature

#108

Post by pradn »

camgunz wrote: post #107
pradn wrote: post #105

Earlier quoted context omitted.

It's more like 1) Application sends a message w/ ordering key "abc" (let's call it msg1) 2) Pub/Sub gets msg1. 3) Pub/Sub acknowledges that msg1 was received. Now, you can be secure we have stored this message durably. 4) Pub/Sub sends msg1 to the subscriber ASAP if there's one available. 5) A subscriber client assigned to ordering key "abc" gets msg1. 6) The subscriber client processes msg1 and sends an ack to Pub/S…

Thank you so much for this answer, really appreciate it.

You're welcome!

Re: Google pub/sub released an ordering feature

#109

Post by heavenlyblue »

duijf wrote: post #86
heavenlyblue wrote: post #78

Earlier quoted context omitted.

There's no ordering in HTTP. If you could send a whole HTTP request as a UDP packet you'd get exactly the same protocol (obviously sans WebSockets - but you could work around that).

There are numerous valid HTTP payloads that are larger than a single TCP/UDP packet. Ordering is important in these cases.

I think I made a point that “if you could fit the whole HTTP request in one UDP packet”, have you read my message?

Re: Google pub/sub released an ordering feature

#110

Post by vp8989 »

jeffbee wrote: post #77
jlokier wrote: post #75

Earlier quoted context omitted.

Technically I agree, but I think in most cases, the alternative is the endpoints need to handle unordered messages correctly and that won't happen either. So which is better: The queue does it, hiding subtle footguns because doing distributed systems reliably is hard; or the queue doesn't do it and the applications hanging off have failures more often (but not often enough for people to fix) because distributed syste…

All of the uses of Kafka I have encountered in my professional career have been deployed simply to make something asynchronous, that is, kafka exists to be a low-impedance sink of things to do later in some other process. I've not run across one that actually needed those items to be processed in order. Therefore it is my impression that ordering can be a false requirement.

Agreed, also "processing in order" != "consuming from the sink in order". If everything upstream and downstream of the sink doesn't also happen in order then using an ordered sink is pointless.
Post reply on HN