Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google pub/sub released an ordering feature

cloud.google.com

Post reply on HN
Print view View original

81–90 of 114 posts

Re: Google pub/sub released an ordering feature

#81

Post by whatshisface »

staticassertion wrote: post #69
jerf wrote: post #61

Earlier quoted context omitted.

"What's a use case where strict ordering is critically important?" In general, as the use case grows, every use case where the developers did not make explicit and careful provision for ensuring that order is not important, with quite non-trivial effort. Even a lot of systems whose developers think they have no ordering dependencies are wrong in at least one subtle way without realizing it. If you need to megascale,…

I can understand your post, but I don't quite buy the TCP thing. I don't think anyone is using TCP for ordering, they're using it because they don't want their packet dropped. I guess all of the systems I build are just built to assume no order/ or to leverage causal ordering, because that feels much easier to reason about - enforcing ordering feels really hard, and like something that a message bus can only do some…

I'm not sure that ordering doesn't matter for most TCP data. For example, HTTP depends on ordering. Any time you are transmitting messages larger than the size of a packet, you need some degree of ordering, even if it's only to reconstruct the individual messages, when you don't care about the order of the messages.

Re: Google pub/sub released an ordering feature

#82

Post by manigandham »

kamalaboulhosn wrote: post #73
manigandham wrote: post #36

Earlier quoted context omitted.

Why? It's just exposing the partitions underneath (which all horizontally scaled event/log systems use) and letting you order by the key within each partition. This is exactly like Kafk, Pulsar, AWS Kinesis, Azure Eventhubs, and many others which are all being used by thousands of applications without issue.

Actually, it's not exactly exposing the partition underneath. The ordering key is the only unit a user has to deal with: think user ID or database row primary key. Throughput is limited to 1MB/s per ordering key, but the number of ordering keys is limited only by what can be represented in a 1KB string. One can have millions of keys if desired and they don't need to be known in advance. The system scales automaticall…

It's hashing the ordering key to map to a partition right?

That's what I meant, access to ordering via partitions but without the full addressing of named partitions like Kafka.

Re: Google pub/sub released an ordering feature

#83

Post by pradn »

andrewmutz wrote: post #62
hongalex wrote: post #10

Earlier quoted context omitted.

By the way, in addition to ordered messages, Pub/Sub has recently gotten a number of new features including message filtering, dead letter queues, and retry policy (some GA, some beta). You can find out more here: https://cloud.google.com/pubsub/docs/release-notes Disclaimer: I work on Pub/Sub

Any plans for permanent message retention? Kafka can be configured to store messages forever. This means it can be used as the canonical store of both current and past data. It is attractive for some applications. Any plans to add the ability to retain messages forever? I know I can store them in GCS and replay, but that's not what I'm looking for.

I can see how this would be useful. You could use the snapshot/seek features to replay messages from a particular point. It'd be useful retain messages for a long time so you could replay messages from any point. I could even imagine a feature for replaying back messages from only a time range. We're also aware of requests for more seamless integration with other GCP services; they may augment or provide a similar set of features. I can't comment on timelines unfortunately. :)

Disclaimer: I work on Cloud Pub/Sub, but this is my own opinion.

Re: Google pub/sub released an ordering feature

#84

Post by pradn »

ldf1111 wrote: post #43

How does this feature work? is ordering key like kinesis partition Key. Does it somehow assign an Id to each subscriber and make sure that all messages for a key are sent to a single subscriber?

CPS doesn't use a notion of partitions under the covers like Kinesis. Each ordering key can be thought of constituting a "virtual" partition. You can have as many keys as you'd like, without specifying them ahead of time.

Disclaimer: I work on Cloud Pub/Sub, but this is my own explanation.

Re: Google pub/sub released an ordering feature

#85

Post by pradn »

deanCommie wrote: post #39

What's the TPS rate limit?

As my colleague, kamalaboulhosn, posted elsewhere in this thread:

"The throughput limitation is 1MB/s per ordering key. The number of unique ordering keys allowed on a topic is limited only by what can be represented by a 1KB string, so very high throughput on a topic is still possible."

So, it's more of a throughput limit, not a message limit, per key. With small messages, you can get high "TPS".

Disclaimer: I work on Cloud Pub/Sub.

Re: Google pub/sub released an ordering feature

#86

Post by duijf »

heavenlyblue wrote: post #78
zeroxfe wrote: post #74

Earlier quoted context omitted.

> I can understand your post, but I don't quite buy the TCP thing. I don't think anyone is using TCP for ordering, they're using it because they don't want their packet dropped. Think of (almost) any modern protocol built on top of TCP, and you'll see that ordering is critical. (http, smtp, telnet/ssh, etc.)

There's no ordering in HTTP. If you could send a whole HTTP request as a UDP packet you'd get exactly the same protocol (obviously sans WebSockets - but you could work around that).

There are numerous valid HTTP payloads that are larger than a single TCP/UDP packet.

Ordering is important in these cases.

Re: Google pub/sub released an ordering feature

#87

Post by kamalaboulhosn »

manigandham wrote: post #82
kamalaboulhosn wrote: post #73

Earlier quoted context omitted.

Actually, it's not exactly exposing the partition underneath. The ordering key is the only unit a user has to deal with: think user ID or database row primary key. Throughput is limited to 1MB/s per ordering key, but the number of ordering keys is limited only by what can be represented in a 1KB string. One can have millions of keys if desired and they don't need to be known in advance. The system scales automaticall…

It's hashing the ordering key to map to a partition right? That's what I meant, access to ordering via partitions but without the full addressing of named partitions like Kafka.

There is a mapping from ordering key to internal storage structures, yes. The goal is that users are abstracted away from any notion of partition and the system handles scaling and ordering on its own, e.g., when there are hot shards.

Re: Google pub/sub released an ordering feature

#88

Post by pradn »

camgunz wrote: post #27

Is this just a buffer? Feels like any MQ could offer this within a given latency/timeout window. I’m pretty naive about this though, is it more involved?

Cloud Pub/Sub offers global topics/subscriptions that scale automatically and are completely managed for you (no dealing with your own cluster). With the ordering keys feature, there's no need to specify the number or ID of the ordering keys ahead of time (as one may need to do for traditional message queue partitions). In general, there's no provisioning ahead of time, so topics/subscriptions will scale as your usage grows. These are a few of the properties Cloud Pub/Sub offers compared to traditional self-hosted message queues.

Disclaimer: I work on the Cloud Pub/Sub team, but this explanation is my own.

Re: Google pub/sub released an ordering feature

#90

Post by staticassertion »

zeroxfe wrote: post #74
staticassertion wrote: post #69

Earlier quoted context omitted.

I can understand your post, but I don't quite buy the TCP thing. I don't think anyone is using TCP for ordering, they're using it because they don't want their packet dropped. I guess all of the systems I build are just built to assume no order/ or to leverage causal ordering, because that feels much easier to reason about - enforcing ordering feels really hard, and like something that a message bus can only do some…

> I can understand your post, but I don't quite buy the TCP thing. I don't think anyone is using TCP for ordering, they're using it because they don't want their packet dropped. Think of (almost) any modern protocol built on top of TCP, and you'll see that ordering is critical. (http, smtp, telnet/ssh, etc.)

Gotcha, ok so specifically protocols built on TCP.
Post reply on HN