Live data from Hacker News

Google pub/sub released an ordering feature

cloud.google.com

91–100 of 114 posts

Re: Google pub/sub released an ordering feature

#91
post #21
post #18

Earlier quoted context omitted.

Consider processing two events without guaranteed ordering: - Create A - Delete A In one ordering, A is created and then deleted as expected, in the other, the delete fails but then A is created and remains.

Does adding timestamps not handle this case?

For a single process on one box with one thread you can use something like that.

If you involve more than 1 box that goes out the window. Sometimes you can still get 'one timestamp' by making something else the owner of the timestamp. It also depends on your resolution of time and the process that does the ingesting. For example if that ingest process has more than one thread to handle things you can still get out of order/sametimestamp if not coded correctly.

Re: Google pub/sub released an ordering feature

#92
post #45

This kind of confuses me a bit. Why would you want to turn Pub/Sub into a queue? How badly does this affect message acknowledgment and retries? I assume just a huge hit to latency. This seems like a horrible idea for anyone expecting to use multiple subscribers or expecting to chunk multiple messages per request. Services relying on Pub/Sub should be idempotent anyway. If you need to work around that for some reason,…

Having used both types. Turning the pub/sub into a queue has some advantages in debugging and processing. Kafka has the idea of each queue being partitioned and having hash keys. Which means you can have a bunch of processes reading from the same queue and no one really steps on each other. Basically sharding at the data stream level with guaranteed ordering. It is a neat concept. Another is playback. Kafka uses a groupid/offset to keep track of where you are at. Another nice bit is messages are decently hard to lose as they stick around and you can playback by just moving the offset. The update is maybe 10 bytes into a memorybacked filestore. At first I too was skeptical of perf but it can scale very nicely and lets you scale a topic horizontally as well as vertically. In the background you have an expire time for a message. So maybe you only keep it for one week. Or you can set it to last years. For something like that you would be better off putting it in a db table though.

Idempotent is a good idea even in a system like this. But it is not always possible as your upstream data sources may be something very different.

Re: Google pub/sub released an ordering feature

#93
post #74

Earlier quoted context omitted.

> I can understand your post, but I don't quite buy the TCP thing. I don't think anyone is using TCP for ordering, they're using it because they don't want their packet dropped. Think of (almost) any modern protocol built on top of TCP, and you'll see that ordering is critical. (http, smtp, telnet/ssh, etc.)

There's no ordering in HTTP. If you could send a whole HTTP request as a UDP packet you'd get exactly the same protocol (obviously sans WebSockets - but you could work around that).

How would you process an unordered response?

Re: Google pub/sub released an ordering feature

#94

Earlier quoted context omitted.

Seriously, I can't shake the feeling that most people who want this are likely very, very wrong and that this "feature" will turn out to be a footgun and a trap more often than a sniper rifle...

Elaborate please, why is receiving messages in an ordered fashion a foot gun and a trap? Isn’t it totally acceptable to rely on this type of behavior if your use case demands it?

One way it can be a foot gun is:

To ensure ordering, systems like these have to block consumption on any given partition until it gets an ACK. If publishers of the data don't understand that, they can publish with too coarse a partition key which greatly decreases the amount of concurrent consumption you can have.

Not knocking how these systems work, it's more that people generally don't "read the fine print", especially these days with cloud services.

Re: Google pub/sub released an ordering feature

#95
post #61

Earlier quoted context omitted.

"What's a use case where strict ordering is critically important?" In general, as the use case grows, every use case where the developers did not make explicit and careful provision for ensuring that order is not important, with quite non-trivial effort. Even a lot of systems whose developers think they have no ordering dependencies are wrong in at least one subtle way without realizing it. If you need to megascale,…

I can understand your post, but I don't quite buy the TCP thing. I don't think anyone is using TCP for ordering, they're using it because they don't want their packet dropped. I guess all of the systems I build are just built to assume no order/ or to leverage causal ordering, because that feels much easier to reason about - enforcing ordering feels really hard, and like something that a message bus can only do some…

[deleted]

Re: Google pub/sub released an ordering feature

#96
post #4

Earlier quoted context omitted.

We used Pub/Sub very extensively (50B messages a day) but moved to Pulsar [0]. It performs equally well and has some nice features. And also no vendor lock-in. [0] https://kesque.com/billions-of-events-a-day-without-breaking...

Pulsar seems operationally quite complex, as it has a dependency on both BookKeeper and ZooKeeper (which BK also needs). ZooKeeper is particularly notorious for being difficult. What's your experience been like?

It definitely is on the more complex side of management. That's why we partnered with Kafkaesque to do the maintenance for us. We were fine handling it ourselves but decided to outsource it as it's less critical for us than many other internal tasks.

They have an open ticket [0] to dilute zookeeper's dependance, but as far as I know it's still pending.

[0] https://github.com/apache/pulsar/issues/572

Re: Google pub/sub released an ordering feature

#97
post #72

Earlier quoted context omitted.

But in any case you can't rely on the order of message ingress to your system to represent anything meaningful either? It would have to ensure that the key for defining order would have some hard logical ordering purpose for which time is not relevant or useful.

The order of message ingress can still be meaningful even if device clocks are skew or jump due to rebooting, reimaging, network time sync, frequency drift, etc. A hard logical order arises from interactions. E.g. if the device receives a message, does something locally, goes through a clock change, and then sends a message dependent on one it fetched earlier, that's a logical order with out-of-order clock. Or if a d…

That guarantee/assumption can never really be made.

Message A on event a' from system a might be sent to system b effecting event b' and thus message B to be sent by , and consumed and correlated by consumer software MC on hardware mc.

However system B might take longer to flush it's hardware/software buffer and the message arrives at mc before message A, for example.

I've encountered this many times. That data has no meaning in itself except in the meta.

> Even if the clocks are synchronised, there will be some drift and the messages may be processed fast enough that the drift puts their timestamps out of order.

If you are consuming from sources which you cannot control the accuracy of the clocks, then you must inherently either reduce your reliance on the need for accuracy (many Windows systems have horrendous clock discipline in their software implementation) or find a way of ensuring accuracy. E.g. close proximity NTP or close proximity PTP etc etc.

Hope that makes sense.

Re: Google pub/sub released an ordering feature

#98
post #83

Earlier quoted context omitted.

Any plans for permanent message retention? Kafka can be configured to store messages forever. This means it can be used as the canonical store of both current and past data. It is attractive for some applications. Any plans to add the ability to retain messages forever? I know I can store them in GCS and replay, but that's not what I'm looking for.

I can see how this would be useful. You could use the snapshot/seek features to replay messages from a particular point. It'd be useful retain messages for a long time so you could replay messages from any point. I could even imagine a feature for replaying back messages from only a time range. We're also aware of requests for more seamless integration with other GCP services; they may augment or provide a similar se…

That makes sense. To add some color to the use case:

We'd like to use a pub sub system to store binlogs from databases (as done in projects like https://debezium.io/).

We'd like any team to be able to bring a replica database online by starting from the beginning of time and playing back the binlogs. And then to keep playing any future binlog messages to keep the replica current.

For usage like this, we could in theory keep all past messages on GCS, and then access the old messages on GCS and then when those are up to date, get the messages from pubsub. Or we could keep them on GCS and replay them on pubsub in the future if we wanted to.

But in a perfect world, I'd prefer the pubsub system to handle this, and allow a consumer to start at the beginning of time and get up to date, and stay up to date in a simple way.

Re: Google pub/sub released an ordering feature

#99
This looks like a naive implementation of adding ordering keys on one side, and waiting until all messages arrive on the other in order, without improving the underlying delivery transport to support ordering.

From what I can tell, the publisher is responsible for now maintaining arbitrary ordering IDs and passing this in the publish method, see https://github.com/googleapis/nodejs-pubsub/blob/master/samp....

Given the documentation now clearly states "Publishing messages with ordering keys might increase latency", I think that what Google is doing is effectively doing is putting lipstick on their service that does not support ordering and instead offering a message reconstruction capability on the subscriber end.

This approach, whilst it may work a lot of the time, is pretty flawed given one failed published message could cause the entire stream to stall indefinitely. And given Google Pub/Sub does not support idempotency, and does not support exactly-once delivery, they recommend "In general, accommodating more-than-once delivery requires your subscriber to be idempotent when processing messages."

Google, I am afraid whilst it's nice to welcome you to the exactly-once semantic party, I think you may need to go back to the drawing board and bake ordering and idempotency into the transport layer, in the same way Kafka and Ably do so that true ordering is supported.

Disclaimed: I am the co-founder of ably.com, a far edge enterprise messaging solution, with exactly-once semantics and real ordering :)

Re: Google pub/sub released an ordering feature

#100
post #72

Earlier quoted context omitted.

The order of message ingress can still be meaningful even if device clocks are skew or jump due to rebooting, reimaging, network time sync, frequency drift, etc. A hard logical order arises from interactions. E.g. if the device receives a message, does something locally, goes through a clock change, and then sends a message dependent on one it fetched earlier, that's a logical order with out-of-order clock. Or if a d…

That guarantee/assumption can never really be made. Message A on event a' from system a might be sent to system b effecting event b' and thus message B to be sent by , and consumed and correlated by consumer software MC on hardware mc. However system B might take longer to flush it's hardware/software buffer and the message arrives at mc before message A, for example. I've encountered this many times. That data has n…

I think you and I are agreeing, but it's not obvious ;-)

> However system B might take longer to flush it's hardware/software buffer and the message arrives at mc before message A, for example.

There are two message As in your system, the one sent to system b, and the one consumed by hardware mc. Let's call them Ab and Amc.

In that situation, message B is a consequence of message Ab which can be tracked, and at system mc (depending on semantics) it might be necessary to use that tracking to process message Amc before message B, at least logically.

For example message Ab/Amc might be "add new user Foo the our database with standard policy", and system B might react with "ok, my job is to set the policy for new users, I'll tell mc to add permission Bar to user Foo then activate Foo".

That works out fine as long as the logical order relation is maintained, and system mc, the database, processes message Amc first, regardless of arrival order.

Dependency tracking can ensure that (without clocks or timestamps), even if messages are transmitted independently. For example by message B containing a header that says it comes logically after message A.

The pubsub queue can also guarantee that order (without clocks or timestamps or dependency tracking), provided all messages go through the same pubsub system, and Ab+Amc are fanned-out by the pubsub system rather than sent independently by A to each destination. All bets are off if Ab and Amc take other routes.

> If you are consuming from sources which you cannot control the accuracy of the clocks, then you must inherently either reduce your reliance on the need for accuracy (many Windows systems have horrendous clock discipline in their software implementation) or find a way of ensuring accuracy. E.g. close proximity NTP or close proximity PTP etc etc.

If you think Windows is bad, try just about any cloud VM, which has a virtual clock and is stalled all the time in the guest, including just after the guest reads its virtual clock and before using the value :-)

I prefer systems which rely on logical ordering guarantees as much as possible, so clock drift doesn't matter.

When you are rely on a global clock order to ensure correct behaviour, you have to slow down some operations to accommodate for clock variance across the system, as well as network latency variance (because it affects clock synchronisation).

If you rely on logical order, then there's no need for time delays; no upper speed limit. Instead you have to keep track of dependencies, or have implicit causal dependencies. And it's more robust on real systems because clock drift and jumps don't matter.

In practice you need some clock dependence for timeouts anyway, and there are protocol advantages when you can depend on a well synchronised clock. So I prefer to mix the two for different layers of the protocol, to get advantages of both. Unfortunately for correct behaviour, timeouts are often the "edge case" that isn't well tested or correctly implemented at endpoints.

Post reply on HN