Live data from Hacker News

Google pub/sub released an ordering feature

cloud.google.com

61–70 of 114 posts

Re: Google pub/sub released an ordering feature

#61
post #2

Whoa! This is a game changer! I was looking into both Kafka and Google Pub/Sub for a event-oriented system my team was designing. Google Pub/Sub looked very promising, but no guarantee of ordering was a deal breaker for us. I’ll consider this more strongly for the next system we build.

What's a use case where strict ordering is critically important?

"What's a use case where strict ordering is critically important?"

In general, as the use case grows, every use case where the developers did not make explicit and careful provision for ensuring that order is not important, with quite non-trivial effort.

Even a lot of systems whose developers think they have no ordering dependencies are wrong in at least one subtle way without realizing it.

If you need to megascale, you're going to have to bite the bullet and build a system that can handle out-of-order, but there's a lot of systems out there where you don't need megascale, and you can get rid of that "quite non-trivial effort" to deal with out-of-orderness by asking for messages to arrive in order.

To get a sense of just how useful that can be... bear in mind that every time you open a TCP socket instead of a UDP one, you just made exactly that choice, to use an ordered message system when you didn't "need" one. Take a look at everything you do with a TCP socket and think about trying to run it over UDP, and not with something like QUIC that basically adds half of TCP back on it, but with UDP straight-up. That's what kind of things can use in-order delivery... lots of things.

Almost everything can be simplified by guaranteed in-order delivery. It's just that some things can't afford the downsides.

Re: Google pub/sub released an ordering feature

#62

I'm floored by this. Last I spoke with that team the response was a few (but not everyone) in leadership getting it and everyone else asking why we would care. How far to come. GCP just got substantially more interesting. Edit: by this I only mean to say well done to the Google team.

By the way, in addition to ordered messages, Pub/Sub has recently gotten a number of new features including message filtering, dead letter queues, and retry policy (some GA, some beta). You can find out more here: https://cloud.google.com/pubsub/docs/release-notes Disclaimer: I work on Pub/Sub

Any plans for permanent message retention?

Kafka can be configured to store messages forever. This means it can be used as the canonical store of both current and past data. It is attractive for some applications.

Any plans to add the ability to retain messages forever? I know I can store them in GCS and replay, but that's not what I'm looking for.

Re: Google pub/sub released an ordering feature

#63
post #56

Earlier quoted context omitted.

> Does adding timestamps not handle this case? If you have one message source (a single thread or some kind of coordination), and the messages have lower frequency than the timestamp resolution, yes. The farther you get from that, the more the answer is no.

Just make the source part of the timestamp? (1,A) Also use serial numbers instead of true timestamps.

That way lies madness, and/or eventually accidentally writing your own Dynamo-paper database.

Re: Google pub/sub released an ordering feature

#64
post #2

Whoa! This is a game changer! I was looking into both Kafka and Google Pub/Sub for a event-oriented system my team was designing. Google Pub/Sub looked very promising, but no guarantee of ordering was a deal breaker for us. I’ll consider this more strongly for the next system we build.

What's a use case where strict ordering is critically important?

[deleted]

Re: Google pub/sub released an ordering feature

#65
post #2

Whoa! This is a game changer! I was looking into both Kafka and Google Pub/Sub for a event-oriented system my team was designing. Google Pub/Sub looked very promising, but no guarantee of ordering was a deal breaker for us. I’ll consider this more strongly for the next system we build.

What's a use case where strict ordering is critically important?

every use case where you haven't proven it isn't.

human conscious thought is local and single-threaded. it takes a lot of experience and training to be able to intuitively reason about non-local multi-threaded computation. if you're smart and humble you can try to simplify the problem by making individual messages independent from each other by e.g. employing redundancy but you still have to be aware that it's even a problem to begin with.

Re: Google pub/sub released an ordering feature

#66
post #4
post #2

Whoa! This is a game changer! I was looking into both Kafka and Google Pub/Sub for a event-oriented system my team was designing. Google Pub/Sub looked very promising, but no guarantee of ordering was a deal breaker for us. I’ll consider this more strongly for the next system we build.

We used Pub/Sub very extensively (50B messages a day) but moved to Pulsar [0]. It performs equally well and has some nice features. And also no vendor lock-in. [0] https://kesque.com/billions-of-events-a-day-without-breaking...

Pulsar seems operationally quite complex, as it has a dependency on both BookKeeper and ZooKeeper (which BK also needs). ZooKeeper is particularly notorious for being difficult. What's your experience been like?

Re: Google pub/sub released an ordering feature

#67
post #9

Head of line blocking as a service!

Head-of-line blocking has two aspects: crashes and long processing times. If the first message for an ordered key can't be acked because it crashes subscribers, we can't make progress on the messages for that key. If the first message, likewise, takes too long, then it delays processing of subsequent messages for that key.

Even without ordering, we still have to deal with variations of these issues. A message may still crash subscribers, be redelivered, and crash yet more subscribers. A message may delay the processing of other messages on the subscription.

Here's one way to deal with crashing messages: keep a side-cache of processing attempts for every message ID and ack those that have been processed too many times before processing them. Cloud Pub/Sub has a dead-letter queues feature to do this automatically. (It is not yet implemented for subscriptions with ordering keys - one can't enable both features.) We can do something similar for long messages - acking the message on timeout or processing more messages in parallel.

So, with ordering keys, as it stands, you do have the extra problem of dealing with messages-of-death (crash-inducing messages) yourself, but the long-processing time issue still exists in a slightly different form. Hopefully, we can add DLQs for ordering keys if there's sufficient customer demand.

Disclaimer: I work on the Cloud Pub/Sub team, but this is my own explanation.

Re: Google pub/sub released an ordering feature

#68
post #37

Earlier quoted context omitted.

There are a couple of options without needing guaranteed ordering: - jobs can have ever increasing ids, workers record the last seen id in one place, and ignore jobs with ids less than last seen - job results are returned for each job to a supervisor. if a job result doesn't match current expected state, resend job. jobs should be idempotent in case a job is sent multiple times If the create job is expensive, the lat…

Creating ever increasing ids reliably at scale is not trivial. You will probably end up having a single server generating these ids which will then become a single point of failure.

(Or a distributed system - this is no trickier to migrate away from SPOF than anything else with global state.)

Re: Google pub/sub released an ordering feature

#69
post #61

Earlier quoted context omitted.

What's a use case where strict ordering is critically important?

"What's a use case where strict ordering is critically important?" In general, as the use case grows, every use case where the developers did not make explicit and careful provision for ensuring that order is not important, with quite non-trivial effort. Even a lot of systems whose developers think they have no ordering dependencies are wrong in at least one subtle way without realizing it. If you need to megascale,…

I can understand your post, but I don't quite buy the TCP thing. I don't think anyone is using TCP for ordering, they're using it because they don't want their packet dropped.

I guess all of the systems I build are just built to assume no order/ or to leverage causal ordering, because that feels much easier to reason about - enforcing ordering feels really hard, and like something that a message bus can only do some of the work of.

Re: Google pub/sub released an ordering feature

#70
post #18

Earlier quoted context omitted.

What's a use case where strict ordering is critically important?

Consider processing two events without guaranteed ordering: - Create A - Delete A In one ordering, A is created and then deleted as expected, in the other, the delete fails but then A is created and remains.

Kinda interesting, I built this just yesterday but with an async "buffer" and "write", and I just used a simple incrementing identifier system.
Post reply on HN