Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google pub/sub released an ordering feature

cloud.google.com

Post reply on HN
Print view View original

31–40 of 114 posts

Re: Google pub/sub released an ordering feature

#31

Post by dbb01 »

hongalex wrote: post #10
erikerikson wrote: post #5

I'm floored by this. Last I spoke with that team the response was a few (but not everyone) in leadership getting it and everyone else asking why we would care. How far to come. GCP just got substantially more interesting. Edit: by this I only mean to say well done to the Google team.

By the way, in addition to ordered messages, Pub/Sub has recently gotten a number of new features including message filtering, dead letter queues, and retry policy (some GA, some beta). You can find out more here: https://cloud.google.com/pubsub/docs/release-notes Disclaimer: I work on Pub/Sub

Having these features built in without having to hack them together ourselves is nothing short of wonderful. Thanks for the great additions!

Re: Google pub/sub released an ordering feature

#32

Post by valleyjo »

tyrust wrote: post #21
trimbo wrote: post #18

Earlier quoted context omitted.

Consider processing two events without guaranteed ordering: - Create A - Delete A In one ordering, A is created and then deleted as expected, in the other, the delete fails but then A is created and remains.

Does adding timestamps not handle this case?

That won’t work in all cases. For instance, if you get messages from devices which can be reimaged they may have clock skew in a period of time before they’re synchronized again.

Re: Google pub/sub released an ordering feature

#33

Post by outsomnia »

snypox wrote: post #26
seg_lol wrote: post #22

Earlier quoted context omitted.

I don't believe I advocated for the use of Cassandra. Owning vs being owned by, we are not arguing about the same things. Stances are not strategies, when we use another's API we form a bond but the the other is free to break it so we are automatically at a weaker position. We have acquiesced. But if we choose a strategically worse choice, we have not only acquiesced, but done self-harm.

I understand. I’m just saying that in my my professional experience, nothing good came out of religously avoiding lock-in but the opposite.

With respect, clearly you cannot extrapolate that particular experience very far.

Otherwise, eg, FOSS is "nothing good".

Re: Google pub/sub released an ordering feature

#34

Post by curiousDog »

hongalex wrote: post #12
fastest963 wrote: post #8

Earlier quoted context omitted.

And do messages without an ordering key incur the same overhead?

In the unordered case, there should be no noticeable performance degradation. This was something that we looked into extensively from both the server side as well as with our client libraries. Disclaimer: I work on Pub/Sub.

Stupid question, but is there an SLO/SLA around the max delay and bandwidth in the ordered case?

Re: Google pub/sub released an ordering feature

#35

Post by manigandham »

chimera77 wrote: post #29
trimbo wrote: post #18

Earlier quoted context omitted.

Consider processing two events without guaranteed ordering: - Create A - Delete A In one ordering, A is created and then deleted as expected, in the other, the delete fails but then A is created and remains.

There are a couple of options without needing guaranteed ordering: - jobs can have ever increasing ids, workers record the last seen id in one place, and ignore jobs with ids less than last seen - job results are returned for each job to a supervisor. if a job result doesn't match current expected state, resend job. jobs should be idempotent in case a job is sent multiple times If the create job is expensive, the lat…

Just ignoring tasks isn't helpful, it means you still process things out of order and just never get to the previous pending item in that case.

Much easier to just have strict order and process it off the line as it comes.

Re: Google pub/sub released an ordering feature

#36

Post by manigandham »

drivebycomment wrote: post #19
jeffbee wrote: post #9

Head of line blocking as a service!

Seriously, I can't shake the feeling that most people who want this are likely very, very wrong and that this "feature" will turn out to be a footgun and a trap more often than a sniper rifle...

Why? It's just exposing the partitions underneath (which all horizontally scaled event/log systems use) and letting you order by the key within each partition.

This is exactly like Kafk, Pulsar, AWS Kinesis, Azure Eventhubs, and many others which are all being used by thousands of applications without issue.

Re: Google pub/sub released an ordering feature

#37

Post by bgdam »

chimera77 wrote: post #29
trimbo wrote: post #18

Earlier quoted context omitted.

Consider processing two events without guaranteed ordering: - Create A - Delete A In one ordering, A is created and then deleted as expected, in the other, the delete fails but then A is created and remains.

There are a couple of options without needing guaranteed ordering: - jobs can have ever increasing ids, workers record the last seen id in one place, and ignore jobs with ids less than last seen - job results are returned for each job to a supervisor. if a job result doesn't match current expected state, resend job. jobs should be idempotent in case a job is sent multiple times If the create job is expensive, the lat…

Creating ever increasing ids reliably at scale is not trivial. You will probably end up having a single server generating these ids which will then become a single point of failure.

Re: Google pub/sub released an ordering feature

#40

Post by zaphirplane »

jedmeyers wrote: post #24
tyrust wrote: post #21

Earlier quoted context omitted.

Does adding timestamps not handle this case?

If events are generated by different processes you cannot really guarantee that time is exactly the same for them, unless you do something fancy to ensure that.

Interesting. The ordering here is when the event was generated or when the event entered the queue ? I think the later and so I think the examples here don’t apply without something on top and a trade off
Post reply on HN