Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

On SQS

tbray.org

Post reply on HN
Print view View original

221–229 of 229 posts

Re: On SQS

#221

Post by Rapzid »

kondro wrote: post #207
Rapzid wrote: post #165

Earlier quoted context omitted.

The biggest gotcha in a design like this IMHO is that you can't post and delete atomically. You may post the new work into the queue and then a failure to delete could occur and the work will stack. Depending on the workload this could be not a big deal or very expensive. Treating a queue as a database, particularly queues that can't participate in XA transactions, can get you in trouble quick.

But you could adjust the message visibility timeout of the message you received so that it appears back later in the queue itself.

For this specific use case, as described, this sounds like a possible very good approach.

Re: On SQS

#222

Post by Rapzid »

nine_k wrote: post #191
Rapzid wrote: post #165

Earlier quoted context omitted.

The biggest gotcha in a design like this IMHO is that you can't post and delete atomically. You may post the new work into the queue and then a failure to delete could occur and the work will stack. Depending on the workload this could be not a big deal or very expensive. Treating a queue as a database, particularly queues that can't participate in XA transactions, can get you in trouble quick.

With a realistic, that is, not 100% reliable, queue you can have either "at most once" or "at least once" delivery anyway. "Exactly once" can't be guaranteed. So a duplicate message should be processed as normal anyway, e.g. by deduplication within a reasonable window, and/or by having idempotent operations.

Yes, it depends on the workload. Idempotency is typically always a good idea, but sometimes the operation itself is very expensive in terms of time, resources, and/or money. I have also seen people try to update the message when writing it back(with checkpoint information and etc) for long running processes. A slew of issues, including at least once delivery, can cause workflow bifurcation. Deduplication via FIFO _can_ help mitigate this, but it has a time window that needs to be accounted for. Once you start managing your own deduplication I'd say it has moved past trying to go databaseless.

Re: On SQS

#223

Post by manigandham »

polskibus wrote: post #198
manigandham wrote: post #184

Earlier quoted context omitted.

NATS Streaming is not as well tested and has some design issues that make scaling hard. NATS itself has a new version 2 that has a protocol update and NATS Streaming should follow with a new design as well, but I would recommend other options if you want persistence.

What are the design flaws that you have in mind? Is it ok for a couple of nodes or even then it would have trouble to keep up with a medium load? Or maybe the design flaws are to do with providing durability and other guarantees? What other options would you recommend, that can provide at least once delivery and are lighweight enough not to require zookeeper etc?

Nats streaming isn't just a persistence layer to NATS. It's an entirely different system that basically acts as a client to NATS and then records messages it sees. Basically think of how you would design a persistent queue on top of the ephemeral NATS pub/sub and that's what NATS streaming is.

Here's a good post (and series) about distributed logs and NATS design issues: https://bravenewgeek.com/building-a-distributed-log-from-scr...

Re: On SQS

#224

Post by manigandham »

tuxychandru wrote: post #194
manigandham wrote: post #184

Earlier quoted context omitted.

NATS Streaming is not as well tested and has some design issues that make scaling hard. NATS itself has a new version 2 that has a protocol update and NATS Streaming should follow with a new design as well, but I would recommend other options if you want persistence.

Can you share some details or point to articles describing the design issues with NATS streaming?

See the other comment but this is a good post/series: https://bravenewgeek.com/building-a-distributed-log-from-scr...

Re: On SQS

#225

Post by Jemaclus »

mateuszf wrote: post #200
Jemaclus wrote: post #173

Earlier quoted context omitted.

Wouldn't sending and later retrieving millions of S3 objects be expensive?

Messaging in our system is not high volume so not in this case. Also as I said - these big messages are pretty rare.

Gotcha. That makes sense.

Re: On SQS

#226

Post by kondro »

Rapzid wrote: post #221
kondro wrote: post #207

Earlier quoted context omitted.

But you could adjust the message visibility timeout of the message you received so that it appears back later in the queue itself.

For this specific use case, as described, this sounds like a possible very good approach.

One thing to be aware of for this approach though is that you can only have 120k in-flight messages per queue.

Re: On SQS

#228

Post by mabbo »

bkanber wrote: post #132
mabbo wrote: post #112

A long time ago, as new-ish developer, I was building a system that needed to take inputs, then run "pass/fail/wait and try again later" until timeout or completion. This wasn't mission-critical stuff, mind you, so a lost message would annoy someone but not cause any actual harm. As I was figuring out how to setup a datastore, query it for running workflows and all that jazz, I happened upon an interesting SQS featur…

Why are you horrified at a design that works well, scales well, is resilient enough for its use case, and is low cost? The whole point of an engineering design process is to find designs that meet these types of requirements. Honestly, this sounds like the perfect solution for what you're trying to accomplish.

Because looking at it now, something feels deeply wrong about it, haha. Honestly, if I'd used a database it probably would have opened up a few more options for future work.

I can't do any analytics about how long things typically take, who my biggest users are, etc. I mean, I could, but I'd have to add a datastore for that.

Adding new details to the parameters of the system requires very careful work to make all changes backwards and forwards compatible so that mid-deployment we don't have messages being pushed that old hosts can't process or new hosts seeing old messages they don't understand. That's good practice generally, but it's super mission critical to get right this way.

Also, a dropped message is invisible. SQS has redrive, sure, and that helps but if there were a bug, an edge case, where the system stopped processing something and quietly failed, that processing would just stop and we'd never know. If the entries were in a datastore, we'd see "Hey, this one didn't finish and I havne't worked on it lately, what gives?".

Re: On SQS

#229

Post by derefr »

cryptonector wrote: post #179
sorokod wrote: post #73

Earlier quoted context omitted.

My question assumed a scenario where a consumer dequeues a batch, commits the deqieued change, and then crashes while processing the batch. Offcourse one could delay the commit until all processing is completed but then reasoning about the queue throughput becomes tricky.

You'd have the same problem with SQS, wouldn't you. The act of dequeueing does not guarantee that the process that received a message will not fail to perform it. If you want a reliable system along those lines than you need to use SKIP LOCKED to SELECT one row to lock, then process it, and then DELETE the row. If your process dies then the lock will be release. You still have a new flavor of the same problem: you mi…

> you might process a message twice because the process might die in between completing processing and deleting the row

The very handy thing about the setup described, is that your data tables are part of the same MVCC world-state as your message queue. So you do all the work for the job, in the context of the same MVCC transaction that is holding the job locked; and anything that causes the job to fail, will fail the entire transaction, and thus rollback any changes that the job's operation made to the data.

Post reply on HN