On SQS

Post by kayoone » Tue, May 28, 2019, 5:37 AM UTC

actuator wrote: post #37

kayoone wrote: post #35

> Those messages will age out and vanish after a little while (14 days is currently the max); but before they go, they’re stored carefully and are very unlikely to go missing can somebody expand on this? I know about the 14 days limitation but this makes it sound like you can store messages for a long time and still recover them somehow?

I think what he meant was, Once you successfully put a message to SQS, it is kept till the 'message retention period' unless you explicitly call delete on it. Right now you can configure the period to be upto 14 days. AFAIK, there is no way to recover messages older than their retention period.

Post by sorokod » Tue, May 28, 2019, 9:32 AM UTC

cryptonector wrote: post #179

sorokod wrote: post #73

Earlier quoted context omitted.

My question assumed a scenario where a consumer dequeues a batch, commits the deqieued change, and then crashes while processing the batch. Offcourse one could delay the commit until all processing is completed but then reasoning about the queue throughput becomes tricky.

You'd have the same problem with SQS, wouldn't you. The act of dequeueing does not guarantee that the process that received a message will not fail to perform it. If you want a reliable system along those lines than you need to use SKIP LOCKED to SELECT one row to lock, then process it, and then DELETE the row. If your process dies then the lock will be release. You still have a new flavor of the same problem: you mi…

Post by skc » Tue, May 28, 2019, 11:42 AM UTC

andrewstuart wrote: post #87

andrewstuart wrote: post #15

I use Postgres SKIP LOCKED as a queue. I used to use SQS but Postgres gives me everything I want. I can also do priority queueing and sorting. I gave up on SQS when it couldn't be accessed from a VPC. AWS might have fixed that now. All the other queueing mechanisms I investigated were dramatically more complex and heavyweight than Postgres SKIP LOCKED.

Here is a complete implementation: import psycopg2 import psycopg2.extras import random db_params = { 'database': 'jobs', 'user': 'jobsuser', 'password': 'superSecret', 'host': '127.0.0.1', 'port': '5432', } conn = psycopg2.connect(**db_params) cur = conn.cursor(cursor_factory=psycopg2.extras.DictCursor) def do_some_work(job_data): if random.choice([True, False]): print('do_some_work FAILED') raise Exception else: pr…

Post by kondro » Tue, May 28, 2019, 12:03 PM UTC

Rapzid wrote: post #165

mabbo wrote: post #112

A long time ago, as new-ish developer, I was building a system that needed to take inputs, then run "pass/fail/wait and try again later" until timeout or completion. This wasn't mission-critical stuff, mind you, so a lost message would annoy someone but not cause any actual harm. As I was figuring out how to setup a datastore, query it for running workflows and all that jazz, I happened upon an interesting SQS featur…

The biggest gotcha in a design like this IMHO is that you can't post and delete atomically. You may post the new work into the queue and then a failure to delete could occur and the work will stack. Depending on the workload this could be not a big deal or very expensive. Treating a queue as a database, particularly queues that can't participate in XA transactions, can get you in trouble quick.

Post by kondro » Tue, May 28, 2019, 12:10 PM UTC

plasma wrote: post #31

etaioinshrdlu wrote: post #13

I really wish SQS had reliably lower latency, like Redis, and also supported priority levels. (Also like redis, now, with sorted sets and the https://redis.io/commands/bzpopmax command.) Has anyone measured the performance of Redis on large sorted sets, say millions of items? Hoping that it's still in single digit milliseconds at that size... And can sustain say 1000QPS...

We use Redis as a job queue and its great; the only limitation is being sometimes concerned about job queue size due to memory limits of the Redis server itself.