On SQS

Post by nine_k » Tue, May 28, 2019, 1:17 AM UTC

Rapzid wrote: post #165

mabbo wrote: post #112

A long time ago, as new-ish developer, I was building a system that needed to take inputs, then run "pass/fail/wait and try again later" until timeout or completion. This wasn't mission-critical stuff, mind you, so a lost message would annoy someone but not cause any actual harm. As I was figuring out how to setup a datastore, query it for running workflows and all that jazz, I happened upon an interesting SQS featur…

The biggest gotcha in a design like this IMHO is that you can't post and delete atomically. You may post the new work into the queue and then a failure to delete could occur and the work will stack. Depending on the workload this could be not a big deal or very expensive. Treating a queue as a database, particularly queues that can't participate in XA transactions, can get you in trouble quick.

Post by cookiecaper » Tue, May 28, 2019, 2:27 AM UTC

chucksmash wrote: post #136

tjanks wrote: post #119

Earlier quoted context omitted.

Can you explain why Redis is not durable? Looking into it for a project but this comment worries me.

Intentionally so. It's not a deficiency or a footgun, it's a design decision to be aware of. Redis is an in-memory database first. You can configure Redis for durability. The docs[1] page for persistence has a good examination of the pros and cons. [1]: https://redis.io/topics/persistence

Post by cookiecaper » Tue, May 28, 2019, 2:41 AM UTC

rtpg wrote: post #44

plasma wrote: post #24

Anyone run a multi-tenant SaaS and handle fairness with jobs “fairly”? Occasionally we use to have all workers tied up on a single customers long running tasks, we mitigated by using a throttler we wrote that can defer a job if too many resources are in use by the customer, but it’s not ideal. I’d love a priority based, customer throttled (eg max concurrent tasks) queue. We can prioritize by low/medium/high using sep…

Using a database lets you make much better decisions on this space Tbh purpose-built queues are taken way too eagerly by programmers who later end up needing the flexibility offered by a more general data store.

Post by _sh » Tue, May 28, 2019, 3:32 AM UTC

andrewstuart wrote: post #87

andrewstuart wrote: post #15

I use Postgres SKIP LOCKED as a queue. I used to use SQS but Postgres gives me everything I want. I can also do priority queueing and sorting. I gave up on SQS when it couldn't be accessed from a VPC. AWS might have fixed that now. All the other queueing mechanisms I investigated were dramatically more complex and heavyweight than Postgres SKIP LOCKED.

Here is a complete implementation: import psycopg2 import psycopg2.extras import random db_params = { 'database': 'jobs', 'user': 'jobsuser', 'password': 'superSecret', 'host': '127.0.0.1', 'port': '5432', } conn = psycopg2.connect(**db_params) cur = conn.cursor(cursor_factory=psycopg2.extras.DictCursor) def do_some_work(job_data): if random.choice([True, False]): print('do_some_work FAILED') raise Exception else: pr…

Post by Jemaclus » Tue, May 28, 2019, 5:15 AM UTC

manigandham wrote: post #186

Jemaclus wrote: post #174

Earlier quoted context omitted.

That's a good idea, but wouldn't sending and later retrieving millions of S3 objects be expensive?

Yes, and slow, if you're actually doing it for millions. The comment was saying the library only does it for messages that are over the size limit though.