Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

On SQS

tbray.org

Post reply on HN
Print view View original

181–190 of 229 posts

Re: On SQS

#181

Post by manigandham »

snisarenko wrote: post #160
zug_zug wrote: post #83

Earlier quoted context omitted.

I LOVE this idea. I usually hear other Sr. engineers denigrate it as "hacky," but I think they aren't really looking at the big picture. 1. By combining services, 1 less service to manage in your stack (e.g. do your demo/local/qa envs all connect to Sqs?) 2. Postgres preserves your data if it goes down 3. You already have the tools on each machine and everybody knows the querying language to examine the stack 4. All…

There is another advantage. If you are using the queue as a log of events (i.e. user actions), you get an atomic guarantee that the db data is updated and the event describing this update has been recorded.

Also being able to run a query over the contents of the entire queue and the messages bodies (for example a JSON blob).

Re: On SQS

#182

Post by manigandham »

sorokod wrote: post #73
napsterbr wrote: post #72

Earlier quoted context omitted.

Not GP but I think this wouldn't be a problem. The consumer dequeues with SELECT FOR UPDATE within a transaction. If it crashes the database would rollback the transaction, and then another consumer would be able to select the work unit. As for acking, I see two common methods: using an additional boolean column, something like is_processed. Consumers skip truthy ones. Or, after the work is done, simply delete the en…

My question assumed a scenario where a consumer dequeues a batch, commits the deqieued change, and then crashes while processing the batch. Offcourse one could delay the commit until all processing is completed but then reasoning about the queue throughput becomes tricky.

Delaying the commit is the standard approach with this. SKIP LOCKED was created specifically to avoid the throughput issues of locked rows (and has similar implementations in other RDBMS).

If you don't want to keep the transaction open than you can just go back to updating a column containing the message status, which avoids keeping a transaction open but might need a background process to check for stalled out consumers.

Re: On SQS

#183

Post by manigandham »

espadrine wrote: post #46
plasma wrote: post #31

Earlier quoted context omitted.

We use Redis as a job queue and its great; the only limitation is being sometimes concerned about job queue size due to memory limits of the Redis server itself.

Also, when you don't want to lose a message, the Redis persistence story requires careful thought. It requires setting up RDB + AOF + appendfsync=always + backups.

You don't need to fsync on every write, that's what the replica is for. At lower/mid-tier hardware, network is faster than storage and your message is on multiple machines before it's even written to disk so fsync of 1 second is usually fine.

Re: On SQS

#184

Post by manigandham »

polskibus wrote: post #169
gizzlon wrote: post #108

Earlier quoted context omitted.

Guess it depends on the definition of "queue". Potentials: - https://nsq.io/ - https://nats.io/

Seems like NATS streaming would fit my case - have you heard of any real world deployments that use it ? Are there any larger issues that don't make it a good choice ?

NATS Streaming is not as well tested and has some design issues that make scaling hard. NATS itself has a new version 2 that has a protocol update and NATS Streaming should follow with a new design as well, but I would recommend other options if you want persistence.

Re: On SQS

#185

Post by manigandham »

Jemaclus wrote: post #140
tjanks wrote: post #119

Earlier quoted context omitted.

Can you explain why Redis is not durable? Looking into it for a project but this comment worries me.

In my experience Redis is plenty durable (especially in Elasticache with replicas + multi-region + backups). Redis _is_ an in-memory data store, so if the server crashes, you lose the data, but if you have replicas it'll fail over, if you have backups you can restore, and if you have multi-region you can failover to the other region. IMO, the idea that Redis is not durable enough is outdated. It's something to be awa…

Redis won't lose data from just a server crash. It has persistence with streaming AOF mode and snapshot RDB mode, and they can be combined. fsync is also configurable. It can be set to write every operation to disk but most set it to every 1 second which is safe enough with replicas.

Re: On SQS

#186

Post by manigandham »

Jemaclus wrote: post #174
mark242 wrote: post #146

Earlier quoted context omitted.

The Java SDK for SQS automatically drops >256k messages into an S3 bucket and stores a pointer to the message in SQS itself. You set the bucket and the client transparently handles retrieving a message from SQS/S3 when necessary. Based on the structure of the message (UUIDv4) you could probably roll your own implementation in any language.

That's a good idea, but wouldn't sending and later retrieving millions of S3 objects be expensive?

Yes, and slow, if you're actually doing it for millions. The comment was saying the library only does it for messages that are over the size limit though.

Re: On SQS

#187

Post by devonkim »

zug_zug wrote: post #83
andrewstuart wrote: post #15

I use Postgres SKIP LOCKED as a queue. I used to use SQS but Postgres gives me everything I want. I can also do priority queueing and sorting. I gave up on SQS when it couldn't be accessed from a VPC. AWS might have fixed that now. All the other queueing mechanisms I investigated were dramatically more complex and heavyweight than Postgres SKIP LOCKED.

I LOVE this idea. I usually hear other Sr. engineers denigrate it as "hacky," but I think they aren't really looking at the big picture. 1. By combining services, 1 less service to manage in your stack (e.g. do your demo/local/qa envs all connect to Sqs?) 2. Postgres preserves your data if it goes down 3. You already have the tools on each machine and everybody knows the querying language to examine the stack 4. All…

I'm not opposed to a separate table or DB used exclusively for messaging separate from business level data, but I think I've never heard of anyone doing that in practice. It's almost always a bunch of hacky joins someone wants to do with the messages and the messaging gets directly coupled with the schema (foreign key constraints on transaction or event IDs and business entities is the usual case). Letting one or two things be used for purposes it wasn't necessarily designed for is alright but coupling the same place that has your OLTP together as your messaging system in itself is usually a culture where the same DB will be used in another month as a time series DB, then as an OLAP system, and then there's no time to properly decouple anything anymore / actually understand your needs and nobody wants to try to touch the very, very brittle DB that runs the entire business on its back.

It is a very, very slippery slope and it's seductive to put everything into Postgres at low scale but I've seen more places sunk by technical debt (because they can't move fast enough to catch up to parity with market demands, can't scale operationally with the budget, or address competitors' features) than ones that don't get enough traction where technical debt becomes a problem. But perhaps my experience is perversely biased and I'm just driven to over-engineer stuff because I've been routinely handed the reins of quite literally business non-viable software systems that were over-sold repeatedly while Silicon Valley sounds like it has the inverse problem of over-engineering for non-viable businesses.

Re: On SQS

#188

Post by andrewstuart »

devonkim wrote: post #187
zug_zug wrote: post #83

Earlier quoted context omitted.

I LOVE this idea. I usually hear other Sr. engineers denigrate it as "hacky," but I think they aren't really looking at the big picture. 1. By combining services, 1 less service to manage in your stack (e.g. do your demo/local/qa envs all connect to Sqs?) 2. Postgres preserves your data if it goes down 3. You already have the tools on each machine and everybody knows the querying language to examine the stack 4. All…

I'm not opposed to a separate table or DB used exclusively for messaging separate from business level data, but I think I've never heard of anyone doing that in practice. It's almost always a bunch of hacky joins someone wants to do with the messages and the messaging gets directly coupled with the schema (foreign key constraints on transaction or event IDs and business entities is the usual case). Letting one or two…

I think you're saying it's not a good approach because developers will do it wrong?

That's hard to argue with.

The example code shows how it should be done - a simple table dedicated to messages which supplies the id of the work/job to be carried out.

Not much can be done architecturally to address developers messing that up.

Re: On SQS

#189

Post by KirinDave »

mcqueenjordan wrote: post #166
KirinDave wrote: post #148

Earlier quoted context omitted.

The box turning off, the disk interface failing, and/or the link to the database failing in mid instruction. Same as SQS.

But you have to explicitly delete the message from SQS, right? You'd only delete after confirming you processed the message, right? So if you die mid-instruction in processing a message, the message just re-appears in the SQS queue after the visibility timeout.

That's a feature, and one your have to replicate in your postgres queue.

Re: On SQS

#190

Post by jgalt212 »

etaioinshrdlu wrote: post #13

I really wish SQS had reliably lower latency, like Redis, and also supported priority levels. (Also like redis, now, with sorted sets and the https://redis.io/commands/bzpopmax command.) Has anyone measured the performance of Redis on large sorted sets, say millions of items? Hoping that it's still in single digit milliseconds at that size... And can sustain say 1000QPS...

I'm gonna give SQS a shot because authentication in redis seems to be quite a challenge, or was so the last time we looked about six months back.
Post reply on HN