Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

On SQS

tbray.org

Post reply on HN
Print view View original

211–220 of 229 posts

Re: On SQS

#211

Post by kondro »

reallydude wrote: post #77
etaioinshrdlu wrote: post #10

Earlier quoted context omitted.

Only the data transfer is free. The API requests are not free. S3 does come out to be more expensive...

Only if you are pushing 1 message per write. 1 write per agent per minute, ends up at just over 2$ per month fox 10 agents, regardless of the number of messages - it's cheaper for any setup I've encountered. AWS services are always middling solutions, which you can often optimize for better cost efficiency.

This is a message queue. But wouldn’t be fit for purpose for the way most people need to use message queues.

Re: On SQS

#212

Post by kondro »

varelaz wrote: post #14

One important drawback of SQS is that it's eventually consistent, you can read the same message twice from different workers. Nevertheless we keep using it with additional checks when it's critical, it's still the cheapest solution by maintenance.

Implementing a queue that does exactly once delivery is a very hard problem and comes with its own trade-offs. Mostly around scalability and reliability.

Re: On SQS

#213

Post by kondro »

akvadrako wrote: post #102
varelaz wrote: post #14

One important drawback of SQS is that it's eventually consistent, you can read the same message twice from different workers. Nevertheless we keep using it with additional checks when it's critical, it's still the cheapest solution by maintenance.

FIFO queues should fix that issue... FIFO (First-In-First-Out) queues are designed to enhance messaging between applications when the order of operations and events is critical, or where duplicates can't be tolerated

And it comes with the trade off of 3000 (batches) messages per second (and I bet some reduced potential availability). Sure, it’s a lot of messages for most people but non-FIFO SQS is one of the truly limitless services offered by AWS. No limits to raise, no soft maximums AWS staff ask you not to exceed. Just limitless scaling.

Re: On SQS

#214

Post by kondro »

cyberferret wrote: post #61
ilogik wrote: post #27

Earlier quoted context omitted.

maybe once you've processed a message, you resend it to a different queue, just in case?

That's actually a neat idea. Since we don't have to pay for number of items in a queue, I am thinking we could create a dummy queue and just handball all incoming messages to it for safekeeping (at least for 14 days)?!?

Or send everything to SNS and have two SQS queues subscribed to the topic.

Re: On SQS

#215

Post by devonkim »

andrewstuart wrote: post #188
devonkim wrote: post #187

Earlier quoted context omitted.

I'm not opposed to a separate table or DB used exclusively for messaging separate from business level data, but I think I've never heard of anyone doing that in practice. It's almost always a bunch of hacky joins someone wants to do with the messages and the messaging gets directly coupled with the schema (foreign key constraints on transaction or event IDs and business entities is the usual case). Letting one or two…

I think you're saying it's not a good approach because developers will do it wrong? That's hard to argue with. The example code shows how it should be done - a simple table dedicated to messages which supplies the id of the work/job to be carried out. Not much can be done architecturally to address developers messing that up.

While the pattern is simple programmatically, it is insufficient operationally and sets a precedent of coupling your data plane and your messaging plane.

Looking solely from a code perspective, how would this be done prior to Postgres 9.5 (before SKIP LOCKED)? Of the four examples I saw (either MySQL or Postgres), all of them choked constantly and had a pathological case of a job failing to execute expediently during peak usage which made on-call a nightmare. So what do we do about the places that can't upgrade their Postgres databases because it's too complicated to decouple messaging and business data now? Based upon my observations the answer is "it's never done" or the company goes under from lack of ability to enact changes caused by the paralysis.

The reality is that simple jobs are almost never enough outside toy examples. Soon, someone wants to be notified about job status changes - then comes an event table with a foreign key constraint on the job table's primary key. Also add in job digraphs with sub-tasks and workflows (AKA weak transactions) - these are non-trivial to do well. Depending upon your SQL variant and version combined with access patterns, row level locking and indexing can be an O(1) lookup or an O(n^2) nightmare that causes tons of contention depending (once again) upon the manner in which jobs are modified.

Instead of thinking about "what are my transaction boundaries and what do my asynchronous jobs look like?" which are much more invariant and important to the business trying to map their existing solution onto as many problems as possible. Then it should be more clear whether you would be fine with a RDBMS table, Airflow, SQS, Redis, RabbitMQ, JMS, etc. Operations-wise more components is certainly a headache, but I've had more headaches in production due to inappropriate technology for the problem domain than "this is just too many parts!"

Re: On SQS

#216

Post by TeeWEE »

Jemaclus wrote: post #139

We love SQS, but one of the problems we're running into lately is the 256kb per message limitation. We do tens of millions of messages per day, with a small percentage of those reaching the 256kb limit. We're approaching the point where most of our messages will hit that limit. What are our options for keeping SQS but somehow sending large payloads? Only thing I can think of is throwing them into another datastore an…

Why are your messages large? Messages should just be signalling events, like follows:

OrderCreated(order_id=23)

UserLoggedIn(user_id=342)

etc.

I guess you are sending the whole user object over the event bus? Isnt that an anti-pattern?

Re: On SQS

#217

Post by cryptonector »

sorokod wrote: post #203
cryptonector wrote: post #179

Earlier quoted context omitted.

You'd have the same problem with SQS, wouldn't you. The act of dequeueing does not guarantee that the process that received a message will not fail to perform it. If you want a reliable system along those lines than you need to use SKIP LOCKED to SELECT one row to lock, then process it, and then DELETE the row. If your process dies then the lock will be release. You still have a new flavor of the same problem: you mi…

With SQS, the act of dequeueing makes the mesage invisible to other consumers for a predefined time period. The consumer can ack the mesage once the procesing is completed resulting in the message being deleted. If the consumers fails to do so - the mesage will eventually become elligible to be processed by another consumer,

I described, in the message you're responding to, how to do the same thing with SKIP LOCKED.

Re: On SQS

#218

Post by anarazel »

devonkim wrote: post #215
andrewstuart wrote: post #188

Earlier quoted context omitted.

I think you're saying it's not a good approach because developers will do it wrong? That's hard to argue with. The example code shows how it should be done - a simple table dedicated to messages which supplies the id of the work/job to be carried out. Not much can be done architecturally to address developers messing that up.

While the pattern is simple programmatically, it is insufficient operationally and sets a precedent of coupling your data plane and your messaging plane. Looking solely from a code perspective, how would this be done prior to Postgres 9.5 (before SKIP LOCKED)? Of the four examples I saw (either MySQL or Postgres), all of them choked constantly and had a pathological case of a job failing to execute expediently during…

> Looking solely from a code perspective, how would this be done prior to Postgres 9.5 (before SKIP LOCKED)?

It's possible to implement SKIP LOCKED in userland in PostgreSQL (using NOWAIT, which has been in PG), although it's obviously a bit slower.

Re: On SQS

#219

Post by actuator »

etaioinshrdlu wrote: post #168
actuator wrote: post #40

Earlier quoted context omitted.

I have worked with sorted sets with millions of items. The latency really depends on what you execute. Commands that involve only few elements are fine. Like the ZPOPMAX you mentioned should be way under a ms if you pop say 10 items, and you should be able to get way more than 1k QPS. The thing with sorted sets like most data structures in Redis is to just read the time complexity well in their documentation. For an…

But actually ZPOPMAX is proportional to log(number of elements in set). So when the set grows to millions you may have a performance problem. I have no idea how fast growing this log function is so I really have no idea of the performance on sets of millions...

Well, that represents the asymptotic time complexity. So, roughly with an increasing N it increases by log of it. Here is an example of the log(base 2) function growth.

Log(4) = 2, Log(16) = 4, Log(256) = 8, Log(65536) = 16, Log(4,294,967,296) = 32

Assuming other things remaining unaffected, theoretically that means in a set of 4 million the operation should be about 4 times slower than in a set of 250 elements.

Re: On SQS

#220

Post by reallydude »

kondro wrote: post #211
reallydude wrote: post #77

Earlier quoted context omitted.

Only if you are pushing 1 message per write. 1 write per agent per minute, ends up at just over 2$ per month fox 10 agents, regardless of the number of messages - it's cheaper for any setup I've encountered. AWS services are always middling solutions, which you can often optimize for better cost efficiency.

This is a message queue. But wouldn’t be fit for purpose for the way most people need to use message queues.

What constitutes "the queue" is a matter of data structuring, not a singular brand (eg RabbitMQ, Redis, SQS, etc each have their own internals).

Any middle tiering of the data before it reaches the consumer, is still "the queue". You don't need to know the internals of SQS, anymore than a consumer need know the black box elements of how you collate the messages within your ad-hoc queue.

Post reply on HN