Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

On SQS

tbray.org

Post reply on HN
Print view View original

161–170 of 229 posts

Re: On SQS

#161

Post by z3t4 »

KirinDave wrote: post #148
gmueckl wrote: post #137

Earlier quoted context omitted.

What possibilities are there for a write to fail that the database server cannot react to? I am under the impression that a write error like this would be reported as a failure to run the statement and the application is responsible for handling that.

The box turning off, the disk interface failing, and/or the link to the database failing in mid instruction. Same as SQS.

Also the FS might report the data as written but its actually in a write cache and will be lost if the plug is pulled.

Re: On SQS

#162

Post by manish_gill »

Jemaclus wrote: post #139

We love SQS, but one of the problems we're running into lately is the 256kb per message limitation. We do tens of millions of messages per day, with a small percentage of those reaching the 256kb limit. We're approaching the point where most of our messages will hit that limit. What are our options for keeping SQS but somehow sending large payloads? Only thing I can think of is throwing them into another datastore an…

I've successfully been using Kafka in production with a tiny percentage of packets ~1Mb. We utilize gzip compression with chunking on some of those messages (system is legacy, no fancy compressed message format in use). IIRC, only had to modify settings at the broker level and it works perfectly fine.

Re: On SQS

#163

Post by cthalupa »

reallydude wrote: post #8
archgoon wrote: post #7

Earlier quoted context omitted.

> Using S3 as an ad-hoc queue is a cheaper solution, which should throw some red flags. Interesting. Can you expand on this? How do you ensure that only one worker takes a message from s3? Or do you only use this setup when you have only one worker?

You encode messages with timestamp and origin (eg 1558945545-1), you write directly to S3 into a (create if not exists) folder for a specific windowing (let's say minute). Every agent writing, you end up with a new folder in the next minute. You have a window with an ordered set of messages by window by sort algorithm...optimally determined by the naming encoding.

Which gets you one of the basic features of SQS, but not the entire rest of the implementation. It's also significantly more work than just setting up an SQS queue.

I guess if you're at the point where your engineering time to implement this + all of the features on top of it that you might need from SQS and future maintenance of this custom solution is cheaper than the cost of using SQS, and you have no other outstanding work that your engineering team should be doing instead, this is a viable cost optimization strategy.

But that's a whole lot of ifs, and with customers I've mostly worked with, they're far better served just using SQS.

Re: On SQS

#164

Post by xchaotic »

avip wrote: post #159
stingraycharles wrote: post #157

Earlier quoted context omitted.

On AWS you would typically use S3 but yes this is definitely the accepted standard: nobody puts big blobs in message queues because they solve a different problem and are (typically) not designed to handle that.

You will definitely not use S3 in a pipeline if latency is an issue.

"In cases where latency is of primary concern, don't use SQS"

Re: On SQS

#165

Post by Rapzid »

mabbo wrote: post #112

A long time ago, as new-ish developer, I was building a system that needed to take inputs, then run "pass/fail/wait and try again later" until timeout or completion. This wasn't mission-critical stuff, mind you, so a lost message would annoy someone but not cause any actual harm. As I was figuring out how to setup a datastore, query it for running workflows and all that jazz, I happened upon an interesting SQS featur…

The biggest gotcha in a design like this IMHO is that you can't post and delete atomically. You may post the new work into the queue and then a failure to delete could occur and the work will stack.

Depending on the workload this could be not a big deal or very expensive. Treating a queue as a database, particularly queues that can't participate in XA transactions, can get you in trouble quick.

Re: On SQS

#166

Post by mcqueenjordan »

KirinDave wrote: post #148
gmueckl wrote: post #137

Earlier quoted context omitted.

What possibilities are there for a write to fail that the database server cannot react to? I am under the impression that a write error like this would be reported as a failure to run the statement and the application is responsible for handling that.

The box turning off, the disk interface failing, and/or the link to the database failing in mid instruction. Same as SQS.

But you have to explicitly delete the message from SQS, right? You'd only delete after confirming you processed the message, right? So if you die mid-instruction in processing a message, the message just re-appears in the SQS queue after the visibility timeout.

Re: On SQS

#167

Post by etaioinshrdlu »

reallydude wrote: post #77
etaioinshrdlu wrote: post #10

Earlier quoted context omitted.

Only the data transfer is free. The API requests are not free. S3 does come out to be more expensive...

Only if you are pushing 1 message per write. 1 write per agent per minute, ends up at just over 2$ per month fox 10 agents, regardless of the number of messages - it's cheaper for any setup I've encountered. AWS services are always middling solutions, which you can often optimize for better cost efficiency.

Interesting point. Should have mentioned this difference though, it's a quite different architecture with different tradeoffs.

Re: On SQS

#168

Post by etaioinshrdlu »

actuator wrote: post #40
etaioinshrdlu wrote: post #13

I really wish SQS had reliably lower latency, like Redis, and also supported priority levels. (Also like redis, now, with sorted sets and the https://redis.io/commands/bzpopmax command.) Has anyone measured the performance of Redis on large sorted sets, say millions of items? Hoping that it's still in single digit milliseconds at that size... And can sustain say 1000QPS...

I have worked with sorted sets with millions of items. The latency really depends on what you execute. Commands that involve only few elements are fine. Like the ZPOPMAX you mentioned should be way under a ms if you pop say 10 items, and you should be able to get way more than 1k QPS. The thing with sorted sets like most data structures in Redis is to just read the time complexity well in their documentation. For an…

But actually ZPOPMAX is proportional to log(number of elements in set). So when the set grows to millions you may have a performance problem. I have no idea how fast growing this log function is so I really have no idea of the performance on sets of millions...

Re: On SQS

#169

Post by polskibus »

gizzlon wrote: post #108
polskibus wrote: post #88

Does anyone know a good, low overhead out-of-process message queue, that's lightweight enough that it can be useful for communicating between processes on the same machine, but if necessary it can scale beyond it? In case of a single-machine product that comprises of several services, a message queue can sometimes be useful for pull model, but adding RabbitMQ to the stack makes installation and ops much more complex…

Guess it depends on the definition of "queue". Potentials: - https://nsq.io/ - https://nats.io/

Seems like NATS streaming would fit my case - have you heard of any real world deployments that use it ? Are there any larger issues that don't make it a good choice ?

Re: On SQS

#170

Post by gmueckl »

z3t4 wrote: post #161
KirinDave wrote: post #148

Earlier quoted context omitted.

The box turning off, the disk interface failing, and/or the link to the database failing in mid instruction. Same as SQS.

Also the FS might report the data as written but its actually in a write cache and will be lost if the plug is pulled.

Not an issue if you follow th e recommendations in the PostgeSQL documentation. The way to comfigure write caches is described there.
Post reply on HN