Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

On SQS

tbray.org

Post reply on HN
Print view View original

11–20 of 229 posts

Re: On SQS

#11

Post by pbourke »

reallydude wrote: post #8
archgoon wrote: post #7

Earlier quoted context omitted.

> Using S3 as an ad-hoc queue is a cheaper solution, which should throw some red flags. Interesting. Can you expand on this? How do you ensure that only one worker takes a message from s3? Or do you only use this setup when you have only one worker?

You encode messages with timestamp and origin (eg 1558945545-1), you write directly to S3 into a (create if not exists) folder for a specific windowing (let's say minute). Every agent writing, you end up with a new folder in the next minute. You have a window with an ordered set of messages by window by sort algorithm...optimally determined by the naming encoding.

What happens when you need to retry the messages from a few minutes in the past because there was a transient failure in a downstream dependency?

Re: On SQS

#12

Post by HatchedLake721 »

reallydude wrote: post #8
archgoon wrote: post #7

Earlier quoted context omitted.

> Using S3 as an ad-hoc queue is a cheaper solution, which should throw some red flags. Interesting. Can you expand on this? How do you ensure that only one worker takes a message from s3? Or do you only use this setup when you have only one worker?

You encode messages with timestamp and origin (eg 1558945545-1), you write directly to S3 into a (create if not exists) folder for a specific windowing (let's say minute). Every agent writing, you end up with a new folder in the next minute. You have a window with an ordered set of messages by window by sort algorithm...optimally determined by the naming encoding.

You reminded me of a post on Dropbox announcement in 2007, that you can do it “yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem”.

Just because you can, doesn’t mean you should.

Re: On SQS

#13

Post by etaioinshrdlu »

I really wish SQS had reliably lower latency, like Redis, and also supported priority levels. (Also like redis, now, with sorted sets and the https://redis.io/commands/bzpopmax command.)

Has anyone measured the performance of Redis on large sorted sets, say millions of items? Hoping that it's still in single digit milliseconds at that size... And can sustain say 1000QPS...

Re: On SQS

#14

Post by varelaz »

One important drawback of SQS is that it's eventually consistent, you can read the same message twice from different workers. Nevertheless we keep using it with additional checks when it's critical, it's still the cheapest solution by maintenance.

Re: On SQS

#15

Post by andrewstuart »

I use Postgres SKIP LOCKED as a queue.

I used to use SQS but Postgres gives me everything I want. I can also do priority queueing and sorting.

I gave up on SQS when it couldn't be accessed from a VPC. AWS might have fixed that now.

All the other queueing mechanisms I investigated were dramatically more complex and heavyweight than Postgres SKIP LOCKED.

Re: On SQS

#16

Post by varelaz »

andrewstuart wrote: post #15

I use Postgres SKIP LOCKED as a queue. I used to use SQS but Postgres gives me everything I want. I can also do priority queueing and sorting. I gave up on SQS when it couldn't be accessed from a VPC. AWS might have fixed that now. All the other queueing mechanisms I investigated were dramatically more complex and heavyweight than Postgres SKIP LOCKED.

There are VPC API endpoints for this. As far as I recall it was all the time there from VPC begging, but I might be wrong.

Re: On SQS

#17

Post by andrewstuart »

varelaz wrote: post #16
andrewstuart wrote: post #15

I use Postgres SKIP LOCKED as a queue. I used to use SQS but Postgres gives me everything I want. I can also do priority queueing and sorting. I gave up on SQS when it couldn't be accessed from a VPC. AWS might have fixed that now. All the other queueing mechanisms I investigated were dramatically more complex and heavyweight than Postgres SKIP LOCKED.

There are VPC API endpoints for this. As far as I recall it was all the time there from VPC begging, but I might be wrong.

Only since December 2018 and you need to pay extra...

https://aws.amazon.com/about-aws/whats-new/2018/12/amazon-sq...

Re: On SQS

#18

Post by cyberferret »

We've used SQS with great results (and reliability) for many years now, but I am interested to hear the author talking about 'replaying queues' to replicate faults. I never realised you could do this with SQS. Or can you? I thought once a queue item was processed and deleted, that was it, it was gone forever, but perhaps you can see historical queue data somewhere? (without having to store it yourself)

Re: On SQS

#19

Post by philliphaydon »

varelaz wrote: post #14

One important drawback of SQS is that it's eventually consistent, you can read the same message twice from different workers. Nevertheless we keep using it with additional checks when it's critical, it's still the cheapest solution by maintenance.

> you can read the same message twice from different workers

If you have a short visibility timeout that it appears back on the queue before you delete it... Sure.

Re: On SQS

#20

Post by tybit »

cyberferret wrote: post #18

We've used SQS with great results (and reliability) for many years now, but I am interested to hear the author talking about 'replaying queues' to replicate faults. I never realised you could do this with SQS. Or can you? I thought once a queue item was processed and deleted, that was it, it was gone forever, but perhaps you can see historical queue data somewhere? (without having to store it yourself)

I think he’s saying that you can publish to a queue without any subscribers, and only play the messages for some recovery operation. It is limited as there’s a maximum life time of 14 days for SQS messages among other limits.
Post reply on HN