On SQS
tbray.org
On SQS
1–10 of 229 posts
Re: On SQS
#2WE are heavy users of AWS. SQS is the only service where we have had zero downtime. The only downside we have about SQS is you can pull out only 10 messages at a time (without batching). You can have parallel readers but they result in some duplicates. There is SQS FIFO but it is throttled.
Re: On SQS
#3Nowhere is cost mentioned. Using S3 as an ad-hoc queue is a cheaper solution, which should throw some red flags. You can easily do what SQS does for so much cheaper (this includes horizontal scaling and failure planning), that I'm consistently surprised anyone uses it. Either you are running at a volume where you need high throughput and it's pricey, or you're at such a low throughput you could use any MQ (even redis).
> Oh but what about ORDERED queues? The only way to get ordered application of writes is to perform them one after the other.
This is another WTF. Talking about ordered queues is like talking about databases, because it's data that's structured. If you can feed data from concurrent sources of unordered data to a system where access can be ordered, you have access to a sorted data. You deal with out-of-order data either in the insertions or a window in the processing or in the consumers. "Write in order" is not a requirement, but an option. Talking about technical subjects on twitter always results in some mind-numbingly idiotic statements for the sake of 144 characters.
Re: On SQS
#4WE are heavy users of AWS. SQS is the only service where we have had zero downtime. The only downside we have about SQS is you can pull out only 10 messages at a time (without batching). You can have parallel readers but they result in some duplicates. There is SQS FIFO but it is throttled.
Plus FIFO isn’t integrated with SNS nor is it integrated with Lambda leaving it in an island of its own within AWS.
Re: On SQS
#5Nowhere is cost mentioned. Using S3 as an ad-hoc queue is a cheaper solution, which should throw some red flags. You can easily do what SQS does for so much cheaper (this includes horizontal scaling and failure planning), that I'm consistently surprised anyone uses it. Either you are running at a volume where you need high throughput and it's pricey, or you're at such a low throughput you could use any MQ (even redis…
In the first subheading, towards the bottom, the author mentions it’s $0.40 per million api calls.
Re: On SQS
#6Nowhere is cost mentioned. Using S3 as an ad-hoc queue is a cheaper solution, which should throw some red flags. You can easily do what SQS does for so much cheaper (this includes horizontal scaling and failure planning), that I'm consistently surprised anyone uses it. Either you are running at a volume where you need high throughput and it's pricey, or you're at such a low throughput you could use any MQ (even redis…
S3's eventual consistency aside, how is it cheaper?
It would seem to me that naively, S3 charges $5 per million POST requests, so it's 10x worse than SQS's $0.40 per million.
Re: On SQS
#7Nowhere is cost mentioned. Using S3 as an ad-hoc queue is a cheaper solution, which should throw some red flags. You can easily do what SQS does for so much cheaper (this includes horizontal scaling and failure planning), that I'm consistently surprised anyone uses it. Either you are running at a volume where you need high throughput and it's pricey, or you're at such a low throughput you could use any MQ (even redis…
> Using S3 as an ad-hoc queue is a cheaper solution, which should throw some red flags.
Interesting. Can you expand on this? How do you ensure that only one worker takes a message from s3? Or do you only use this setup when you have only one worker?
Re: On SQS
#8Nowhere is cost mentioned. Using S3 as an ad-hoc queue is a cheaper solution, which should throw some red flags. You can easily do what SQS does for so much cheaper (this includes horizontal scaling and failure planning), that I'm consistently surprised anyone uses it. Either you are running at a volume where you need high throughput and it's pricey, or you're at such a low throughput you could use any MQ (even redis…
> Using S3 as an ad-hoc queue is a cheaper solution, which should throw some red flags. Interesting. Can you expand on this? How do you ensure that only one worker takes a message from s3? Or do you only use this setup when you have only one worker?
You encode messages with timestamp and origin (eg 1558945545-1), you write directly to S3 into a (create if not exists) folder for a specific windowing (let's say minute). Every agent writing, you end up with a new folder in the next minute. You have a window with an ordered set of messages by window by sort algorithm...optimally determined by the naming encoding.
Re: On SQS
#9Nowhere is cost mentioned. Using S3 as an ad-hoc queue is a cheaper solution, which should throw some red flags. You can easily do what SQS does for so much cheaper (this includes horizontal scaling and failure planning), that I'm consistently surprised anyone uses it. Either you are running at a volume where you need high throughput and it's pricey, or you're at such a low throughput you could use any MQ (even redis…
S3's eventual consistency aside, how is it cheaper? It would seem to me that naively, S3 charges $5 per million POST requests, so it's 10x worse than SQS's $0.40 per million.
Writing directly from EC2 is free (within same region).
Re: On SQS
#10Earlier quoted context omitted.
S3's eventual consistency aside, how is it cheaper? It would seem to me that naively, S3 charges $5 per million POST requests, so it's 10x worse than SQS's $0.40 per million.
Writing directly from EC2 is free (within same region).
Only the data transfer is free. The API requests are not free. S3 does come out to be more expensive...