is SNS + SQS a reasonable solution for realtime irc style topic chatrooms?
Message fanout isn’t really solved there. Also probably quite expensive for the task
On SQS
101–110 of 229 posts
Re: On SQS
#102One important drawback of SQS is that it's eventually consistent, you can read the same message twice from different workers. Nevertheless we keep using it with additional checks when it's critical, it's still the cheapest solution by maintenance.
FIFO queues should fix that issue...
FIFO (First-In-First-Out) queues are designed to enhance messaging between applications when the order of operations and events is critical, or where duplicates can't be tolerated
Re: On SQS
#103Earlier quoted context omitted.
If you wanted multiple pull-based consumers for the stream, wouldn't you need a separate SQS queue per consumer, with each queue hooked up to SNS? Perhaps I'm mistaken, but that seems brittle to me. With Kinesis/Kafka, you only need to register a new appName/consumer group on the single queue for fan-out. Plus, both are FIFO by default, at least within a partition.
That's exactly how you do it. To me, it's the opposite of brittle - every consumer owns a queue, and is isolated from all other consumers. Clients are totally unaware of other systems, and there's no shared resource under contention.
Hmm. It seems a bit awkward if you have a variable number of consumers?
Re: On SQS
#104WE are heavy users of AWS. SQS is the only service where we have had zero downtime. The only downside we have about SQS is you can pull out only 10 messages at a time (without batching). You can have parallel readers but they result in some duplicates. There is SQS FIFO but it is throttled.
It went down on us once for extended period in 2015. It was chaotic as you don't expect it to fail. If memory serves me right, even S3 suffered that day.
A whole slew of AWS services went down that day. Tim's not wrong when he indicates that almost every service in AWS has a dependency on it (Amazon has services split up in to tiers based on how much they can rely on other services for critical components, SQS is pretty high up in the tiering.)
I was on-call that day for an AWS service. There wasn't much I could do but sit muted on the conference call and watch some TV, waiting for the outage to be over.
Re: On SQS
#105I use Postgres SKIP LOCKED as a queue. I used to use SQS but Postgres gives me everything I want. I can also do priority queueing and sorting. I gave up on SQS when it couldn't be accessed from a VPC. AWS might have fixed that now. All the other queueing mechanisms I investigated were dramatically more complex and heavyweight than Postgres SKIP LOCKED.
I LOVE this idea. I usually hear other Sr. engineers denigrate it as "hacky," but I think they aren't really looking at the big picture. 1. By combining services, 1 less service to manage in your stack (e.g. do your demo/local/qa envs all connect to Sqs?) 2. Postgres preserves your data if it goes down 3. You already have the tools on each machine and everybody knows the querying language to examine the stack 4. All…
I don't think it's hacky - it's using documented Postgres functionality in the way it's intended. Engineers tend to react that way to anything unfamiliar, until they decide it's a good idea then they evangelise.
What does "hacky" even mean? If it means using side effects for a primary purpose then no, SKIP LOCKED is not a side effect.
I researched alot of alternative queues to SQS and tried several of them but all of them were complex, heavyweight, with questionable library support and more trouble than they were worth.
The good thing about using a database as a queue is that you get to easily customise queue behaviour to implement things like ordering and priority and whatever other stuff you want and its all as easy as SQL.
As you say, using Postgres as a queue cut out alot of complexity associated with using a standalone queueing system.
I think MySQL/Oracle/SQL server might also support SKIP LOCKED.
Re: On SQS
#106Does anyone know a good, low overhead out-of-process message queue, that's lightweight enough that it can be useful for communicating between processes on the same machine, but if necessary it can scale beyond it? In case of a single-machine product that comprises of several services, a message queue can sometimes be useful for pull model, but adding RabbitMQ to the stack makes installation and ops much more complex…
Depending on the project a database (Postgres) can make a nice database, queue and cache. You get transactions among all three which which makes life much simpler in the beginning and it will happily march along processing thousands or tens of thousands of TPS. You can scale by adding read replicas and eventually moving to separate databases for each purpose.
If you're on the JVM then ActiveMQ (and other Java queues) will usually run embedded and have options to use a database for persistence.
Re: On SQS
#107Earlier quoted context omitted.
I LOVE this idea. I usually hear other Sr. engineers denigrate it as "hacky," but I think they aren't really looking at the big picture. 1. By combining services, 1 less service to manage in your stack (e.g. do your demo/local/qa envs all connect to Sqs?) 2. Postgres preserves your data if it goes down 3. You already have the tools on each machine and everybody knows the querying language to examine the stack 4. All…
> I LOVE this idea. I usually hear other Sr. engineers denigrate it as "hacky," but I think they aren't really looking at the big picture. Because it is hacky from the perspective of a distributed system architecture. It's coupling 2 components that probably ought not be coupled because it's perceived as "convenient" to do so. The idea that your system's control and data planes are tightly coupled is a dangerous one…
>>>> 2. Postgres preserves your data if it goes down
>>>Not if you're using a SKIP LOCKED queue,
Can you provide a reference for this? I'm not aware that using SKIP LOCKED data is treated any differently to other Postgres data in terms of durability.
Re: On SQS
#108Does anyone know a good, low overhead out-of-process message queue, that's lightweight enough that it can be useful for communicating between processes on the same machine, but if necessary it can scale beyond it? In case of a single-machine product that comprises of several services, a message queue can sometimes be useful for pull model, but adding RabbitMQ to the stack makes installation and ops much more complex…
Guess it depends on the definition of "queue". Potentials:
- https://nsq.io/
- https://nats.io/Re: On SQS
#109I use Postgres SKIP LOCKED as a queue. I used to use SQS but Postgres gives me everything I want. I can also do priority queueing and sorting. I gave up on SQS when it couldn't be accessed from a VPC. AWS might have fixed that now. All the other queueing mechanisms I investigated were dramatically more complex and heavyweight than Postgres SKIP LOCKED.
Do you have an example of such a queue somewhere? I think I have a rough idea about how it works because I implemented something similar about four years ago in PostgreSQL but kept getting locking issues I couldn't get out of: - https://stackoverflow.com/questions/33467813/concurrent-craw... - https://stackoverflow.com/questions/29807033/postgresql-conc... Also, what kind of queue size / concurrency on the polling si…
Refer example code above. I don't think Postgres included SKIP LOCKED 4 years ago.
Re: On SQS
#110Earlier quoted context omitted.
> I LOVE this idea. I usually hear other Sr. engineers denigrate it as "hacky," but I think they aren't really looking at the big picture. Because it is hacky from the perspective of a distributed system architecture. It's coupling 2 components that probably ought not be coupled because it's perceived as "convenient" to do so. The idea that your system's control and data planes are tightly coupled is a dangerous one…
>>>> 2. Postgres preserves your data if it goes down >>>Not if you're using a SKIP LOCKED queue, Can you provide a reference for this? I'm not aware that using SKIP LOCKED data is treated any differently to other Postgres data in terms of durability.
It's exactly the same case as SQS in that if your disk interface dies in mid write of a message you lose it the same as if your interface to SQS or SQS itself dies mid-write, you lose it.
The rest of the sentence is a challenge, "show me a data loss bug in SQS that extends beyond the failure cases you've already implicitly agreed to using postgres itself."