31–40 of 171 posts
Good, we need open implementations of all the AWS stuff. I swear they reimplement stuff we have just so there are more places to bill us.
Curious: if you were going to switch, how would you want to run this? Would you want to deploy it to your own EC2 instances, or would you want a hosted solution (just as SQS itself is?)
If you are greenfield and scared of hitching yourself to Amazon, why not go something like RabbitMQ? There is also RabbitMQ cloud providers as well.
Actually pretty excited to try this, there are so many cases where I need a bare-bones (aka "simple"), local, persistent queue, but all the usual suspects (amqp, apache whatever, cloud nonsense, etc.) are way too heavy. I'll probably try poking at it directly through the HTTP API rather than an SDK ... does it need AWS V4 auth signatures or anything?
If you do use your language's AWS SDK, the code handles [1] all of the V4 auth stuff. https://github.com/poundifdef/SmoothMQ/blob/main/protocols/s...
I'd love your feedback! Particularly the difficulties you find in running it, of which I'm sure there are many, so I can fix them or update docs.
Actually pretty excited to try this, there are so many cases where I need a bare-bones (aka "simple"), local, persistent queue, but all the usual suspects (amqp, apache whatever, cloud nonsense, etc.) are way too heavy. I'll probably try poking at it directly through the HTTP API rather than an SDK ... does it need AWS V4 auth signatures or anything?
Earlier quoted context omitted.
I'm using WAL mode - I'd almost given up on sqlite until I remembered to enable it. I've been playing with benchmarks, yes! I haven't written them up yet because I worry about doing them the "right" way. but since you asked, I did a quick test with a single server and single client on a t2.nano. With 3 sending threads and 5 receiving threads, and mesg size of 2kb, I can send 700 msgs/s and receiver 500 msgs/s. It is…
You might need dedicated resources, including storage speed guarantees, to get reproducible benchmarks out of a cloud provider. Love the red sweater btw very classy yet simple
The goal is for this to be able to run well on commodity hardware, which I think is possible. If I can run this on $platform with $cheap instances and an $autoscaler then that would be my ideal design goal because I think it matches the setup that most people have access to.
However, I agree - I am a big fan of Hetzner's servers and am excited to try out benchmarks on beefier hardware too.
Earlier quoted context omitted.
I'm using WAL mode - I'd almost given up on sqlite until I remembered to enable it. I've been playing with benchmarks, yes! I haven't written them up yet because I worry about doing them the "right" way. but since you asked, I did a quick test with a single server and single client on a t2.nano. With 3 sending threads and 5 receiving threads, and mesg size of 2kb, I can send 700 msgs/s and receiver 500 msgs/s. It is…
I had to build a queue implementation not too long ago. We used t2.nano for our benchmarks as well. Your results are in-line with ours pre-optimizations (except we are using nats jetstream for queuing). I’d also recommend reducing the number of threads as that can increase performance on single-processor machines (context switching will kill you). Try to find the sweet spot (for us, it was 2x-4x the number of cpus to…
1. When new messages are inserted, immediately append to a file on disk. Then, in batch, insert to SQLite.
2. When dequeuing messages, keep n message IDs in memory as a ready queue, and then keep dequeued message IDs in another list. Those can be served immediately (using a SELECT which is fast) and then updating messages to the dequeued status can happen in batch.
Appreciate the tips!
The goals are a little different but I think it's worth pointing out ElasticMQ. I use it simulate sqs in a docker environment. https://github.com/softwaremill/elasticmq