21–30 of 171 posts
Good, we need open implementations of all the AWS stuff. I swear they reimplement stuff we have just so there are more places to bill us.
I assume this would work without much issue with Litestream, though I'm curious if you've already tried it. This would make a great ephemeral queue system without having to worry about coordinating backend storage.
This is super cool! I love projects that aim to create simple self-hostable alternatives to popular services. I assume this would work without much issue with Litestream, though I'm curious if you've already tried it. This would make a great ephemeral queue system without having to worry about coordinating backend storage.
The nice thing about queues is that backend storage doesn't really need to be coordinated. Like, you could have two servers, with two sets of messages, and the client can just pull from them round robin. They (mostly) don't need to coordinate at all for this to work.
However, this is different for replication where we have multiple nodes as backups.
Earlier quoted context omitted.
Have you run any benchmarks yet? I'm guessing it happily handles thousands of queue operations a second given that it's SQLite and Go - would be interesting to see how settings like SQLite WAL mode affect its performance.
I'm using WAL mode - I'd almost given up on sqlite until I remembered to enable it. I've been playing with benchmarks, yes! I haven't written them up yet because I worry about doing them the "right" way. but since you asked, I did a quick test with a single server and single client on a t2.nano. With 3 sending threads and 5 receiving threads, and mesg size of 2kb, I can send 700 msgs/s and receiver 500 msgs/s. It is…
I’d also recommend reducing the number of threads as that can increase performance on single-processor machines (context switching will kill you). Try to find the sweet spot (for us, it was 2x-4x the number of cpus to threads, at least, for our workload).
If you are using go, it kinda sucks at single-cpu things, and if you detect that you have a single core, you can lock a go-routine to a thread and that sometimes helps.
Also, try to batch sends (aka, for-loop) to attempt to saturate the send-side so by the time you come back with your next batch they’re still messages sitting in a network buffer somewhere. For example, we have a channel that wakes up our sender, then we honest-to-god wait like 60ms in the hopes there will be more than one message to pick up — and there usually is in production.
SQS: Amazon Simple Queue Service
[flagged]
I'll probably try poking at it directly through the HTTP API rather than an SDK ... does it need AWS V4 auth signatures or anything?
Earlier quoted context omitted.
Have you run any benchmarks yet? I'm guessing it happily handles thousands of queue operations a second given that it's SQLite and Go - would be interesting to see how settings like SQLite WAL mode affect its performance.
I'm using WAL mode - I'd almost given up on sqlite until I remembered to enable it. I've been playing with benchmarks, yes! I haven't written them up yet because I worry about doing them the "right" way. but since you asked, I did a quick test with a single server and single client on a t2.nano. With 3 sending threads and 5 receiving threads, and mesg size of 2kb, I can send 700 msgs/s and receiver 500 msgs/s. It is…
Love the red sweater btw very classy yet simple