Live data from Hacker News

Show HN: Drop-in SQS replacement based on SQLite

github.com

1–10 of 171 posts

Show HN: Drop-in SQS replacement based on SQLite

#1
Hi! I wanted to share an open source API-compatible replacement for SQS. It's written in Go, distributes as a single binary, and uses SQLite for underlying storage.

I wrote this because I wanted a queue with all the bells and whistles - searching, scheduling into the future, observability, and rate limiting - all the things that many modern task queue systems have.

But I didn't want to rewrite my app, which was already using SQS. And I was frustrated that many of the best solutions out there (BullMQ, Oban, Sidekiq) were language-specific.

So I made an SQS-compatible replacement. All you have to do is replace the endpoint using AWS' native library in your language of choice.

For example, the queue works with Celery - you just change the connection string. From there, you can see all of your messages and their status, which is hard today in the SQS console (and flower doesn't support SQS.)

It is written to be pluggable. The queue implementation uses SQLite, but I've been experimenting with RocksDB as a backend and you could even write one that uses Postgres. Similarly, you could implement multiple protocols (AMQP, PubSub, etc) on top of the underlying queue. I started with SQS because it is simple and I use it a lot.

It is written to be as easy to deploy as possible - a single go binary. I'm working on adding distributed and autoscale functionality as the next layer.

Today I have search, observability (via prometheus), unlimited message sizes, and the ability to schedule messages arbitrarily in the future.

In terms of monetization, the goal is to just have a hosted queue system. I believe this can be cheaper than SQS without sacrificing performance. Just as Backblaze and Minio have had success competing in the S3 space, I wanted to take a crack at queues.

I'd love your feedback!

Show HN: Drop-in SQS replacement based on SQLite
github.com

Re: Show HN: Drop-in SQS replacement based on SQLite

#5
post #4

[flagged]

MongoDB is web scale :) https://www.youtube.com/watch?v=b2F-DItXtZs

Have you run any benchmarks yet? I'm guessing it happily handles thousands of queue operations a second given that it's SQLite and Go - would be interesting to see how settings like SQLite WAL mode affect its performance.

Re: Show HN: Drop-in SQS replacement based on SQLite

#7
> In terms of monetization, the goal is to just have a hosted queue system. I believe this can be cheaper than SQS without sacrificing performance. Just as Backblaze and Minio have had success competing in the S3 space, I wanted to take a crack at queues.

are you monetizing this as a separate business from: https://www.ycombinator.com/companies/scratch-data

Re: Show HN: Drop-in SQS replacement based on SQLite

#8
post #5
post #4

Earlier quoted context omitted.

MongoDB is web scale :) https://www.youtube.com/watch?v=b2F-DItXtZs

Have you run any benchmarks yet? I'm guessing it happily handles thousands of queue operations a second given that it's SQLite and Go - would be interesting to see how settings like SQLite WAL mode affect its performance.

I'm using WAL mode - I'd almost given up on sqlite until I remembered to enable it.

I've been playing with benchmarks, yes! I haven't written them up yet because I worry about doing them the "right" way. but since you asked, I did a quick test with a single server and single client on a t2.nano. With 3 sending threads and 5 receiving threads, and mesg size of 2kb, I can send 700 msgs/s and receiver 500 msgs/s.

It is way faster on my laptop, and I have a lot of tricks that I've been playing with to improve this.

Post reply on HN