A testcontainer could be useful. I use always this adobe-mock from s3 for my test. Hmm.
Show HN: Drop-in SQS replacement based on SQLite
11–20 of 171 posts
Re: Show HN: Drop-in SQS replacement based on SQLite
#12> In terms of monetization, the goal is to just have a hosted queue system. I believe this can be cheaper than SQS without sacrificing performance. Just as Backblaze and Minio have had success competing in the S3 space, I wanted to take a crack at queues. are you monetizing this as a separate business from: https://www.ycombinator.com/companies/scratch-data
Truthfully, I don't know yet - I haven't even built a paid/hosted version at all. It is related to my existing business in the sense that it deals with realtime data.
But I started working this as something I wish existed as opposed to having some big VC strategy and pitch deck behind it.
(Also, I appreciate all of your feedback on this a month ago! It was really helpful to encourage me to keep looking into this and also figuring out the "first" things to launch with!)
Re: Show HN: Drop-in SQS replacement based on SQLite
#13Minor thing - the gif browser demo should have been maximized. The video zooming into the action on what would have fit onto a single screen was distracting.
Are there maintenance actions that the admin needs to perform on the database? How are those done?
Re: Show HN: Drop-in SQS replacement based on SQLite
#14A testcontainer could be useful. I use always this adobe-mock from s3 for my test. Hmm.
I was not aware of testcontainer until you just mentioned it! I have a very basic Dockerfile for deploying to fly.io. What/how would I get started?
It looks like localstack is a supported testcontainer and they do support SQS (but I haven't tried it myself.)
Re: Show HN: Drop-in SQS replacement based on SQLite
#15Minor thing - the gif browser demo should have been maximized. The video zooming into the action on what would have fit onto a single screen was distracting. Are there maintenance actions that the admin needs to perform on the database? How are those done?
Thank you for the feedback - agree on the gif. Also it should probably show real messages an an example instead of rand().
re: maintenance - I have tried to build this to be hands-off. The only storage this uses is SQLite, and I have the code set to automatically vacuum as space increases.
It also has a /metrics endpoint which has disk size. This is going to be used for two things in the future: first, as a metric for autoscaling (scale when disk is full) and second so that a server can stop serving requests when its disk is full (to prevent catastrophic failure.)
Re: Show HN: Drop-in SQS replacement based on SQLite
#16Very cool. I needed something like this.
Looking at short video in readme - I suspect adding live stats similar to sidekiq would make UI look more dynamic and allow quick diagnostics.
Docs for current features are more important though.
Re: Show HN: Drop-in SQS replacement based on SQLite
#17This looks great! How were you planning on tackling distributed?
This is a great question and something I've been thinking about a lot. Big picture:
Each queue node can operate (mostly) independently, and this is good. As a consumer, I don't really care where my next message comes from, so I can minimize the amount of data that needs to have a "leader".
The only data that needs to be synced is the list of queues, which doesn't change often. If one server is full, it should be able to route a request to another server.
When we downscale, we can use S3/Dynamo (GCS/firestore) to store items and redistribute.
There's more nitty gritty here (what about FIFO queues? What about replication?) but the fact that the main actions, "enqueue" and "dequeue", don't require lots of coordination makes this easier to reason about comapred to a full RDBMS.
Re: Show HN: Drop-in SQS replacement based on SQLite
#18Very cool. I needed something like this. Looking at short video in readme - I suspect adding live stats similar to sidekiq would make UI look more dynamic and allow quick diagnostics. Docs for current features are more important though.
Can you tell me more about why you needed this (and what you ended up using?)
Totally agree on adding more metrics and information to the UI - but how much of that should be on the dashboard vs exposed as a prometheus metric for someone to use their dashboard tool of choice?
I am not very good at visual design and have chosen the simplest possible tech to build it (static rendering + fomanticUI). I sometimes wonder if the lack of react or tailwind or truly beautiful elements will hold the project back.
Re: Show HN: Drop-in SQS replacement based on SQLite
#19Good, we need open implementations of all the AWS stuff.
I swear they reimplement stuff we have just so there are more places to bill us.
Re: Show HN: Drop-in SQS replacement based on SQLite
#20Keeping questions from scale & benchmarks aside, this is a cool thing for functional/unit testing module that uses SQS, instead of dumb mocks.
Thank you! Yes, you should definitely use this instead of localstack :)