Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Drop-in SQS replacement based on SQLite

github.com

Post reply on HN
Print view View original

51–60 of 171 posts

Re: Show HN: Drop-in SQS replacement based on SQLite

#51

Post by philippta »

memset wrote: post #49
philippta wrote: post #47

One quick suggestion on project structure: Move all the structs from models/ into the root directory. This allow users of this package to have nice and short names like: q.Message and q.Queue, and avoids import naming conflicts if the user has its own „models“ package.

Thanks for the tip - and for even looking at the code! I always struggle to figure out how to organize things.

Just noticed that your root directory is already „package main“ so you can either move that to /cmd/something/ or simply rename models/ to q/. That would have the same effect and is also idiomatic.

Re: Show HN: Drop-in SQS replacement based on SQLite

#52

Post by mannyv »

So I assume it does the back-end as well?

I never cared to figure out what parts of SQS are clients-side and server side, but - does SmoothMQ support long polling, batch delivery, visibility timeouts, error handling, and - triggers? Or are triggers left to whatever is implementing the queue? Both FiFo and simple queues? Do you have throughput numbers?

As an SQS user, a table of SQS features vs SmoothMQ would be handy. If it's just an API-compatible front-end then that would be good to know. But if it does more that would also be good to know.

The reason you'd use this is because there are lots of clients who still want on-prem solutions (go figure). Being able to switch targets this way would be handy.

Re: Show HN: Drop-in SQS replacement based on SQLite

#53

Post by deskr »

Perhaps do a small example application.

  go  get github.com/poundifdef/SmoothMQ/models
  go: github.com/poundifdef/SmoothMQ@v0.0.0-20240630162953-46f8b2266d60 requires go >= 1.22.2; switching to go1.22.4
  go: github.com/poundifdef/SmoothMQ@v0.0.0-20240630162953-46f8b2266d60 (matching github.com/poundifdef/SmoothMQ/models@upgrade) requires github.com/poundifdef/SmoothMQ@v0.0.0-20240630162953-46f8b2266d60: parsing go.mod:
          module declares its path as: q
                  but was required as: github.com/poundifdef/SmoothMQ

Re: Show HN: Drop-in SQS replacement based on SQLite

#54

Post by withinboredom »

memset wrote: post #38
withinboredom wrote: post #25

Earlier quoted context omitted.

I had to build a queue implementation not too long ago. We used t2.nano for our benchmarks as well. Your results are in-line with ours pre-optimizations (except we are using nats jetstream for queuing). I’d also recommend reducing the number of threads as that can increase performance on single-processor machines (context switching will kill you). Try to find the sweet spot (for us, it was 2x-4x the number of cpus to…

These are great suggestions. Today every db write is indeed wrapped in a mutex. The two optimizations I am experimenting with are: 1. When new messages are inserted, immediately append to a file on disk. Then, in batch, insert to SQLite. 2. When dequeuing messages, keep n message IDs in memory as a ready queue, and then keep dequeued message IDs in another list. Those can be served immediately (using a SELECT which i…

For 1. We went with a per-routine bytes.Buffer that was batch "inserted" every x milliseconds or n messages. However, we don't care if we lose some messages on a crash. For integrity, some queues are set to 0ms, 1msg because we don't want to lose anything, but when it is ok that messages are lost, this is great for perf.

For 2. you could probably do something like this:

    BEGIN TRANSACTION;
    -- Select the oldest pending message
    SELECT id, message FROM queue WHERE status = 'pending' ORDER BY created_at ASC LIMIT 100;
    -- Mark messages as 'processing'
    UPDATE queue SET status = 'processing' WHERE created_at
Basically, select a batch and then abuse the ordering properties to batch mark them. Then all messages in your select you can dispatch evenly to sender threads. Sender threads can then signal a buffered channel that they've completed/failed, and the database can be updated. At startup, you can just SELECT where status = 'processing' and recover.

This is a pretty decent translation of how ours works.

Re: Show HN: Drop-in SQS replacement based on SQLite

#55

Post by memset »

mannyv wrote: post #52

So I assume it does the back-end as well? I never cared to figure out what parts of SQS are clients-side and server side, but - does SmoothMQ support long polling, batch delivery, visibility timeouts, error handling, and - triggers? Or are triggers left to whatever is implementing the queue? Both FiFo and simple queues? Do you have throughput numbers? As an SQS user, a table of SQS features vs SmoothMQ would be handy…

Everything here is on the backend. The client does very little except make api calls.

It implements many of these features so far (ie, visibility timeouts) and there are some that are still in progress (long polling.) a compatibility table is a good idea.

Re: Show HN: Drop-in SQS replacement based on SQLite

#56

Post by jpalomaki »

stackskipton wrote: post #32
memset wrote: post #21

Earlier quoted context omitted.

Curious: if you were going to switch, how would you want to run this? Would you want to deploy it to your own EC2 instances, or would you want a hosted solution (just as SQS itself is?)

Self hosted probably. Why buy knock off SQS in the cloud when real thing is right there? If you are greenfield and scared of hitching yourself to Amazon, why not go something like RabbitMQ? There is also RabbitMQ cloud providers as well.

Unless you need the features, I would steer away from the "real message queues". Running these adds complexity, especially if you want higher availability and they are not always cheap.

HTTP based solution is easy to understand and implemented (if basic functionality is enough for your use case). Real MQs obviously have more complex protocols and not all client libraries are perfect.

Re: Show HN: Drop-in SQS replacement based on SQLite

#57

Post by memset »

deskr wrote: post #53

Perhaps do a small example application. go get github.com/poundifdef/SmoothMQ/models go: github.com/poundifdef/SmoothMQ@v0.0.0-20240630162953-46f8b2266d60 requires go >= 1.22.2; switching to go1.22.4 go: github.com/poundifdef/SmoothMQ@v0.0.0-20240630162953-46f8b2266d60 (matching github.com/poundifdef/SmoothMQ/models@upgrade) requires github.com/poundifdef/SmoothMQ@v0.0.0-20240630162953-46f8b2266d60: parsing go.mod: m…

Good idea! fyi this is not meant to be used as a library. It runs as a standalone server, and then your application connects to it using an existing AWS sdk.

Re: Show HN: Drop-in SQS replacement based on SQLite

#58

Post by julius »

memset wrote: post #40
4RealFreedom wrote: post #30

The goals are a little different but I think it's worth pointing out ElasticMQ. I use it simulate sqs in a docker environment. https://github.com/softwaremill/elasticmq

I have looked at elasticmq but not played with it myself. You might also be interested in their benchmarks of all of the existing queues out there: https://softwaremill.com/mqperf/

I also just used ElasticMQ in a docker environment. Did you see any specific downsides, when you looked at it? In which scenarios should I consider replacing it with your solution?

Re: Show HN: Drop-in SQS replacement based on SQLite

#59

Post by jmspring »

skeeter2020 wrote: post #45

I love that we're seeing a lot of projects apply the KISS principal and leverage (or take inspiration) from SQLite, like this, PocketBase and even DuckDB. An entire generation of developers (including myself) were tricked into thinking you had to build for scale from day one, or worse, took the path of least resistance right to the most expensive place for cloud services: the middle. I'm hopeful the next generation w…

"build for scale", it really depends what needs to scale and what is "scale". Nearly every interview, every company these days thinks they need google or such level scaling. Nearly all do not.

Re: Show HN: Drop-in SQS replacement based on SQLite

#60

Post by stackskipton »

jpalomaki wrote: post #56
stackskipton wrote: post #32

Earlier quoted context omitted.

Self hosted probably. Why buy knock off SQS in the cloud when real thing is right there? If you are greenfield and scared of hitching yourself to Amazon, why not go something like RabbitMQ? There is also RabbitMQ cloud providers as well.

Unless you need the features, I would steer away from the "real message queues". Running these adds complexity, especially if you want higher availability and they are not always cheap. HTTP based solution is easy to understand and implemented (if basic functionality is enough for your use case). Real MQs obviously have more complex protocols and not all client libraries are perfect.

I find basics of most AMQP systems to be extremely reliable. Message on, one time delivery with message hiding until expiration of TTL. Sure, you can get into some insane setups with RabbitMQ/Kafka and that's where libraries features don't always match up. But SQS probably wouldn't work for you either in those cases.
Post reply on HN