Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Drop-in SQS replacement based on SQLite

github.com

Post reply on HN
Print view View original

41–50 of 171 posts

Re: Show HN: Drop-in SQS replacement based on SQLite

#41

Post by Onavo »

jerrygenser wrote: post #7

> In terms of monetization, the goal is to just have a hosted queue system. I believe this can be cheaper than SQS without sacrificing performance. Just as Backblaze and Minio have had success competing in the S3 space, I wanted to take a crack at queues. are you monetizing this as a separate business from: https://www.ycombinator.com/companies/scratch-data

What's the point of AGPL though? The enterprises with the budget for self hosting this sort of software usually have requirements and scale beyond what SQLite can offer out of the box.

Re: Show HN: Drop-in SQS replacement based on SQLite

#42

Post by nilamo »

memset wrote: post #21
stuaxo wrote: post #19

Good, we need open implementations of all the AWS stuff. I swear they reimplement stuff we have just so there are more places to bill us.

Curious: if you were going to switch, how would you want to run this? Would you want to deploy it to your own EC2 instances, or would you want a hosted solution (just as SQS itself is?)

Ideally this would be run within a k8s cluster, to easily fit with the other services.

Re: Show HN: Drop-in SQS replacement based on SQLite

#43

Post by sgarland »

memset wrote: post #17
sgarland wrote: post #9

This looks great! How were you planning on tackling distributed?

This is a great question and something I've been thinking about a lot. Big picture: Each queue node can operate (mostly) independently, and this is good. As a consumer, I don't really care where my next message comes from, so I can minimize the amount of data that needs to have a "leader". The only data that needs to be synced is the list of queues, which doesn't change often. If one server is full, it should be able…

This would increase the complexity, but you could always run something like rqlite [0], even if only for the items that require distribution and synchronization.

Or if you truly only need to store simple values in a distributed fashion, you could probably use etcd for that part.

[0]: https://rqlite.io/

Re: Show HN: Drop-in SQS replacement based on SQLite

#44

Post by memset »

slotrans wrote: post #39

Not sure about the goal of providing a hosted service cheaper than SQS. SQS is already one of the cheapest services on Earth. It's pretty hard to spend more than a few bucks a month, even if you really try!

That is fair. Two things to validate from a biz perspective:

1. Is there some threshold where this would make sense financially (n billions of messages.)

2. Are the extra developer features (ie, larger message sizes, observability, DAGs) worth it for people to switch?

Would love your thoughts - what, if anything, would make you even entertain moving to a different queue system?

Re: Show HN: Drop-in SQS replacement based on SQLite

#45

Post by skeeter2020 »

I love that we're seeing a lot of projects apply the KISS principal and leverage (or take inspiration) from SQLite, like this, PocketBase and even DuckDB. An entire generation of developers (including myself) were tricked into thinking you had to build for scale from day one, or worse, took the path of least resistance right to the most expensive place for cloud services: the middle. I'm hopeful the next generation will have their introduction to building apps with simple, easy to manage & deploy stacks. The more I learn about SQLite, the more I love it.

Re: Show HN: Drop-in SQS replacement based on SQLite

#46

Post by davidthewatson »

The whole idea here is superlative.

+1 for k8s, kubernetes, cloud native, self-hosted, edge-enabled at low cost, no cost.

I ran rq and minio for years on k8s, but been watching sqlite as a drop-in-replacement since most of my work has been early stage at or near the edge.

Private cloud matters. This is an enabler. We've done too much already in public cloud where many things don't belong.

BTLE sensors are perfectly happy talking to my Apple Watch directly with enough debugging.

I'd argue the trip through cloud was not a win and should be corrected in the next generation of tools like this, where mobile is already well-primed for SQLite.

Re: Show HN: Drop-in SQS replacement based on SQLite

#47

Post by philippta »

One quick suggestion on project structure:

Move all the structs from models/ into the root directory.

This allow users of this package to have nice and short names like: q.Message and q.Queue, and avoids import naming conflicts if the user has its own „models“ package.

Re: Show HN: Drop-in SQS replacement based on SQLite

#48

Post by memset »

skeeter2020 wrote: post #45

I love that we're seeing a lot of projects apply the KISS principal and leverage (or take inspiration) from SQLite, like this, PocketBase and even DuckDB. An entire generation of developers (including myself) were tricked into thinking you had to build for scale from day one, or worse, took the path of least resistance right to the most expensive place for cloud services: the middle. I'm hopeful the next generation w…

Thank you for the kind words! I wanted to get something out even if it doesn't have the scalability stuff built. But I have been thinking and architecting that behind the scenes, and now I "only" need to turn it into code.

I think the queue itself will end up using a number of technologies: SQLite for some data (ie, organizing messages by date and queue), RocksDB for other things (fast lookup for messages), DuckDB (for message statistics and metadata), and other data structures.

I find that some of the most performant software often uses a mix of different data structures and algorithms by introspecting the nature of the data it has. And if I can make one node really sing, then I can have confidence that distributing that will yield results.

I think SQLite is a great start, but I really do want the software to be able to utilize the full resources the underlying hardware.

Re: Show HN: Drop-in SQS replacement based on SQLite

#49

Post by memset »

philippta wrote: post #47

One quick suggestion on project structure: Move all the structs from models/ into the root directory. This allow users of this package to have nice and short names like: q.Message and q.Queue, and avoids import naming conflicts if the user has its own „models“ package.

Thanks for the tip - and for even looking at the code! I always struggle to figure out how to organize things.

Re: Show HN: Drop-in SQS replacement based on SQLite

#50

Post by memset »

davidthewatson wrote: post #46

The whole idea here is superlative. +1 for k8s, kubernetes, cloud native, self-hosted, edge-enabled at low cost, no cost. I ran rq and minio for years on k8s, but been watching sqlite as a drop-in-replacement since most of my work has been early stage at or near the edge. Private cloud matters. This is an enabler. We've done too much already in public cloud where many things don't belong. BTLE sensors are perfectly h…

Really interesting. Question: when it comes to running these software on k8s, do you prefer to manage and host yourself, or do you use managed solutions on top of your own infra? (Do you pay for minio support?)

Asking from a business perspective - I of course intend to keep developing this, but am also really trying to think through the business case as well.

Post reply on HN