Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Drop-in SQS replacement based on SQLite

github.com

Post reply on HN
Print view View original

151–160 of 171 posts

Re: Show HN: Drop-in SQS replacement based on SQLite

#151

Post by memset »

avinassh wrote: post #150
memset wrote: post #17

Earlier quoted context omitted.

This is a great question and something I've been thinking about a lot. Big picture: Each queue node can operate (mostly) independently, and this is good. As a consumer, I don't really care where my next message comes from, so I can minimize the amount of data that needs to have a "leader". The only data that needs to be synced is the list of queues, which doesn't change often. If one server is full, it should be able…

Have you considered libsql? It is a distributed SQLite database, written mostly in Rust. There is a go database/sql and GORM driver available - https://github.com/tursodatabase/libsql disclaimer: I am one of the maintainers

I did not know about libsql! Thanks for sharing. How do you think I should reason about making this distributed - should I use one of the (various) sqlite replicatoin libraries? Or should this be something I roll on my own (on top of some other protocol, such as raft, or something else.)

Re: Show HN: Drop-in SQS replacement based on SQLite

#152

Post by avinassh »

memset wrote: post #151
avinassh wrote: post #150

Earlier quoted context omitted.

Have you considered libsql? It is a distributed SQLite database, written mostly in Rust. There is a go database/sql and GORM driver available - https://github.com/tursodatabase/libsql disclaimer: I am one of the maintainers

I did not know about libsql! Thanks for sharing. How do you think I should reason about making this distributed - should I use one of the (various) sqlite replicatoin libraries? Or should this be something I roll on my own (on top of some other protocol, such as raft, or something else.)

> How do you think I should reason about making this distributed - should I use one of the (various) sqlite replicatoin libraries? Or should this be something I roll on my own (on top of some other protocol, such as raft, or something else.)

It really depends on the semantics of the MQ you want to provide. There is rqlite if you want a distributed SQLite over Raft.

The question is what sort of guarantees you'd like to provide and how much of latency / performance you are willing to compromise

Re: Show HN: Drop-in SQS replacement based on SQLite

#153

Post by kmacdough »

memset wrote: post #50
davidthewatson wrote: post #46

The whole idea here is superlative. +1 for k8s, kubernetes, cloud native, self-hosted, edge-enabled at low cost, no cost. I ran rq and minio for years on k8s, but been watching sqlite as a drop-in-replacement since most of my work has been early stage at or near the edge. Private cloud matters. This is an enabler. We've done too much already in public cloud where many things don't belong. BTLE sensors are perfectly h…

Really interesting. Question: when it comes to running these software on k8s, do you prefer to manage and host yourself, or do you use managed solutions on top of your own infra? (Do you pay for minio support?) Asking from a business perspective - I of course intend to keep developing this, but am also really trying to think through the business case as well.

Managed systems make a lot of sense in small companies, where keeping head count down is the key to agility & communication. For personally funded projects it's never worth the cost and for big teams (100+) its typically worth the savings to self-host IF you can find good people. Low quality/rushed/novice self-hosting is almost never a good idea for any size company.

Re: Show HN: Drop-in SQS replacement based on SQLite

#154

Post by meiraleal »

pc86 wrote: post #148
meiraleal wrote: post #118

Earlier quoted context omitted.

AGPL don't stop nobody from monetizing anything, they just gotta make their modifications public.

If you're anywhere close to a technical or developer space it's pretty clear how making your entire codebase public could negatively impact a monetization strategy. So yeah you're right that AGPL doesn't explicitly prevent monetization (other than "pay us for the source code" stuff of course) but in practice nobody with a serious monetization strategy is going to be releasing all their code AGPL either.

> but in practice nobody with a serious monetization strategy is going to be releasing all their code AGPL

Which fulfills the developer will of attracting higher quality users that plan to collaborate with him. You got it right :)

You got something wrong tho, If you're anywhere close to a free software/open source space, you should know that using his AGPL SQS replacement, the only thing that would need to be public is whatever you change in it, not the things you use it.

Re: Show HN: Drop-in SQS replacement based on SQLite

#155

Post by okr »

memset wrote: post #14
okr wrote: post #11

A testcontainer could be useful. I use always this adobe-mock from s3 for my test. Hmm.

I was not aware of testcontainer until you just mentioned it! I have a very basic Dockerfile for deploying to fly.io. What/how would I get started? It looks like localstack is a supported testcontainer and they do support SQS (but I haven't tried it myself.)

Hmm, i never provided a testcontainer myself, but i have used docker images in a testcontainer. That is pretty straight-forward.

The specialized testcontainers just pass down or expose more config parameters, imho, because they possess more knowledge about the docker container that it is about to start. For SQS i could imagine, that it is convenient to expose uri or arn or even auth parameters, that one then can refer to when setting up tests with dynamic config parameters and to setup the sqs client.

Localstack i have not used so far. I use containers often for database tests also.

Re: Show HN: Drop-in SQS replacement based on SQLite

#156

Post by davidthewatson »

raihansaputra wrote: post #127
davidthewatson wrote: post #46

The whole idea here is superlative. +1 for k8s, kubernetes, cloud native, self-hosted, edge-enabled at low cost, no cost. I ran rq and minio for years on k8s, but been watching sqlite as a drop-in-replacement since most of my work has been early stage at or near the edge. Private cloud matters. This is an enabler. We've done too much already in public cloud where many things don't belong. BTLE sensors are perfectly h…

can you share more about the BTLE sensors talking with Apple Watch? I'm aware that BLE heartrate sensors are detected as 'Health Devices' if I want to connect to it, but what other sensors are you working with?

Yes, thanks for asking.

Continous Glucose Monitors (CGM). That used to be a diabetic-only problem until Abbott, Dexcom,and other vendors expanded their markets beyond diagnosed diabetics into pre-diabetes markets and exercise, health, and well-being applications like:

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10635370/

and:

https://www.levelshealth.com/

This has exploded beyond the hundred years of ketogenic research and as research on glycemic variability in mental health has grown.

My current kit includes an Apple Watch series 9 and an iPhone SE. Prior to that it was Google Pixel and a Fitbit Sense, though direct BTLE->Watch was not an option in that generation.

I have two complications: Abbott Freestyle Libre 3 and Dexcom G7. Continuous Glucose Monitors (CGM). The Dexcom is entirely proprietary. The Abbott is accomplished via a series of 3rd party hacks:

https://www.youtube.com/watch?v=YqUZjXo5VXY

I'd say open source, but I'm not certain that every link in the chain is open source. I've run many generations of various open source tools on Android and iPhone since no vendor ships a complete end-to-end solution that is perfect.

Nightscout and watchdrip are two open source examples:

https://nightscout.github.io/

https://watchdrip.org/

When G7 originally shipped last year, sensor data left the sensor and used the iPhone as a proxy-to-cloud storage, as has become the default mode across many IoT devices because it was easy, obvious, despite the unintended consequences of the design choices here.

At that point, because the data has already taken a long and perilous journey into cloud when BTLE->Watch was dramatically shorter, cheaper (in terms of hops and requirement for service), and arguably better. Hence, any data request pays that full routing price into and out of cloud, even the most trivial display, such as watch.

After a latent Dexcom BTLE->Apple Watch update a year later, I don't even need to carry my iPhone anymore, despite the fact that I don't have service beyond WiFi on my Apple Watch, since the data exchange is entirely BTLE.

The sad fact is that straightforward questions around a person's glycemia were not answerable directly without an entire belt-worn cloud ecosystem being paid for and fully functional 24x7x365.

BTLE->Apple Watch is by no means perfect, but it's dramatically better than my previous 5 years of 24x7x365 routing through cloud.

HTH!

Re: Show HN: Drop-in SQS replacement based on SQLite

#157

Post by davidthewatson »

pfix wrote: post #65
davidthewatson wrote: post #46

The whole idea here is superlative. +1 for k8s, kubernetes, cloud native, self-hosted, edge-enabled at low cost, no cost. I ran rq and minio for years on k8s, but been watching sqlite as a drop-in-replacement since most of my work has been early stage at or near the edge. Private cloud matters. This is an enabler. We've done too much already in public cloud where many things don't belong. BTLE sensors are perfectly h…

rq as in Redis Queue?

That's right!

https://python-rq.org/

Re: Show HN: Drop-in SQS replacement based on SQLite

#158

Post by davidthewatson »

philsnow wrote: post #122
pfix wrote: post #65

Earlier quoted context omitted.

rq as in Redis Queue?

I had assumed https://github.com/rqlite/rqlite

I linked Redis Queue on the previous reply but had not heard of rqlite so thanks for this!

It looks like this may solve at least another part of my quest to replace various back-ends with sqlite.

Re: Show HN: Drop-in SQS replacement based on SQLite

#159

Post by davidthewatson »

cpill wrote: post #62
memset wrote: post #50

Earlier quoted context omitted.

Really interesting. Question: when it comes to running these software on k8s, do you prefer to manage and host yourself, or do you use managed solutions on top of your own infra? (Do you pay for minio support?) Asking from a business perspective - I of course intend to keep developing this, but am also really trying to think through the business case as well.

k3s for on premises deployments. I'm also using it for local self hosting side projects.

Definitely a fan of k3s as k8s lite - complexity that your particular project may not need, particularly since every project doesn't need k8s and many are better off with less.

If you look at the history from J2EE to the k8s prototype in Java to what we have now, it's a great idea to encapsulate all of these things into a single container, particularly at Google scale, but many unintended consequences arise from complexity accruing to features and functions which weren't actually requirements for your particular project being supported, i.e. the notion that YAGNI because few orgs have Google scale problems. If so, great! Carry on... If not, consider k3s or aptible or more emergent platforms I haven't actually used.

The mere presence of unneeded items in source and documentation presents a "why am I here?" choice paradox. That's before we even get into keeping track of deprecations in the never-at-rest source/release evolution.

Federation is a good example. I've worked at places that needed it and places that didn't.

Re: Show HN: Drop-in SQS replacement based on SQLite

#160

Post by localfirst »

jen20 wrote: post #110
localfirst wrote: post #85

Earlier quoted context omitted.

i didnt ask for a product catalogue, all im asking is to clarify whether you are truly FOSS or not by being transparent about the license. Any type of licensing conditions like AGPL 3.0 to force you to share your code is a no go for many many companies. I understand the desire to make money and you can do that without AGPL 3.0

AGPL 3.0 is _absolutely_ “truly FOSS”. Probably the most “truly Free Open Source Software” license in existence.

[flagged]
Post reply on HN