Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Drop-in SQS replacement based on SQLite

github.com

Post reply on HN
Print view View original

161–170 of 171 posts

Re: Show HN: Drop-in SQS replacement based on SQLite

#161

Post by ahachete »

ctippett wrote: post #107
andrewstuart wrote: post #99

Earlier quoted context omitted.

Ugh I didn't see that. My enthusiasm has instantly waned. Why is AGPL needed? Just be MIT and make it easy for people, espefcially if you're not planning on monetising it. I won't use AGPL code just on principle.

Why? Does AGPL really matter if you're not planning on monetising whatever it is you're using it for? Not trying to be snarky, I'm genuinely curious why you'd be so vehemently opposed to it.

AGPL is the best OSS license to ensure project continuity as Open Source. That simple.

Permissive licenses allow for proprietary forks, which may become more successful than the upstream project.

AGPL would be able to benefit from any improvements from any fork, and all those will remain OSS for everyone.

Nothing written here is related in any way to monetization.

Re: Show HN: Drop-in SQS replacement based on SQLite

#162

Post by chromanoid »

westurner wrote: post #125
chromanoid wrote: post #34

Could you elaborate why you didn't choose e.g. RabbitMQ? I mean you talk about AMQP and such, it seems to me, that a client-side abstraction would be much more efficient in providing an exit strategy for SQS than creating an SQS "compatible" broker. For example in the Java ecosystem there is https://smallrye.io/smallrye-reactive-messaging/latest/ that serves a similar purpose.

Where AWS is the likely migration path for an app if it needs to be scaled beyond dev containers, already having tested with an SQS workalike prevents rework. The celery Backends and Brokers docs compare SQS and RabbitMQ AMQP: https://docs.celeryq.dev/en/stable/getting-started/backends-... Celery's flower utility doesn't work with SQS or GCP's {Cloud Tasks, Cloud Pub/Sub, Firebase Cloud Messaging FWIU} but does work…

I don't get where you are pointing at.

RabbitMQ and other MOMs like Kafka are very versatile. What is the use case for not using SQS right now, but maybe later?

And if there is a use case (e.g. production-grade on-premise deployment), why not a client-side facade for a production-grade MOM (e.g. in Celery instead of sqs: amqp:)? Most MOMs should be more feature-rich than SQS. At-least-once delivery without pub/sub is usually the baseline and easy to configure.

I mean if this project reaches its goal to provide an SQS compatible replacement, that is nice, but I wonder if such a maturity comes with the complexity this project originally wants to avoid.

Re: Show HN: Drop-in SQS replacement based on SQLite

#163

Post by westurner »

chromanoid wrote: post #162
westurner wrote: post #125

Earlier quoted context omitted.

Where AWS is the likely migration path for an app if it needs to be scaled beyond dev containers, already having tested with an SQS workalike prevents rework. The celery Backends and Brokers docs compare SQS and RabbitMQ AMQP: https://docs.celeryq.dev/en/stable/getting-started/backends-... Celery's flower utility doesn't work with SQS or GCP's {Cloud Tasks, Cloud Pub/Sub, Firebase Cloud Messaging FWIU} but does work…

I don't get where you are pointing at. RabbitMQ and other MOMs like Kafka are very versatile. What is the use case for not using SQS right now, but maybe later? And if there is a use case (e.g. production-grade on-premise deployment), why not a client-side facade for a production-grade MOM (e.g. in Celery instead of sqs: amqp:)? Most MOMs should be more feature-rich than SQS. At-least-once delivery without pub/sub is…

When you are developing an information system but don't want to pay for SQS in development or in production; for running tests with SQLite instead of MySQL/Postgres though.

SQS and heavier ESBs are overkill for some applications, and underkill for others where an HA configuration for the MQ / task queue is necessary.

Re: Show HN: Drop-in SQS replacement based on SQLite

#164

Post by chromanoid »

westurner wrote: post #163
chromanoid wrote: post #162

Earlier quoted context omitted.

I don't get where you are pointing at. RabbitMQ and other MOMs like Kafka are very versatile. What is the use case for not using SQS right now, but maybe later? And if there is a use case (e.g. production-grade on-premise deployment), why not a client-side facade for a production-grade MOM (e.g. in Celery instead of sqs: amqp:)? Most MOMs should be more feature-rich than SQS. At-least-once delivery without pub/sub is…

When you are developing an information system but don't want to pay for SQS in development or in production; for running tests with SQLite instead of MySQL/Postgres though. SQS and heavier ESBs are overkill for some applications, and underkill for others where an HA configuration for the MQ / task queue is necessary.

Tests might be a valid use case, but doesn't seem to be the goal here and there are some solutions out there if you dont want to start a lightweight broker in a container.

Why would you want to use the SQS protocol in production without targeting the SQS "broker" as well? The timing and the AWS imposed quotas are bound to be different.

There are plenty brokers that fit different needs. I don't see the SQS protocol especially with security and so on in mind as a good fit in this case.

Re: Show HN: Drop-in SQS replacement based on SQLite

#165

Post by mannyv »

crazysim wrote: post #81
john-tells-all wrote: post #79

Why not use LocalStack? It has SQS and a lot of AWS services for testing/development. Well documented, open source. https://docs.localstack.cloud/overview/

I think the vibe might be that this is something someone could use for production.

"Production" is a big word to use for a piece of software. There are different levels of "production ready," depending on what you care about.

Some enterprise production questions:

* Is it HA? What HA topology is it designed for? * How many messages-per-second can it ingest? * What about with multiple clients? * Can you front-end it via a load balancer? * Can it guarantee only-one delivery? * How does it handle errors? ie: when a message is retrieved by a client and the client errors after the server dies, what happens to the message - does it redrive or expire correctly when the server comes back up? * How does it guarantee consistency in a multi-node environment? * How dies it behave when a node dies/runs out of memory? * How do you monitor it? * How do you recover a corrupted instance? * what happens if/when the SQLite partition runs out of space?

etc etc.

I use SQS as a main pipeline, and I would want to know a bunch of these things before I'd even consider replacing it. But for someone with one box on a VPS somewhere, you probably only care if it runs for a year without crashing

Re: Show HN: Drop-in SQS replacement based on SQLite

#166

Post by westurner »

chromanoid wrote: post #164
westurner wrote: post #163

Earlier quoted context omitted.

When you are developing an information system but don't want to pay for SQS in development or in production; for running tests with SQLite instead of MySQL/Postgres though. SQS and heavier ESBs are overkill for some applications, and underkill for others where an HA configuration for the MQ / task queue is necessary.

Tests might be a valid use case, but doesn't seem to be the goal here and there are some solutions out there if you dont want to start a lightweight broker in a container. Why would you want to use the SQS protocol in production without targeting the SQS "broker" as well? The timing and the AWS imposed quotas are bound to be different. There are plenty brokers that fit different needs. I don't see the SQS protocol es…

The switching cost from local almost-SQS to expensive HA SQS for scale and/or the client.

SQS is not a reliable exactly-once messaging protocol like AMQP, and it doesn't do task-level accounting or result storage (which SQLite also solves for).

Apache Kafka > See also: https://en.wikipedia.org/wiki/Apache_Kafka

Re: Show HN: Drop-in SQS replacement based on SQLite

#167

Post by chromanoid »

westurner wrote: post #166
chromanoid wrote: post #164

Earlier quoted context omitted.

Tests might be a valid use case, but doesn't seem to be the goal here and there are some solutions out there if you dont want to start a lightweight broker in a container. Why would you want to use the SQS protocol in production without targeting the SQS "broker" as well? The timing and the AWS imposed quotas are bound to be different. There are plenty brokers that fit different needs. I don't see the SQS protocol es…

The switching cost from local almost-SQS to expensive HA SQS for scale and/or the client. SQS is not a reliable exactly-once messaging protocol like AMQP, and it doesn't do task-level accounting or result storage (which SQLite also solves for). Apache Kafka > See also: https://en.wikipedia.org/wiki/Apache_Kafka

SQS is not particularly expensive.

I don't know where you are getting at. I work on a project with SQS and I worked with RabbitMQ, ArtemisMQ and other MOM technologies as well. At least once delivery is something you can achieve easily. This would be the common ground that SQS can also provide. The same is true for Kafka.

Re: Show HN: Drop-in SQS replacement based on SQLite

#168

Post by raihansaputra »

davidthewatson wrote: post #156
raihansaputra wrote: post #127

Earlier quoted context omitted.

can you share more about the BTLE sensors talking with Apple Watch? I'm aware that BLE heartrate sensors are detected as 'Health Devices' if I want to connect to it, but what other sensors are you working with?

Yes, thanks for asking. Continous Glucose Monitors (CGM). That used to be a diabetic-only problem until Abbott, Dexcom,and other vendors expanded their markets beyond diagnosed diabetics into pre-diabetes markets and exercise, health, and well-being applications like: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10635370/ and: https://www.levelshealth.com/ This has exploded beyond the hundred years of ketogenic resea…

i'm kinda interested in doing that too. do you wear the CGM all day? what's your recommendation? looking to link to my Apple Watch as well if possible.

Re: Show HN: Drop-in SQS replacement based on SQLite

#170

Post by threecheese »

memset wrote: post #94
threecheese wrote: post #91

Earlier quoted context omitted.

Operations. At my $bigCo we are always rolling our own ops tooling for queues, as it’s more than just “reprocess message”; there is always some business logic that separates different types of consumers/destinations, and so q management is never straightforward. Might be a differentiable feature.

Can you tell me more about the types of queue things you need to implement? (Feel free to email me too - I’d be grateful for the feedback!)

I’ll give you a real example, and it aligns with one of your feature goals that you mentioned elsewhere in these threads: allow you to rate limit without change on the client side.

I have a workload that processes transactions for P external partners (P>1000) in a market with diverse technology sets; some partners are aggregated behind MSPs and some we integrate with directly, and so those P partners represent say 500 unique destinations. This workload handles This is not a strange architecture, it represents the fact that every business-system has a lifecycle, there are legacy xml cruft and shiny graphql both exposed to the world and sitting next to each other, making money, and nearly every integrator has this. These systems go down, or networks become unavailable, or they perform maintenance, or maybe they can only support 2 concurrent connections - and every partner is different. Some MSPs come in and get some % of the market for handling this for businesses that don’t have the capability, but there are many that do.

If you were in the business of exposing products that use these services, would you want your engineers having to care about any of it? Me neither, and so it is CRITICAL that we operationalize it, which for a store-and-forward integration workload basically ends up being an operational tool for queues and their destinations.

I do not however find any tools that meet our requirements, which are simply “allow an operations/support team to manage transaction T for partner P who may be hiding behind MSP M” which includes rate limiting, circuit breaker, observability, routing, deployment, billing etc etc. This is IMO tied very closely to the queue - it is in fact the workload’s primary data store - and I do not understand why queue providers don’t differentiate their offerings with this kind of management capability.

If I think about a business journey from startup to enterprise, maybe a small shop does not need this - they only support a small number of transactions and a small number of partners. As they grow, this requires engineering, and so they just build the tools they need to scale. But it doesn’t have to be that way.

Post reply on HN