Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: PicoMQ – Durable Streams over HTTP, on object storage

picomq.com

Post reply on HN
Print view View original

21–30 of 41 posts

Re: Show HN: PicoMQ – Durable Streams over HTTP, on object storage

#21

Post by alasano »

adesh_nalpet wrote: post #15
Jonovono wrote: post #14

I’ve been obsessed with everything durable lately and this is good timing as electric just got acquired so the state of their durable streams project is unknown. Have you looked into the semantics of their like StreamDB stuff ? Pretty interesting Also there is this that was built on top of it. Sounds somewhat similar goal as yours but I could be off. Need to explore more https://ursula.tonbo.io/

On the same boat, and honestly, it’s not getting the attention it deserves. Given Databricks acquired ElectricSQL primarily for PGlite, the Durable Streams project is likely going to be abandoned unless new maintainers step in. I did, in fact, initially write PicoMQ with OpenRaft, similar to Ursala, but I really wanted the operational complexity to be minimal and the nodes to be stateless (at least for most use cases…

I was planning to use electric cloud for durable streams recently and found out they are winding it down post acquisition.

Unfortunate timing and self hosting isn't complex or anything but still as you said, durable streams as a project is most likely going to be abandoned.

Re: Show HN: PicoMQ – Durable Streams over HTTP, on object storage

#22

Post by adesh_nalpet »

alasano wrote: post #21
adesh_nalpet wrote: post #15

Earlier quoted context omitted.

On the same boat, and honestly, it’s not getting the attention it deserves. Given Databricks acquired ElectricSQL primarily for PGlite, the Durable Streams project is likely going to be abandoned unless new maintainers step in. I did, in fact, initially write PicoMQ with OpenRaft, similar to Ursala, but I really wanted the operational complexity to be minimal and the nodes to be stateless (at least for most use cases…

I was planning to use electric cloud for durable streams recently and found out they are winding it down post acquisition. Unfortunate timing and self hosting isn't complex or anything but still as you said, durable streams as a project is most likely going to be abandoned.

True, there are also a lot of open PRs and issues. I'm surprised no other developers or orgs are stepping in to keep the project afloat. No wonder the Apache Way goes a long way! Either way, their approach of building around an open-protocol was a good decision.

If you'd like to consider using PicoMQ, I have examples to deploy on Fly.io and AWS. ursula.tonbo.io is also a good option to try.

Re: Show HN: PicoMQ – Durable Streams over HTTP, on object storage

#24

Post by alasano »

adesh_nalpet wrote: post #22
alasano wrote: post #21

Earlier quoted context omitted.

I was planning to use electric cloud for durable streams recently and found out they are winding it down post acquisition. Unfortunate timing and self hosting isn't complex or anything but still as you said, durable streams as a project is most likely going to be abandoned.

True, there are also a lot of open PRs and issues. I'm surprised no other developers or orgs are stepping in to keep the project afloat. No wonder the Apache Way goes a long way! Either way, their approach of building around an open-protocol was a good decision. If you'd like to consider using PicoMQ, I have examples to deploy on Fly.io and AWS. ursula.tonbo.io is also a good option to try.

[dead]

Re: Show HN: PicoMQ – Durable Streams over HTTP, on object storage

#27

Post by adesh_nalpet »

raiyanr wrote: post #26

PicoMQ looks worth experimenting with during your prototype phase, especially because it is designed to run as a single binary and supports a relatively simple HTTP-based model

Indeed! The binary even embeds the admin dashboard. A good starting point to get the hang of PicoMQ: https://picomq.com/docs/playground

Re: Show HN: PicoMQ – Durable Streams over HTTP, on object storage

#29

Post by atombender »

adesh_nalpet wrote: post #19
atombender wrote: post #18

Earlier quoted context omitted.

Have you look at Google's new Rapid Bucket offering? Google Cloud Storage is arguably as good as S3, and is protocol compatible; Rapid Buckets are a type of bucket which supports appendable objects and low latency I/O. The downside is they can only be zonal, and they're a bit more expensive.

PicoMQ works with any S3-compatible object store. But I wasn't aware of GCS Rapid Bucket, it sounds a lot like AWS S3 Express, which is also zonal. And it does help with durability ACK latency quite a bit, keeping it closer to ~50ms. I'll be setting up a GCP deployment example similar to AWS soon. I'll be sure to try Rapid Bucket as well, thanks for sharing!

Ah, S3 Express does look like the same thing! Looks like directory buckets S3 Express also allows appending to any object, while I believe GCS only allows you to append to a new object and then "finalize" it.

I would also check out Tigris [1], which has an S3-compatible API. Their main claim to fame is that buckets are low-latency, multi-region and replicated by default, so supposedly you get region-local latency no matter where you are reading or writing from. I have not done any rigorous performance comparisons, though. What's amazing, if it does perform well, is that egress is free, and the pricing is otherwise the same as GCS/S3.

[1] https://www.tigrisdata.com/

Post reply on HN