Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: PicoMQ – Durable Streams over HTTP, on object storage

picomq.com

Post reply on HN
Print view View original

31–40 of 40 posts

Re: Show HN: PicoMQ – Durable Streams over HTTP, on object storage

#31

Post by jdcasale »

adesh_nalpet wrote: post #4
KaiserPro wrote: post #2

Can you help an old man understand? This sounds like a kafka-like streaming system, but backed onto s3-objects? Doesn't this mean that write performance is going to be bad?

I'm glad you asked, and yes, PicoMQ does have some Kafka-like semantics. However, Kafka is great at being a huge pipe, so you'd create topics like tables. PicoMQ, on the other hand, recommends creating granular streams that make the most sense, say, by user, session, or vehicle (still bottomless). And it's also fair to question write performance, since it's backed by object storage. The optimization is primarily from…

Fwiw the only two times I've used kafka in my career have been for traffic on the order of GBs/sec. And the folks I know who have relied on streaming pipes for genuinely realtime stuff built bespoke systems with RTT on the order of 10s of micros.

There are plenty of usecases for lower scale or higher latency (my examples are somewhat unique), and owning the opinionated middle instead of claiming to cover everything is a really useful thing, but acknowledging that the system is opinionated such that it covers a specific set of things well is generally a better argument than 'this basically does everything that people need'.

Re: Show HN: PicoMQ – Durable Streams over HTTP, on object storage

#32

Post by adesh_nalpet »

atombender wrote: post #29
adesh_nalpet wrote: post #19

Earlier quoted context omitted.

PicoMQ works with any S3-compatible object store. But I wasn't aware of GCS Rapid Bucket, it sounds a lot like AWS S3 Express, which is also zonal. And it does help with durability ACK latency quite a bit, keeping it closer to ~50ms. I'll be setting up a GCP deployment example similar to AWS soon. I'll be sure to try Rapid Bucket as well, thanks for sharing!

Ah, S3 Express does look like the same thing! Looks like directory buckets S3 Express also allows appending to any object, while I believe GCS only allows you to append to a new object and then "finalize" it. I would also check out Tigris [1], which has an S3-compatible API. Their main claim to fame is that buckets are low-latency, multi-region and replicated by default, so supposedly you get region-local latency no…

Totally agree! Tigris is awesome, although I haven't used it at scale. PicoMQ (even cluster mode) is a one click deployment on Fly.io and they use Tigris as the Object Store. https://picomq.com/docs/operations/deployment/fly

Re: Show HN: PicoMQ – Durable Streams over HTTP, on object storage

#33

Post by adesh_nalpet »

dzonga wrote: post #30

only if it could be run serverless. but nice product overall - just like s2 lite.

I'd say PicoMQ is close to being fully serverless, since nodes are stateless. But would miss out benefits of batching, caching and low-latency.

S2-lite is pretty cool, but it's single-node only and built on SlateDB. So, PicoMQ is closer S2 Cloud.

Re: Show HN: PicoMQ – Durable Streams over HTTP, on object storage

#34

Post by adesh_nalpet »

jdcasale wrote: post #31
adesh_nalpet wrote: post #4

Earlier quoted context omitted.

I'm glad you asked, and yes, PicoMQ does have some Kafka-like semantics. However, Kafka is great at being a huge pipe, so you'd create topics like tables. PicoMQ, on the other hand, recommends creating granular streams that make the most sense, say, by user, session, or vehicle (still bottomless). And it's also fair to question write performance, since it's backed by object storage. The optimization is primarily from…

Fwiw the only two times I've used kafka in my career have been for traffic on the order of GBs/sec. And the folks I know who have relied on streaming pipes for genuinely realtime stuff built bespoke systems with RTT on the order of 10s of micros. There are plenty of usecases for lower scale or higher latency (my examples are somewhat unique), and owning the opinionated middle instead of claiming to cover everything i…

Precisely this! I might even add a section in the docs, “Not a replacement for Kafka,” under the FAQ.

Where Kafka starts to fall short is routing. If you want to access the data of one user from user-events-topic, that’s expensive to do. Most other streaming technologies are built around the same design, such as Kinesis.

There are other implementations that support the Kafka wire protocol and are cheaper in exchange for latency, e.g., AutoMQ and WarpStream.

That said, I’ll release Disk/EBS-staged WAL soon enough: https://github.com/PicoMQ/picomq/issues/13 as an add-on to cover low-latency needs.

Re: Show HN: PicoMQ – Durable Streams over HTTP, on object storage

#35

Post by adesh_nalpet »

srinivaskarre wrote: post #28

I've been recently exploring the object store based data stores such as turbopuffer, slateDB. have tons of questions revolving in my mind about latency and relaibility. I'll try out this to understand low level details.

The team at opendata.dev/Responsive is killing it, with a strong, opinionated stance on what works well and what doesn't. I did consider building on SlateDB, but it was forcing an unnecessary primitive on streams. I have documented the design choices here: https://picomq.com/docs/design/overview

Re: Show HN: PicoMQ – Durable Streams over HTTP, on object storage

#38

Post by Farrynet »

Awesome work on this, The "one binary" approach with an embedded admin dashboard makes the barrier to entry delightfully low.

Since you mentioned nodes are stateless and you have examples for Fly.io, how does the system handle rapid node scaling (scaling to zero and back up) in response to bursty traffic? Does the WAL recovery add significant cold-start latency for consumers?

Re: Show HN: PicoMQ – Durable Streams over HTTP, on object storage

#39

Post by jdcasale »

adesh_nalpet wrote: post #34
jdcasale wrote: post #31

Earlier quoted context omitted.

Fwiw the only two times I've used kafka in my career have been for traffic on the order of GBs/sec. And the folks I know who have relied on streaming pipes for genuinely realtime stuff built bespoke systems with RTT on the order of 10s of micros. There are plenty of usecases for lower scale or higher latency (my examples are somewhat unique), and owning the opinionated middle instead of claiming to cover everything i…

Precisely this! I might even add a section in the docs, “Not a replacement for Kafka,” under the FAQ. Where Kafka starts to fall short is routing. If you want to access the data of one user from user-events-topic, that’s expensive to do. Most other streaming technologies are built around the same design, such as Kinesis. There are other implementations that support the Kafka wire protocol and are cheaper in exchange…

Also (in case this isn't obvious) I'm 100% a fan of the decision to _not_ tightly couple to the kafka wire protocol. Kafka's apis are full of landmines and gotchas, and you can do much better from a UX perspective if you're not married to their quirky semantics.

I had heard people talk about the operation pain involved in keeping Kafka alive (which is a thing for sure), but what I was surprised by was how many things behaved in a slightly unobvious manner that wasn't loudly-documented (e.g. if you're using transactions for RWP loops the default rebalance protocol is unsound and transaction markers take up an index in the log so you no longer have contiguous indices in your message stream etc).

Re: Show HN: PicoMQ – Durable Streams over HTTP, on object storage

#40

Post by adesh_nalpet »

Farrynet wrote: post #38

Awesome work on this, The "one binary" approach with an embedded admin dashboard makes the barrier to entry delightfully low. Since you mentioned nodes are stateless and you have examples for Fly.io, how does the system handle rapid node scaling (scaling to zero and back up) in response to bursty traffic? Does the WAL recovery add significant cold-start latency for consumers?

[dead]
Post reply on HN