1–10 of 41 posts
S3Stream is the stream storage primitive, used in AutoMQ, shipped as a Rust library. Coordination is a command log in Postgres.
Show HN: PicoMQ – Durable Streams over HTTP, on object storage
picomq.com
This sounds like a kafka-like streaming system, but backed onto s3-objects?
Doesn't this mean that write performance is going to be bad?
Can you help an old man understand? This sounds like a kafka-like streaming system, but backed onto s3-objects? Doesn't this mean that write performance is going to be bad?
Can you help an old man understand? This sounds like a kafka-like streaming system, but backed onto s3-objects? Doesn't this mean that write performance is going to be bad?
And it's also fair to question write performance, since it's backed by object storage. The optimization is primarily from the shared WAL across streams, server-side batching, and client-side in-memory pipelining, especially with HTTP/2, without as much connection pool overhead.
In practice, you can go to the extent of achieving up to 100 MiB/s throughput per stream. Considering how granular streams can be, you'd rarely need as much. The latency for a durability ACK is, however, the price to pay, which is going to be ~250 ms, or lower with S3 Express, which I'd say covers most real-time use-cases. The design itself is easy enough to extend to a disk-staged WAL for single-digit durability ACK latency.
Can you help an old man understand? This sounds like a kafka-like streaming system, but backed onto s3-objects? Doesn't this mean that write performance is going to be bad?
Not familiar with this particular library, but similar libraries use S3 Express One Zone which has write latency Regular S3 has write latency 100-150ms, which might be fine depending on your workload anyway.
So - in theory - something like a massive chat client, discord like, can be implemented via this solution? And what would be the pricing of such a solution. Cheap-serverless-discord
Sounds great, and documentation is very clear. So - in theory - something like a massive chat client, discord like, can be implemented via this solution? And what would be the pricing of such a solution. Cheap-serverless-discord
Exactly, streams can essentially be rooms, and since the ordering is preserved, a Discord-like application is a strong use case. I’m even considering building one using PicoMQ as an example showcase.
The pricing is going to be dirt cheap, and the best part is how easy it is to scale up vertically or add nodes. For some raw numbers, assuming 1M messages/day, 200B per message, ~6GB/month, all-inclusive, it would be $30 to $150 a month, and storage would be the cheapest part.
Sounds great, and documentation is very clear. So - in theory - something like a massive chat client, discord like, can be implemented via this solution? And what would be the pricing of such a solution. Cheap-serverless-discord
Thank you! I spent a decent chunk of time designing PicoMQ, so the documentation had a natural progression. Exactly, streams can essentially be rooms, and since the ordering is preserved, a Discord-like application is a strong use case. I’m even considering building one using PicoMQ as an example showcase. The pricing is going to be dirt cheap, and the best part is how easy it is to scale up vertically or add nodes.…
Is this back of envelope pricing include the traffic/bandwidth of the readers?
That could easily be 10-100x of number of messages.
This solution together with a cheap/free caching layer (especially for non members/writers) could be amazing.
BTW, a classic example would be an hn mirror. ;)