Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Siberite: A Simple LevelDB-Backed Message Queue in Go

github.com

Post reply on HN
Print view View original

11–20 of 37 posts

Re: Siberite: A Simple LevelDB-Backed Message Queue in Go

#11

Post by Bogdanovich »

eis wrote: post #6

Why would you choose a LSM Tree based storage mechanism for a message queue? The only reason I can come up with would be because it's a read-to-use library you can just plug in which gives OK performance and some handy features because you can use the KV store for other things. But it doesn't scale well and backups with LevelDB are not really easy either (close DB, copy all files). Message queues when they are ordere…

Yes, goleveldb was chosen because it's a ready to use library with a decent write and read performance, and no external non-Go dependencies. It can also be used to store multiple consumers offsets in future.

Regarding provided guarantees, with simple 'get work_queue' reads it provides at-most-once delivery. With two phase reliable reads 'get work_queue/open', 'get work_queue/close' it provides at-least-once delivery (although message is kept in memory on server during a reliable read and will be lost if you SIGKILL siberite. On SIGTERM and SIGINT siberite will gracefully abort the read and save the message).

Re: Siberite: A Simple LevelDB-Backed Message Queue in Go

#12

Post by Bogdanovich »

krat0sprakhar wrote: post #3

This couldn't have come at a better time - I was actually looking for a durable message-queue written in Go. Is there any way to read more about the architecture of this system? I find systems like these to be quite fascinating but taking the time to go through the code can sometimes be very time-consuming. It would be awesome if more projects have a writeup as detailed as cockroachdb[0]! Aside: There used to be a si…

It's really simple. Each queue is a separate leveldb database on disk. Messages are stored as key/value using incremental ids. Head and tail of the queue are kept in memory and get initialized on startup via db scan.

Re: Siberite: A Simple LevelDB-Backed Message Queue in Go

#13

Post by biot »

eis wrote: post #6

Why would you choose a LSM Tree based storage mechanism for a message queue? The only reason I can come up with would be because it's a read-to-use library you can just plug in which gives OK performance and some handy features because you can use the KV store for other things. But it doesn't scale well and backups with LevelDB are not really easy either (close DB, copy all files). Message queues when they are ordere…

And you can find a fantastic list of questions about queue guarantees/properties here: https://news.ycombinator.com/item?id=8709146

Re: Siberite: A Simple LevelDB-Backed Message Queue in Go

#14

Post by biokoda »

Bogdanovich wrote: post #11
eis wrote: post #6

Why would you choose a LSM Tree based storage mechanism for a message queue? The only reason I can come up with would be because it's a read-to-use library you can just plug in which gives OK performance and some handy features because you can use the KV store for other things. But it doesn't scale well and backups with LevelDB are not really easy either (close DB, copy all files). Message queues when they are ordere…

Yes, goleveldb was chosen because it's a ready to use library with a decent write and read performance, and no external non-Go dependencies. It can also be used to store multiple consumers offsets in future. Regarding provided guarantees, with simple 'get work_queue' reads it provides at-most-once delivery. With two phase reliable reads 'get work_queue/open', 'get work_queue/close' it provides at-least-once delivery…

Why is it the queues responsibility to store consumer offsets? Consumer is the only side that knows how far along his processing is. Why is the queue storing this data, when all the consumer has to do is tell it: send me events for topic X from point P forward.

Re: Siberite: A Simple LevelDB-Backed Message Queue in Go

#15

Post by dwenzek »

Bogdanovich wrote: post #11
eis wrote: post #6

Why would you choose a LSM Tree based storage mechanism for a message queue? The only reason I can come up with would be because it's a read-to-use library you can just plug in which gives OK performance and some handy features because you can use the KV store for other things. But it doesn't scale well and backups with LevelDB are not really easy either (close DB, copy all files). Message queues when they are ordere…

Yes, goleveldb was chosen because it's a ready to use library with a decent write and read performance, and no external non-Go dependencies. It can also be used to store multiple consumers offsets in future. Regarding provided guarantees, with simple 'get work_queue' reads it provides at-most-once delivery. With two phase reliable reads 'get work_queue/open', 'get work_queue/close' it provides at-least-once delivery…

I'm puzzled by your mention of consumer offsets.

Indeed, either Siberite is a queue system which purpose is to dispatch each message to one and only one consumer for further processing and which requires the consumers to acknowledge fully processed messages ;

or Siberite is a journal system (in the spirit of Kafka) which purpose is to replay the full log to any consumer asking for it and which offers the consumer a watermark mechanism to keep track of their progress.

In the former case, the queue system is responsible of what to do in case of a missing or late acknowledgement (choosing between "at least once" or "at most once" message delivering). In the later case, the consumers are responsible of how to maintain an atomic view of message consumption and message processing (for instance using a transaction to persist an offset with a state).

Re: Siberite: A Simple LevelDB-Backed Message Queue in Go

#16

Post by rakoo »

eis wrote: post #6

Why would you choose a LSM Tree based storage mechanism for a message queue? The only reason I can come up with would be because it's a read-to-use library you can just plug in which gives OK performance and some handy features because you can use the KV store for other things. But it doesn't scale well and backups with LevelDB are not really easy either (close DB, copy all files). Message queues when they are ordere…

A LSM tree is actually a good idea if you think about it.

The R/W patterns for a message queue are simple:

- Messages are key/value

- key is an autoincrementing id

- Writes are at the end, Reads are from the beginning

- Once a message is processed, it's deleted

So in practice this means that the items are written in an append-only fashion, get merged in bigger chunks, and then get progressively deleted. So at higher levels you don't see the huge latencies due to compaction because all records are deleted. Knowing that keys are only incrementing could also lead to a simple optimization: the compaction phase can be a simple concatenation of files.

So you get an append-only system that progressively removes older entries as they are deleted without resorting to mad science hackery [1]. Why didn't it work for InfluxDB ? All I can guess is that individual entries for each series are all mixed together (InfluxDB wants to be able to manage many series with many tags) and older entries are not deleted as frantically, so you get the latencies we all know with compaction and unpredictable reads.

Now, this is purely theoretical and of course further experimentations are needed to make sure this is correct, but LSM is in my opinion a correct pattern here.

[1] https://gist.github.com/CAFxX/571a1558db9a7b393579

Re: Siberite: A Simple LevelDB-Backed Message Queue in Go

#18

Post by justinsaccount »

krat0sprakhar wrote: post #3

This couldn't have come at a better time - I was actually looking for a durable message-queue written in Go. Is there any way to read more about the architecture of this system? I find systems like these to be quite fascinating but taking the time to go through the code can sometimes be very time-consuming. It would be awesome if more projects have a writeup as detailed as cockroachdb[0]! Aside: There used to be a si…

http://nsq.io/

http://nsq.io/overview/internals.html

The service you are thinking of might be https://github.com/ddollar/godist

You can see the download links on https://github.com/ddollar/forego use it.

Re: Siberite: A Simple LevelDB-Backed Message Queue in Go

#19

Post by dave_ops »

biokoda wrote: post #14
Bogdanovich wrote: post #11

Earlier quoted context omitted.

Yes, goleveldb was chosen because it's a ready to use library with a decent write and read performance, and no external non-Go dependencies. It can also be used to store multiple consumers offsets in future. Regarding provided guarantees, with simple 'get work_queue' reads it provides at-most-once delivery. With two phase reliable reads 'get work_queue/open', 'get work_queue/close' it provides at-least-once delivery…

Why is it the queues responsibility to store consumer offsets? Consumer is the only side that knows how far along his processing is. Why is the queue storing this data, when all the consumer has to do is tell it: send me events for topic X from point P forward.

If you wanted the consumers to be stateless, assuming they otherwise had a deterministic identity, then you could have the queue operate like a journal internally, but present a unified queue API to consumers.

So the queue keeps track of the high-watermark on a per consumer basis and all the consumer has to do is show up, tell the queue its deterministic name/id (might be driven by imaging, configuration, or SDN), and the queue will serve up the next new item that consumer hasn't seen yet.

This would be handy for really dynamic transient worker topologies because it keeps the mutable state and state tracking concerns entirely outside the transient worker.

That said, I still wouldn't use LevelDB. Unless I was expecting to do multi-attribute range queries or something (now we're well outside queue territory), but even then you're still folding over the data for knowable start/end markers and a linear scan over a binary term file will be faster than the multiple seeks + segment scans that LevelDB requires.

Re: Siberite: A Simple LevelDB-Backed Message Queue in Go

#20

Post by dave_ops »

Bogdanovich wrote: post #12
krat0sprakhar wrote: post #3

This couldn't have come at a better time - I was actually looking for a durable message-queue written in Go. Is there any way to read more about the architecture of this system? I find systems like these to be quite fascinating but taking the time to go through the code can sometimes be very time-consuming. It would be awesome if more projects have a writeup as detailed as cockroachdb[0]! Aside: There used to be a si…

It's really simple. Each queue is a separate leveldb database on disk. Messages are stored as key/value using incremental ids. Head and tail of the queue are kept in memory and get initialized on startup via db scan.

Why don't you just store the head and tail as K/V entries? You have a durable K/V store at your disposal.
Post reply on HN