Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ultra-low-latency, batching and concurrent queue for IPC in Java

github.com

Post reply on HN
Print view View original

11–20 of 76 posts

Re: Ultra-low-latency, batching and concurrent queue for IPC in Java

#11

Post by kasey_junk »

capmorganbih wrote: post #8

Our kafka isn’t reliable enough. I need to write data on disk before flushing it to kafka. Can I use this lis to write data to disk and then consume inside same jvm. I need data to live through restarts

You could but its not purpose built for that. You'd probably be happier using some other memory mapped file format for that.

Re: Ultra-low-latency, batching and concurrent queue for IPC in Java

#12

Post by capmorganbih »

sriram_malhar wrote: post #9
capmorganbih wrote: post #8

Our kafka isn’t reliable enough. I need to write data on disk before flushing it to kafka. Can I use this lis to write data to disk and then consume inside same jvm. I need data to live through restarts

Can you explain what your issue is with Kafka? What makes it not reliable enough?

Kafka is good. The problem is we don’t have a dedicated person to manage it so sometimes we have kafka outages

Re: Ultra-low-latency, batching and concurrent queue for IPC in Java

#13

Post by capmorganbih »

kasey_junk wrote: post #11
capmorganbih wrote: post #8

Our kafka isn’t reliable enough. I need to write data on disk before flushing it to kafka. Can I use this lis to write data to disk and then consume inside same jvm. I need data to live through restarts

You could but its not purpose built for that. You'd probably be happier using some other memory mapped file format for that.

Could you please suggest java library for this if you know one?

Re: Ultra-low-latency, batching and concurrent queue for IPC in Java

#14

Post by kasey_junk »

capmorganbih wrote: post #13
kasey_junk wrote: post #11

Earlier quoted context omitted.

You could but its not purpose built for that. You'd probably be happier using some other memory mapped file format for that.

Could you please suggest java library for this if you know one?

https://github.com/OpenHFT/Chronicle-Queue?tab=readme-ov-fil...

Re: Ultra-low-latency, batching and concurrent queue for IPC in Java

#15

Post by papercrane »

smarx007 wrote: post #3
WeaselNo7 wrote: post #2

Weird to see this here! I've used CoralBlocks in the low-latency trading domain previously. Highly recommend. The API is kind, they're very responsive, and the latency is exceptional (and comes with all the basics like thread pinning built-in for convenience)

How does it compare to LMAX Disruptor if you have any experience with both?

They're both similar in design, the main difference is Coral Queue can be used for IPC between JVMs, using a mmap'd file.

Re: Ultra-low-latency, batching and concurrent queue for IPC in Java

#16

Post by kllrnohj »

https://github.com/coralblocks/CoralRing/blob/1168b047e0183c...

Am I missing something here or does the BlockingRingConsumer not actually block? And worse doesn't it just return garbage if poll is called without first checking availableToPoll?

The example sure looks like it... https://github.com/coralblocks/CoralRing/blob/main/src/main/...

Which if so isn't this like 1/4th a library for doing IPC? It doesn't seem to do much itself

Re: Ultra-low-latency, batching and concurrent queue for IPC in Java

#17

Post by arnold_palmur »

papercrane wrote: post #15
smarx007 wrote: post #3

Earlier quoted context omitted.

How does it compare to LMAX Disruptor if you have any experience with both?

They're both similar in design, the main difference is Coral Queue can be used for IPC between JVMs, using a mmap'd file.

See https://github.com/real-logic/aeron (also from the creator of the disruptor)

Re: Ultra-low-latency, batching and concurrent queue for IPC in Java

#19

Post by jojohohanon »

capmorganbih wrote: post #12
sriram_malhar wrote: post #9

Earlier quoted context omitted.

Can you explain what your issue is with Kafka? What makes it not reliable enough?

Kafka is good. The problem is we don’t have a dedicated person to manage it so sometimes we have kafka outages

There’s a whole conference in Vegas next week that want to sell you something.

Re: Ultra-low-latency, batching and concurrent queue for IPC in Java

#20

Post by dingi »

capmorganbih wrote: post #8

Our kafka isn’t reliable enough. I need to write data on disk before flushing it to kafka. Can I use this lis to write data to disk and then consume inside same jvm. I need data to live through restarts

Write your data to a MognoDB database. Then use the Kafka Mongo connector to pull data from MongoDB to Kafka.
Post reply on HN