IBM to acquire Confluent

    > ...and internal systems they have to use, whose sole purpose is to make sure nobody does anything

I once had to use Lotus Notes after the company I was at was acquired by the now defunct Computer Sciences Corporation. I decided I would never, ever work for another company that used Lotus Notes.

Nothing inherently wrong with the core product IMHO. The issue is more with Confluent, who have been constantly swinging from hot buzzword to hot buzzword for the last few years in search of growth. Confluent cloud is very expensive, and you still have to deal with a surprising amount of scaling headaches. I have people I consider friends that work there, so I don't want to go too deep into their various missteps, but the Kafka ecosystem has been largely stagnant outside of getting rid of Zookeeper and simplifying operations/deployment. There have been some decent quality of life fixes, but the platform is very expensive, yet if you are really all-in on Kafka, you would be insane to not get support from Confluent- it can break in surprising ways.

So you are stuck with some really terrible tradeoffs- Go with Confluent Cloud, pay a fortune, and still likely have some issues to deal with. Or you could go with Confluent Platform, still have to pay people to operate it, while Confluent the company focuses most of their attention on Cloud and still charges you a fortune. Or you could just go completely OS and forgo anything Confluent and risk being really up the river when something inevitably breaks, or you have to learn the hard way that librdkafka has poor support for a lot of the shiny features discussed in the release notes.

Redpanda has surpassed them from a technical quality perspective, but Kafka has them beat on the ecosystem and the sheer inertia of moving from one platform to another. Kafka for example was built in a time of spinning rust hard disks, and expects to be run on general purpose compute nodes, where Redpanda will actually look at your hardware and optimize the number of threads its spawns for the box it is on- assuming it is going to be the only real app running there, which is true for anything but a toy deployment.

This is my experience from running platform teams and being head of messaging at multiple companies.

I imagine that's done via JIRA tcket/IT before onboarding.

So if they somehow can get past initial device deployment/user account logon, and get other resources IE; slack....well that speaks to how difficult/pointless it would be to get proper VPN/Jira access.

As I read the release, it just sounded like "something something something data, something something something AI."

AI is just the lastest buzzword. Everyone has it, because they have to. Don't look behind the curtain.

I've implemented a distributed worker system on top of this paradigm.

I used ZMQ to connect nodes and the worker nodes would connect to an indexer/coordinator node that effectively did a `SELECT FROM ORDER BY ASC`.

It's easier than you may think and the bits here ended up with probably

    - Coordinator node ingests from a SQL table
    - There is a discriminator key for each row in the table for ordering by stacking into an in-memory list-of-lists
    - Worker nodes are started with _n_ threads
    - Each thread sends a "ready" message to the coordinator and coordinator replies with a "work" message
    - On each cycle, the coordinator advances the pointer on the list, locks the list, and marks the first item in the child list as "pending"
    - When worker thread finishes, it sends a "completed" message to the coordinator and coordinator replies with another "work" message
    - Coordinator unlocks the list the work item originated from and dequeues the finished item.
    - When it reaches the end of the list, it cycles to the beginning of the list and starts over, skipping over any child lists marked as locked (has a pending work item)

Effectively a distributed event loop with the events queued up via a simple SQL query.

Dead simple design, extremely robust, very high throughput, very easy to scale workers both horizontally (more nodes) and vertically (more threads). ZMQ made it easy to connect the remote threads to the centralized coordinator. It was effectively "self balancing" because the workers would only re-queue their thread once it finished work. Very easy to manage, but did not have hot failovers since we kept the materialized, "2D" work queue in memory. Though very rarely did we have issues with this.