Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

IBM to acquire Confluent

confluent.io

Post reply on HN
Print view View original

251–260 of 374 posts

Re: IBM to acquire Confluent

#251

Post by kerblang »

CharlieDigital wrote: post #99
alexjplant wrote: post #62

Earlier quoted context omitted.

Ah yes, and every consumer should just do this in a while (true) loop as producers write to it. Very efficient and simple with no possibility of lock contention or hot spots. Genius, really.

I've implemented a distributed worker system on top of this paradigm. I used ZMQ to connect nodes and the worker nodes would connect to an indexer/coordinator node that effectively did a `SELECT FROM ORDER BY ASC`. It's easier than you may think and the bits here ended up with probably - Coordinator node ingests from a SQL table - There is a discriminator key for each row in the table for ordering by stacking into an…

Kafka is really not intended to improve on this. Instead, it's intended for very high-volume ETL processing, where a classical message queue delivering records would spend too much time on locking. Kafka is hot-rodding the message queue design and removing guard rails to get more messages thru faster.

Generally I say, "Message queues are for tasks, Kafka is for data." But in the latter case, if your data volume is not huge, a message queue for async ETL will do just fine and give better guarantees as FIFO goes.

In essence, Kafka is a very specialized version of much more general-purpose message queues, which should be your default starting point. It's similar to replacing a SQL RDBMS with some kind of special NoSQL system - if you need it, okay, but otherwise the general-purpose default is usually the better option.

Re: IBM to acquire Confluent

#252

Post by ethbr1 »

paxys wrote: post #110
notepad0x90 wrote: post #28

This is so fascinating to me. I mean how IBM keeps taking over other companies, but they consistently deliver low quality/bottom-tier services and products. Why do they keep doing the same thing again and again? How are they generating actual revenue this way? Ok, so does anyone remember 'Watson'? It was the chatgpt before chatgpt. they built it in house. Why didn't they compete with OpenAI like Google and Anthropic…

Everything will make sense when you realize that IBM is a consulting company. They don't care about building great products. In fact building self-serve products will directly take away from their consulting revenue. They instead need to be good at marketing and selling their services. Watson was exactly that - a marketing demo that got them in the news cycle and helped them sell a giant wave of contracts under a sin…

>> Ok, so does anyone remember 'Watson'? [...] Why didn't they compete with OpenAI like Google and Anthropic are doing, with in-house tools?

> Everything will make sense when you realize that IBM is a consulting company.

This and.

The 'and' being that consulting companies, in their DNA, build solutions for their customers.

Which is a very different business than building products for all users.

Not least because the former is guided by understanding a customer's requirements, while the later is having a strong intuition (backed up by market fit) about what all users want.

I'm pretty sure there might not be a full end user capable (in the sense of design-build-iterate) product team in IBM at this point.

Mostly because I don't think they've any middle/upper management that can think that way. They've got the engineers!

Re: IBM to acquire Confluent

#253

Post by ethbr1 »

ericol wrote: post #218
paxys wrote: post #117

Earlier quoted context omitted.

Watson was nothing like ChatGPT. The first iteration was a system specifically built to play Jeopardy. It did some neat stuff with NLP and information retrieval, but it was all still last generation AI/ML technology. It then evolved into a brand that IBM used to sell its consulting services. The product itself was a massive failure because it had no real applications and was too weak as a general purpose chat bot.

I had no idea about what Watson was initially meant to solve. I do remember they tried to sell it - at least in the meeting I went - as a general purpose chatbot. I did try briefly to understand how to use it, but the documentation was horrendous (As in, "totally devoid of any technical information")

Watson was intended to solve fuzzy optimization problems.

Unfortunately, the way it solved fuzzy was 'engineer the problem to fit Watson, then engineer the output to be usable.'

Which required every project to be a huge custom implementation lift. Similar to early Palantir.

Re: IBM to acquire Confluent

#254

Post by rw_grim »

jhallenworld wrote: post #227
framebit wrote: post #219

Earlier quoted context omitted.

Fun fact: there's an IBM/Lotus Sametime theme song. https://www.youtube.com/watch?v=daitUOzVpvc The lyrics rhyme "PC" with "easy."

That's awesome. Before ~2007 they allowed you to use open-source Pidgin to connect to the Domino servers. A friend of mine and I used it to make a bot: if you sametimed me, you got Zork. It reminds me of another IBM IT rule: they wanted your chat history (and email) older than two years to be all deleted for legal liability reasons. It was important to save your sametime chat history (an XML file) and export your ema…

We (Pidgin) probably could have kept that code working if we had access to a server. But we didn't, and it bitrot, so we've since archived it to https://github.com/pidgin/retro-prpl/tree/main/sametime

Re: IBM to acquire Confluent

#255

Post by CharlieDigital »

kerblang wrote: post #251
CharlieDigital wrote: post #99

Earlier quoted context omitted.

I've implemented a distributed worker system on top of this paradigm. I used ZMQ to connect nodes and the worker nodes would connect to an indexer/coordinator node that effectively did a `SELECT FROM ORDER BY ASC`. It's easier than you may think and the bits here ended up with probably - Coordinator node ingests from a SQL table - There is a discriminator key for each row in the table for ordering by stacking into an…

Kafka is really not intended to improve on this. Instead, it's intended for very high-volume ETL processing, where a classical message queue delivering records would spend too much time on locking. Kafka is hot-rodding the message queue design and removing guard rails to get more messages thru faster. Generally I say, "Message queues are for tasks, Kafka is for data." But in the latter case, if your data volume is no…

Of course this is not the same as Kafka, but the comment I'm replying to:

    > Ah yes, and every consumer should just do this in a while (true) loop as producers write to it. Very efficient and simple with no possibility of lock contention or hot spots. Genius, really.
Seemed to imply that it's not possible to build a high performance pub/sub system using a simple SQL select. I do not think that is true and it is in fact fairly easy to build a high performance pub/sub system with a simple SQL select. Clearly, this design as proposed is not the same as Kafka.

Re: IBM to acquire Confluent

#256

Post by ycombinatrix »

ahoka wrote: post #140
Romario77 wrote: post #111

Earlier quoted context omitted.

pull vs push. Plus if you start storing the last timestamp so you only select the delta and if you start sharding your db and dealing with complexities of having different time on different tables/replication issues it quickly becomes evident that Kafka is better in this regard. But yeah, for a lot of implementations you don't need streaming. But for pull based apps you design your architecture differently, some thin…

Funny you mention that, because Kafka consumers actually pull messages.

Not by busy waiting in a loop on a database query though.

Re: IBM to acquire Confluent

#257

Post by selcuka »

Lu2025 wrote: post #231
notepad0x90 wrote: post #28

This is so fascinating to me. I mean how IBM keeps taking over other companies, but they consistently deliver low quality/bottom-tier services and products. Why do they keep doing the same thing again and again? How are they generating actual revenue this way? Ok, so does anyone remember 'Watson'? It was the chatgpt before chatgpt. they built it in house. Why didn't they compete with OpenAI like Google and Anthropic…

> they consistently deliver low quality/bottom-tier services and products I worked with IBMers. The main priority for a lot of them is to ensure continuous employment for themselves and their buddies. They'd add unnecessary complexity to a product to stretch out the development for another couple of years. And they work at leisure pace for tech. Actual 9 to 5, many coffee breaks. They can't compete.

I worked with IBM several decades ago for a customer project, and the solution suggested by an IBM'er for backing up a NoSQL database (Lotus Notes) on a daily basis was to translate and migrate the data to a relational one (DB2), then use a DB2 tape backup system to back it up.

When I pointed out that this was a stupid way to do it, they openly told me that they just wanted to sell DB2.

Re: IBM to acquire Confluent

#259

Post by parpfish »

Lammy wrote: post #237
Lu2025 wrote: post #231

Earlier quoted context omitted.

> they consistently deliver low quality/bottom-tier services and products I worked with IBMers. The main priority for a lot of them is to ensure continuous employment for themselves and their buddies. They'd add unnecessary complexity to a product to stretch out the development for another couple of years. And they work at leisure pace for tech. Actual 9 to 5, many coffee breaks. They can't compete.

> And they work at leisure pace for tech. Actual 9 to 5, many coffee breaks. Ultra-based. We should all be so lucky.

You mean you DONT work a leisurely 6-8 hour day with breaks? I thought everybody did that until there was some urgent firefighting

Re: IBM to acquire Confluent

#260

Post by alexjplant »

CharlieDigital wrote: post #255
kerblang wrote: post #251

Earlier quoted context omitted.

Kafka is really not intended to improve on this. Instead, it's intended for very high-volume ETL processing, where a classical message queue delivering records would spend too much time on locking. Kafka is hot-rodding the message queue design and removing guard rails to get more messages thru faster. Generally I say, "Message queues are for tasks, Kafka is for data." But in the latter case, if your data volume is no…

Of course this is not the same as Kafka, but the comment I'm replying to: > Ah yes, and every consumer should just do this in a while (true) loop as producers write to it. Very efficient and simple with no possibility of lock contention or hot spots. Genius, really. Seemed to imply that it's not possible to build a high performance pub/sub system using a simple SQL select. I do not think that is true and it is in fac…

No, I implied that implementing pub/sub with just a select statement is silly because it is. Your implementation accounts for the downfalls of this approach with smart design using a message queue and intelligent locking semantics. Parent of my comment was glib and included none of this.
Post reply on HN