Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The future of kdb+?

timestored.com

Post reply on HN
Print view View original

51–60 of 72 posts

Re: The future of kdb+?

#51

Post by BWStearns »

lordnacho wrote: post #36

I thought I'd throw in TimeScale. It's a postgres extension, so all your SQL stuff is just the same (replication, auth, etc). It's also a column store, with compression. Runs super fast, I've used it in a couple of financial applications. Huge amounts of tick data, all coming down to your application nearly as fast as the hardware will allow. Good support, the guys on Slack are responsive. No, I don't have shares in…

I am using Timescale at work and I like them a lot. If your data is nicely structured it's a breeze. But my data is kind of pathological (source can just change the structure and I gotta put up with it), so I'd honestly use Influx in a heartbeat if their pricing wasn't totally insane.

Re: The future of kdb+?

#52

Post by wenc »

wenc wrote: post #31
Labo333 wrote: post #25

I actually quit a quant trading job after 2 weeks because they used kdb+. I could use it but the experience was so bad... People could complain about abysmal language design or debugging but what I found the most frustration in the coding conventions that they had (or had not), and I think the language and the community play a big role there. But also the company culture: I asked why the code was so poorly documented…

The article calls out Python and DuckDB as possible successors. I remember being very impressed by Kdb+ (went to their meetups in Chicago). Large queries ran almost instantaneously. The APL like syntax was like a magic incantation that only math types were privy to. The salesperson mentioned KdB was so optimized that it fit in the L1 cache of a processor of the day. Fast forward 10 years. I’m doing the same thing tod…

I just realized all the data tools I use are animals.

Pandas

Polars (polar bear)

DuckDB

Python

Re: The future of kdb+?

#53

Post by thaufeki »

shrubble wrote: post #16
gricardo99 wrote: post #7

Get a free version out there that can be used for many things… I think this has been the biggest impediment to kdb+ gaining recognition as a great technology/product and growing amongst the developer community. Having used kdb+ extensively in the finance world for years, I became a convert and a fan. There’s an elegance in its design and simplicity that seems very much rooted in the Unix philosophy. After I left fina…

Were any of the open source versions such as ngn/k or Kerf etc. usable for you?

Well you would have to know how to code in k, not just q, the syntax is a lot more terse and there are a lot of features missing

Re: The future of kdb+?

#54

Post by belfthrow »

Labo333 wrote: post #25

I actually quit a quant trading job after 2 weeks because they used kdb+. I could use it but the experience was so bad... People could complain about abysmal language design or debugging but what I found the most frustration in the coding conventions that they had (or had not), and I think the language and the community play a big role there. But also the company culture: I asked why the code was so poorly documented…

Not surviving more than 2 weeks in a QF role because of kdb, and then suggesting they should rewrite everything to LISP is one of the more HN level recidivous comments I think I have ever seen.

Re: The future of kdb+?

#55

Post by jcul »

gricardo99 wrote: post #7

Get a free version out there that can be used for many things… I think this has been the biggest impediment to kdb+ gaining recognition as a great technology/product and growing amongst the developer community. Having used kdb+ extensively in the finance world for years, I became a convert and a fan. There’s an elegance in its design and simplicity that seems very much rooted in the Unix philosophy. After I left fina…

Isn't there a free version or something?

I had to write some C++ code in the past to send data into kdb and also a decoder for their wire protocol. For both I definitely had a kdb binary to test against.

I just needed to test against it. Maybe Kx gave us a development license or something, it was a good few years ago.

Re: The future of kdb+?

#56

Post by gricardo99 »

menthe wrote: post #15

Not 100% sure why it’s often idolized on HN. We’ve maintained a financial exchange w/ margining for 8 years with it, and I guarantee you that everyone was more than relieved - customers and employees alike, once we were able to lift and shift the whole thing to Java. The readability and scalability is abysmal as soon as you move on from a quant desk scenario (which everyone agrees, it is more than amazing at.. panda…

   And this is the architecture as designed and recommended by the KX consultants that you end up having to hire to “scale”
I think this hits on one of the major shortcomings of how FD/Kx have managed the technology going back 15+ years, IMHO.

Historically it’s the consultants that brought in a lot of income, with each one building ad-hoc solutions for their clients and solving much more complicated enterprise-scale integration and resilience challenges. FD/Kx failed to identify the massive opportunity here, which was to truly invest in R&D and develop a set of common IP, based on robust architectures, libraries and solutions around the core kdb+ product that would be vastly more valuable and appealing to more customers. This could have led to a path where open sourcing kdb+ made sense, if they had a suite of valuable, complementary functionality that they could sell. But instead, they parked their consultants for countless billable hours at their biggest paying customer’s sites and helped them build custom infra around kdb+, reinventing wheels over and over again.

They were in a unique position for decades, with a front row seat to the pain points and challenges of top financial institutions, and somehow never produced a product that came close to the value and utility of kdb+, even though clearly it was only ever going to be a part of a larger software solution.

In fairness they produced the delta suite, but its focus and feature set seemed to be constantly in flux and underwhelming, trying to bury and hide kdb+ behind frustratingly pointless UI layers. The more recent attempts with Kx.ai I’m less familiar with, but seem to be a desperate marketing attempt to latch onto the next tech wave.

They have had some very talented technical staff over the years, including many of their consultants. I just think that if the leadership had embraced the core technology and understood the opportunity to build a valuable ecosystem, with a goal towards FOSS, things could look very different. All hindsight of course :)

Maybe it’s not too late to try that…

Re: The future of kdb+?

#57

Post by andrewl-hn »

lordnacho wrote: post #36

I thought I'd throw in TimeScale. It's a postgres extension, so all your SQL stuff is just the same (replication, auth, etc). It's also a column store, with compression. Runs super fast, I've used it in a couple of financial applications. Huge amounts of tick data, all coming down to your application nearly as fast as the hardware will allow. Good support, the guys on Slack are responsive. No, I don't have shares in…

Another superpower of TimeScale is that it plays nicely with other Postgres extensions. We had a really good experience with it combined with PostGIS. Scenarios like "Show sensors on a map with value graphs for each sensor" can be done in a single query, and it's fast and beautiful.

Re: The future of kdb+?

#58

Post by keithalewis »

socksy wrote: post #38
eru wrote: post #34

Earlier quoted context omitted.

It's not a good filter in that case. I can learn obscure languages just fine, but that doesn't make me any more pleasant to hang out with.

I'm not sure that was ever a requirement in these industries

It is not a requirement. Just a way to weed out people who think they are special snowflakes.

Re: The future of kdb+?

#60

Post by 7thaccount »

wenc wrote: post #48
singhrac wrote: post #32

Earlier quoted context omitted.

We use DuckDB similarly but productionize by writing pyarrow code. All the modern tools (DuckDB, pyarrow, polars) are fast enough if you store your data well (parquet), though we work with not quite “big data” most of the time. It’s worth remembering that all the modern progress builds on top of years of work by Wes McKinney & co (many, many contributors).

Yes Wes McKinney was involved in both Pandas and Parquet and Arrow.

I remember reading something awhile back that when building pandas he was getting a lot of inspiration from things like APL and I assume Kdb+.
Post reply on HN