Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The future of kdb+?

timestored.com

Post reply on HN
Print view View original

31–40 of 72 posts

Re: The future of kdb+?

#31

Post by wenc »

Labo333 wrote: post #25

I actually quit a quant trading job after 2 weeks because they used kdb+. I could use it but the experience was so bad... People could complain about abysmal language design or debugging but what I found the most frustration in the coding conventions that they had (or had not), and I think the language and the community play a big role there. But also the company culture: I asked why the code was so poorly documented…

The article calls out Python and DuckDB as possible successors.

I remember being very impressed by Kdb+ (went to their meetups in Chicago). Large queries ran almost instantaneously. The APL like syntax was like a magic incantation that only math types were privy to. The salesperson mentioned KdB was so optimized that it fit in the L1 cache of a processor of the day.

Fast forward 10 years. I’m doing the same thing today with Python and DuckDB and Jupyter on Parquet files. DuckDB not only parallelizes, it vectorizes. I’m not sure how it benchmarks against kdb+ but the responsiveness of DuckDB at least feels as fast as kdb+ on large datasets. (Though I’m sure kdb+ is vastly more optimized). The difference? DuckDB is free.

Re: The future of kdb+?

#32

Post by singhrac »

wenc wrote: post #31
Labo333 wrote: post #25

I actually quit a quant trading job after 2 weeks because they used kdb+. I could use it but the experience was so bad... People could complain about abysmal language design or debugging but what I found the most frustration in the coding conventions that they had (or had not), and I think the language and the community play a big role there. But also the company culture: I asked why the code was so poorly documented…

The article calls out Python and DuckDB as possible successors. I remember being very impressed by Kdb+ (went to their meetups in Chicago). Large queries ran almost instantaneously. The APL like syntax was like a magic incantation that only math types were privy to. The salesperson mentioned KdB was so optimized that it fit in the L1 cache of a processor of the day. Fast forward 10 years. I’m doing the same thing tod…

We use DuckDB similarly but productionize by writing pyarrow code. All the modern tools (DuckDB, pyarrow, polars) are fast enough if you store your data well (parquet), though we work with not quite “big data” most of the time.

It’s worth remembering that all the modern progress builds on top of years of work by Wes McKinney & co (many, many contributors).

Re: The future of kdb+?

#33

Post by keithalewis »

Labo333 wrote: post #25

I actually quit a quant trading job after 2 weeks because they used kdb+. I could use it but the experience was so bad... People could complain about abysmal language design or debugging but what I found the most frustration in the coding conventions that they had (or had not), and I think the language and the community play a big role there. But also the company culture: I asked why the code was so poorly documented…

[flagged]

Re: The future of kdb+?

#34

Post by eru »

keithalewis wrote: post #33
Labo333 wrote: post #25

I actually quit a quant trading job after 2 weeks because they used kdb+. I could use it but the experience was so bad... People could complain about abysmal language design or debugging but what I found the most frustration in the coding conventions that they had (or had not), and I think the language and the community play a big role there. But also the company culture: I asked why the code was so poorly documented…

[flagged]

It's not a good filter in that case. I can learn obscure languages just fine, but that doesn't make me any more pleasant to hang out with.

Re: The future of kdb+?

#35

Post by cout »

wenc wrote: post #31
Labo333 wrote: post #25

I actually quit a quant trading job after 2 weeks because they used kdb+. I could use it but the experience was so bad... People could complain about abysmal language design or debugging but what I found the most frustration in the coding conventions that they had (or had not), and I think the language and the community play a big role there. But also the company culture: I asked why the code was so poorly documented…

The article calls out Python and DuckDB as possible successors. I remember being very impressed by Kdb+ (went to their meetups in Chicago). Large queries ran almost instantaneously. The APL like syntax was like a magic incantation that only math types were privy to. The salesperson mentioned KdB was so optimized that it fit in the L1 cache of a processor of the day. Fast forward 10 years. I’m doing the same thing tod…

Do you use duckdb for real-time queries or just historical? You mentioned parquet but afaik it's not well suited for appending data.

Re: The future of kdb+?

#36

Post by lordnacho »

I thought I'd throw in TimeScale. It's a postgres extension, so all your SQL stuff is just the same (replication, auth, etc).

It's also a column store, with compression. Runs super fast, I've used it in a couple of financial applications. Huge amounts of tick data, all coming down to your application nearly as fast as the hardware will allow.

Good support, the guys on Slack are responsive. No, I don't have shares in it, I just like it.

Regarding kdb, I've used it, but there are significant drawbacks. Costs a bunch of money, that's a big one. And the language... I mean it's nice to nerd out sometimes with a bit of code golf, but at some point you are going to snap out of it and decide that single characters are not as expressive as they seem.

If your thing is ad-hoc quant analysis, then maybe you like kdb. You can sit there and type little strings into the REPL all day in order to find money. But a lot of things are more like cron jobs, you know you need this particular query run on a schedule, so just turn it into something legible that the next guy will understand and maintain.

Re: The future of kdb+?

#37

Post by nhourcard »

TLDR from the article;

Alternatives (which are open source) to KDB+ are split into two categories:

New Database Technologies (tick data store & ASOF JOIN): Clickhouse & QuestDB

Local Quant Analysis: Python – with DuckDB & Polars

Some personal thoughts:

Q is very expressive, and impressive performance can be extracted from kdb+, but the drawbacks are proprietary formats, vendor lock-in, costs, proprietary language and reliance on external consultants to make the system run adequately, which can increase operational costs.

I'm personally excited to see the open-source alternative stack emerging. Open Source time-series databases and tools like duckdb/polars for data science are a good combination. Storing everything in open formats like Parquet and leveraging high-performance frameworks like Arrow is probably where things are heading.

Seeing some disruption in this industry specifically is interesting; I think it will be beneficial, particularly for developers.

NB: disclosing that I'm from questdb to put thoughts in perspective

Re: The future of kdb+?

#39

Post by puzpuzpuz-hn »

Nice article, thanks for sharing it. It's a pity kdb+ has a DeWitt Clause, so that no one can benchmark it against other databases from the article. I wonder if they have any public benchmarks held by a 3rd-party.

Re: The future of kdb+?

#40

Post by anonu »

I built a (moderately successful) startup using kdb+. It was what I knew and it helped us build robust product, quickly. But as we scaled we had to rewrite in FOSS to ensure we could scale the team.

Agree with all the recommendations, except I think kx should open source the platform. This will attract the breed of developer that will want to contribute back to the ecosystem with improvements and tools.

Post reply on HN