41–50 of 72 posts
I built a (moderately successful) startup using kdb+. It was what I knew and it helped us build robust product, quickly. But as we scaled we had to rewrite in FOSS to ensure we could scale the team. Agree with all the recommendations, except I think kx should open source the platform. This will attract the breed of developer that will want to contribute back to the ecosystem with improvements and tools.
Greenfield development though would use Python.
I actually quit a quant trading job after 2 weeks because they used kdb+. I could use it but the experience was so bad... People could complain about abysmal language design or debugging but what I found the most frustration in the coding conventions that they had (or had not), and I think the language and the community play a big role there. But also the company culture: I asked why the code was so poorly documented…
I find it much more likely that you couldn’t understand their code and quit out of frustration.
If you were a highly skilled quant dev and this was a good seat, quitting after two weeks would have been a disaster to manage the next transition given the terms these contracts always have.
I actually quit a quant trading job after 2 weeks because they used kdb+. I could use it but the experience was so bad... People could complain about abysmal language design or debugging but what I found the most frustration in the coding conventions that they had (or had not), and I think the language and the community play a big role there. But also the company culture: I asked why the code was so poorly documented…
Their pykx integration is going a long way to fix some of the gaps in: - charting - machine learning/statsmodels - html processing/webscrapes Because for example you can just open a Jupyter Notebook and do: import pykx as kx df = kx.q(“select from foo where bar”) plt.plot(df[“x”], df[“y”]) It’s truly an incredibly seamless and powerful integration. You get the best of both worlds and it may be the saving feature of t…
---
[1] RDB - RAM DB (recent in-memory data), IDB (Intraday DB - recent data which doesn't fit into RAM), HDB - Historical DB (usually partitioned by date or other time-based or integral column).
Earlier quoted context omitted.
Their pykx integration is going a long way to fix some of the gaps in: - charting - machine learning/statsmodels - html processing/webscrapes Because for example you can just open a Jupyter Notebook and do: import pykx as kx df = kx.q(“select from foo where bar”) plt.plot(df[“x”], df[“y”]) It’s truly an incredibly seamless and powerful integration. You get the best of both worlds and it may be the saving feature of t…
I think this will only work with regular qSQL on a specific database node, i.e. RDB, IDB, HDB[1]. It will be much harder for a mortal Python developer to use Functional qSQL[2] which will join/merge/aggregate data from all these nodes. The join/merge/aggregation is usually application-specific and done on some kind of gateway node(s). Querying each of them is slightly different, with different keys and secondary indi…
I think you touch on something really interesting which is the kink in the kdb+ learning curve when you go from really simple functions,tables, etc. to actually building a performant kdb architecture.
Earlier quoted context omitted.
The article calls out Python and DuckDB as possible successors. I remember being very impressed by Kdb+ (went to their meetups in Chicago). Large queries ran almost instantaneously. The APL like syntax was like a magic incantation that only math types were privy to. The salesperson mentioned KdB was so optimized that it fit in the L1 cache of a processor of the day. Fast forward 10 years. I’m doing the same thing tod…
Do you use duckdb for real-time queries or just historical? You mentioned parquet but afaik it's not well suited for appending data.
Also, not sure what you mean by Parquet is not good at appending? On the contrary, Parquet is designed for an append-only paradigm (like Hadoop back in the day). You can just drop a new parquet file and it’s appended.
If you have 1.parquet, all you have you to do is drop 2.parquet in the same folder or Hive hierarchy. Then query>
Select * from ‘*.parquet’
In practice we use a directory structure called Hive partitioning, which helps DuckDB do partition elimination to skip over irrelevant partitions, making it even faster.
https://duckdb.org/docs/data/partitioning/hive_partitioning
Parquet is great for appending!
Now, it's not so good at updating because it's a write-once format (not read-write). To update a single record in a Parquet file entails regenerating the entire Parquet file. So if you have late-arriving updates, you need to do extra work to identify the partition involved and overwrite. Either that or use bitemporal modeling (add data arrival timestamp [1]) and do a latest date clause in your query (entailing more compute). If you have a scenario where existing data changes a lot, Parquet is not a good format for you. You should look into Timescale (time-series database based on Postgres)
Earlier quoted context omitted.
The article calls out Python and DuckDB as possible successors. I remember being very impressed by Kdb+ (went to their meetups in Chicago). Large queries ran almost instantaneously. The APL like syntax was like a magic incantation that only math types were privy to. The salesperson mentioned KdB was so optimized that it fit in the L1 cache of a processor of the day. Fast forward 10 years. I’m doing the same thing tod…
We use DuckDB similarly but productionize by writing pyarrow code. All the modern tools (DuckDB, pyarrow, polars) are fast enough if you store your data well (parquet), though we work with not quite “big data” most of the time. It’s worth remembering that all the modern progress builds on top of years of work by Wes McKinney & co (many, many contributors).
Earlier quoted context omitted.
The article calls out Python and DuckDB as possible successors. I remember being very impressed by Kdb+ (went to their meetups in Chicago). Large queries ran almost instantaneously. The APL like syntax was like a magic incantation that only math types were privy to. The salesperson mentioned KdB was so optimized that it fit in the L1 cache of a processor of the day. Fast forward 10 years. I’m doing the same thing tod…
Do you use duckdb for real-time queries or just historical? You mentioned parquet but afaik it's not well suited for appending data.
S3 is high-throughput but also high-latency storage. It's good for bulk reads, but not random reads, and querying Parquet involves random reads. Parquet on S3 is ok for batch jobs (like Spark jobs) but it's very slow for interactive queries (Presto, Athena, DuckDB).
The solution is to store Parquet on low-latency storage. S3 has something called S3 Express Zones (which is low-latency S3, costs slightly more). Or EBS, which is block storage that doesn't suffer from S3's high latency.
Earlier quoted context omitted.
The article calls out Python and DuckDB as possible successors. I remember being very impressed by Kdb+ (went to their meetups in Chicago). Large queries ran almost instantaneously. The APL like syntax was like a magic incantation that only math types were privy to. The salesperson mentioned KdB was so optimized that it fit in the L1 cache of a processor of the day. Fast forward 10 years. I’m doing the same thing tod…
Do you use duckdb for real-time queries or just historical? You mentioned parquet but afaik it's not well suited for appending data.