I feel kdb is like the equivalent of a drag racer - useless generally. Great at a one (or few) things in very limited environments.
The future of kdb+?
21–30 of 72 posts
Re: The future of kdb+?
#22Kdb+ seems really cool and I've learned it a little bit for fun along with APL. It would actually be pretty cool for a lot of uses in my industry too, but the price is just crazy. We can't pay like $100k/cpu or whatever it is that the financial banks pay. So they've basically ignored a HUGE amount of potential customers.
They found a niche that can pay the price to have an innovative product. I believe they did the right thing, after all it is not a product trying to solve all problems in the world. Other people could learn from their techniques and do the same for other areas and languages.
Not quite where I was going. The product does seem to be good and there is demand for it in many industries I'd think, but instead of using discriminatory pricing and having people pay less that have a much lower ability to pay, they just ignore the segment entirely. Maybe they know what they're doing though. It's a shame I don't get to use it at work
Re: The future of kdb+?
#23Is it still possible to learn from scratch and make big bucks developing for kdb+ (k/q)? I remember seeing an open position a few years ago which paid like 1MM per year. Astounding.
Re: The future of kdb+?
#24Not 100% sure why it’s often idolized on HN. We’ve maintained a financial exchange w/ margining for 8 years with it, and I guarantee you that everyone was more than relieved - customers and employees alike, once we were able to lift and shift the whole thing to Java. The readability and scalability is abysmal as soon as you move on from a quant desk scenario (which everyone agrees, it is more than amazing at.. panda…
I think the adulation is mainly driven by the a few things: 1. it was fast by a huge margin for its time 2. the reason for its speed is the language behind it 3. it uses an esoteric language and still attains success 4. the core engine is implemented using surprisingly few lines of code 5. the core has been written and maintained by one person All of these are things I've heard so I can't claim it's 100% true but I'm…
Compared to similar dynamic scripting languages, Q is very vast. Compared to statically compiled languages, it can be surprisingly competitive, but is usually slower. The truly distinctive thing about Q is its efficiency as a user interface: at a REPL you can rattle off a short sequence of characters to transform and interrogate large datasets at interactive speeds and flexibly debug complex distributed systems live. In the right hands, it's a stunningly effective rapid-application-development tool (the above "quant desk scenario"); this was perhaps even more true in the k2 days when it was possible to build ugly but blisteringly fast and utilitarian data-bound GUIs for K programs in a few lines of code. There's certainly an abundance of romanticism and mythology surrounding it, but some of the claims are real and enduringly unmatched.
Re: The future of kdb+?
#25I actually quit a quant trading job after 2 weeks because they used kdb+. I could use it but the experience was so bad...
People could complain about abysmal language design or debugging but what I found the most frustration in the coding conventions that they had (or had not), and I think the language and the community play a big role there. But also the company culture: I asked why the code was so poorly documented (no comments, single letter parameters, arcane function names). "We understand it after some time and this way other teams cannot use our ideas."
Overall, their whole stack was outdated and ofc they could not do very interesting things with a tool such as Q. For example, they plotted graphs by copying data from qStudio to Excel...
The only good thing was they did not buy the docker / k8s bs and were deploying directly on servers. It makes sense that quants should be able to fix things in production very quickly but I think it would also make sense for web app developers not to wait 10 minutes (and that's when you have good infra) to see a fix in production.
I have a theory on why quants actually like kdb: it's a good *weapon*. It serves some purpose but I would not call it a *tool* as building with it is tedious. People like that it just works out of the box. But although you can use a sword to drive nails, it is not its purpose.
Continuing on that theory, LISP (especially Racket) would be the best *tool* available as it is not the most powerful language out of the box but allows to build a lot of abstractions with features to modify the language itself. C++ and Python are just great programming languages as you can build good software on them, Python being also a fairly good weapon.
Q might give the illusion of being the best language to explore quant data, but that's just because quants do not invest enough time into building good software and using good tools. When you actually master a Python IDE, you are definitely more productive than any Q programmer.
And don't get me started on performance (the link covers it anyway even though the prose is bad).
Re: The future of kdb+?
#26Not 100% sure why it’s often idolized on HN. We’ve maintained a financial exchange w/ margining for 8 years with it, and I guarantee you that everyone was more than relieved - customers and employees alike, once we were able to lift and shift the whole thing to Java. The readability and scalability is abysmal as soon as you move on from a quant desk scenario (which everyone agrees, it is more than amazing at.. panda…
I'm very curious about this rewrite in Java, especially the orders of magnitude improvement. That sounds extremely impressive, and something that I wouldn't have considered possible. Can you share a bit more about how this performance improvement is achieved?
Re: The future of kdb+?
#27A few corrections to the article. 1. ClickHouse is not a new technology — it has been open-source since 2016 and in development since 2009. 2. ClickHouse can do all three use cases: historical and real-time data, distributed and local processing (check clickhouse-local and chdb). 3. ClickHouse was the first SQL database with ASOF JOIN in the main product (in 2019) - after kdb+, which is not SQL.
I run a data consultancy with a big focus on ClickHouse. There is a lot of interest in replacing KDB with it. I’ve had probably 10 conversations with companies looking at a migration.
Tellingly, nobody has pulled the trigger on a migration yet as I think it’s a big call with all of the integrations that KDB sprouts, but it definetly feels like the spiritual successor.
Re: The future of kdb+?
#28Earlier quoted context omitted.
I think the adulation is mainly driven by the a few things: 1. it was fast by a huge margin for its time 2. the reason for its speed is the language behind it 3. it uses an esoteric language and still attains success 4. the core engine is implemented using surprisingly few lines of code 5. the core has been written and maintained by one person All of these are things I've heard so I can't claim it's 100% true but I'm…
Compared to similar dynamic scripting languages, Q is very vast. Compared to statically compiled languages, it can be surprisingly competitive, but is usually slower. The truly distinctive thing about Q is its efficiency as a user interface : at a REPL you can rattle off a short sequence of characters to transform and interrogate large datasets at interactive speeds and flexibly debug complex distributed systems live…
Python in a Notebook is “REPL like” and much more modern.
And though I agree low code is important, Streamlit or Dash are a much more fully featured and open way to do that.
I agree KDB has a good development workflow, but I think the same is available in an open source stack like ClickHouse + Python + Jupyter.
Re: The future of kdb+?
#29Earlier quoted context omitted.
I agree that the vertical integration capability of kdb+/Q is amazing, and it is beyond comprehension why Kx themselves don’t effectively leverage it. Kx Platform appears to be mostly written in Java, and the API’s callable from Q are not very well documented. My team and I find the dashboards product is difficult to use, and there are some nasty bugs that cause frequent editor crashes for dashboards of moderate comp…
I'm a little too close to be throwing stones, but without going into specifics I believe that key leaders at Kx do not properly appreciate the unique characteristics and benefits of their own technology, and are trapped in a mindset of trying to make their products more similar to their competition in order to make sales and marketing easier. In the process, they discard their competitive advantage. Tale as old as ti…
I think it is very difficult to judge how much of an advantage your competitive advantage actually is. It’s very easy to look at the things which directly cost you sales and conclude that those are the things you need to fix rather than doubling down on your strengths. The most common way to avoid that is to go too far in the other direction and become convinced that your niche technology is vastly superior to the mainstream choice and anyone who rejects you for your shortcomings is just shortsighted and wrong.
From the outside it’s always seemed that kdb fans tend to land in the second camp, and I think it would be understandable for Kx to have overcorrected into undervaluing their work instead.
Re: The future of kdb+?
#30I actually quit a quant trading job after 2 weeks because they used kdb+. I could use it but the experience was so bad... People could complain about abysmal language design or debugging but what I found the most frustration in the coding conventions that they had (or had not), and I think the language and the community play a big role there. But also the company culture: I asked why the code was so poorly documented…
Their pykx integration is going a long way to fix some of the gaps in:
- charting
- machine learning/statsmodels
- html processing/webscrapes
Because for example you can just open a Jupyter Notebook and do:
import pykx as kx
df = kx.q(“select from foo where bar”)
plt.plot(df[“x”], df[“y”])