Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kdb+ and Python: EmbedPy and PyQ

kx.com

Post reply on HN
Print view View original

31–40 of 40 posts

Re: Kdb+ and Python: EmbedPy and PyQ

#31

Post by kthielen »

yalph wrote: post #30
osrec wrote: post #5

Earlier quoted context omitted.

Your comment made me chuckle. In fact, made me immediately think you were employed by Kx at some point; your profile confirmed this was the case! I think Kx has built a wonderful business, but as an ex MD at a bulge bracket bank, I would not sanction its use any more. It is expensive, and it's difficult to hire good people to work with it (not every guy with kdb on their CV is great). The product is good, but so are…

So what are those alternatives?

How about hobbes?

Re: Kdb+ and Python: EmbedPy and PyQ

#32

Post by yalph »

kthielen wrote: post #31
yalph wrote: post #30

Earlier quoted context omitted.

So what are those alternatives?

How about hobbes?

I will check thanks but wnated to learn from him. I do not think there are many alternatives by the way. Is this one a full alternative? Does it provide all functionalities of kdb?

Re: Kdb+ and Python: EmbedPy and PyQ

#33

Post by yalph »

smnrchrds wrote: post #17
bkeroack wrote: post #7

I like it when kdb is mentioned because then I can post this link: https://github.com/KxSystems/kdb/blob/master/c/c/k.h#L96 It's one of my all-time favorites. A window into a certain type of mind.

I imagine this looks more intuitive than normal code to a mathematician: They are no strangers to complex notations and they prefer the brevity they offer.

Sort of a mathematician here, it looks like a disease.

Re: Kdb+ and Python: EmbedPy and PyQ

#34

Post by kthielen »

yalph wrote: post #32
kthielen wrote: post #31

Earlier quoted context omitted.

How about hobbes?

I will check thanks but wnated to learn from him. I do not think there are many alternatives by the way. Is this one a full alternative? Does it provide all functionalities of kdb?

It’s better in a lot of ways. It has a type system, can produce much better code, can do cross-process compilation (for multi-process IPC). The FFI/binding process is much simpler, there are more options to record precisely-typed data from applications.

And hobbes is used in major high-volume and low-latency systems at Morgan Stanley (where q originated, as you may know).

Re: Kdb+ and Python: EmbedPy and PyQ

#35

Post by yalph »

kthielen wrote: post #34
yalph wrote: post #32

Earlier quoted context omitted.

I will check thanks but wnated to learn from him. I do not think there are many alternatives by the way. Is this one a full alternative? Does it provide all functionalities of kdb?

It’s better in a lot of ways. It has a type system, can produce much better code, can do cross-process compilation (for multi-process IPC). The FFI/binding process is much simpler, there are more options to record precisely-typed data from applications. And hobbes is used in major high-volume and low-latency systems at Morgan Stanley (where q originated, as you may know).

How come kx is ok with this one? I heard they sue every moving object around.

Re: Kdb+ and Python: EmbedPy and PyQ

#36

Post by kthielen »

yalph wrote: post #35
kthielen wrote: post #34

Earlier quoted context omitted.

It’s better in a lot of ways. It has a type system, can produce much better code, can do cross-process compilation (for multi-process IPC). The FFI/binding process is much simpler, there are more options to record precisely-typed data from applications. And hobbes is used in major high-volume and low-latency systems at Morgan Stanley (where q originated, as you may know).

How come kx is ok with this one? I heard they sue every moving object around.

Kx does seem to bully people with threats of lawsuits, not sure how often they actually follow through.

hobbes is not a k/q clone, it's much more like Haskell actually. The features that make hobbes especially compelling for its production use-cases, like its complex type system, are features that kdb has never had and probably never will have.

Re: Kdb+ and Python: EmbedPy and PyQ

#37

Post by yalph »

kthielen wrote: post #34
yalph wrote: post #32

Earlier quoted context omitted.

I will check thanks but wnated to learn from him. I do not think there are many alternatives by the way. Is this one a full alternative? Does it provide all functionalities of kdb?

It’s better in a lot of ways. It has a type system, can produce much better code, can do cross-process compilation (for multi-process IPC). The FFI/binding process is much simpler, there are more options to record precisely-typed data from applications. And hobbes is used in major high-volume and low-latency systems at Morgan Stanley (where q originated, as you may know).

Hi thank you very much, I work in the industry and I will definitely take a look at it. You seem to be one of the developers behind this project. Is there any way I can contact you in the future?

Re: Kdb+ and Python: EmbedPy and PyQ

#38

Post by kthielen »

yalph wrote: post #37
kthielen wrote: post #34

Earlier quoted context omitted.

It’s better in a lot of ways. It has a type system, can produce much better code, can do cross-process compilation (for multi-process IPC). The FFI/binding process is much simpler, there are more options to record precisely-typed data from applications. And hobbes is used in major high-volume and low-latency systems at Morgan Stanley (where q originated, as you may know).

Hi thank you very much, I work in the industry and I will definitely take a look at it. You seem to be one of the developers behind this project. Is there any way I can contact you in the future?

Yes I do work on the project at Morgan Stanley. Our group email is hobbes-dev@morganstanley.com and we're pretty responsive on that list (as well as on the github page).

Re: Kdb+ and Python: EmbedPy and PyQ

#39

Post by oper8or »

oper8or wrote: post #15
osrec wrote: post #5

Earlier quoted context omitted.

Your comment made me chuckle. In fact, made me immediately think you were employed by Kx at some point; your profile confirmed this was the case! I think Kx has built a wonderful business, but as an ex MD at a bulge bracket bank, I would not sanction its use any more. It is expensive, and it's difficult to hire good people to work with it (not every guy with kdb on their CV is great). The product is good, but so are…

What are the open source offerings that you referring to? The ones that provide 1) no-copy analytics 2) i/o stack bypass i.e. memory-map.

Thank you for your answers. To summarize, we have 1) Hobbes 2) hacked LMDB 3) C++ memory-mapped store of arrays.

Given that options #2 and #3 require some (non-trivial) work, they are not really options.

We left with #1,-- hobbes, which was uploaded to GitHub about 5 months ago and has a whopping team of 2 contributors, both employed by Morgan Stanley.

This is more than nothing, but not much.

I do not have experience with KDB, and looking at the language syntax, not a fan. Integration with Python (depending on implementation) may push KDB towards larger acceptance.

So far I was mostly relying on a variation of the option #3.

Re: Kdb+ and Python: EmbedPy and PyQ

#40

Post by kthielen »

oper8or wrote: post #39
oper8or wrote: post #15

Earlier quoted context omitted.

What are the open source offerings that you referring to? The ones that provide 1) no-copy analytics 2) i/o stack bypass i.e. memory-map.

Thank you for your answers. To summarize, we have 1) Hobbes 2) hacked LMDB 3) C++ memory-mapped store of arrays. Given that options #2 and #3 require some (non-trivial) work, they are not really options. We left with #1,-- hobbes, which was uploaded to GitHub about 5 months ago and has a whopping team of 2 contributors, both employed by Morgan Stanley. This is more than nothing, but not much. I do not have experience…

You must be able to come up with better criticism than that. Number of people who work on the project, time it's been on github, contributors' employers ... these are completely irrelevant to the question of whether this project is a viable alternative. There's not a single technical argument here! :)

For what it's worth (not much), from a purely superficial standpoint, kdb itself started out as a one or two person project at Morgan Stanley! :D

We've managed to get this thing right in the hot path (not just for analysis off on the side, though that use-case is important too) where a significant portion of global trading happens, in one of the biggest investment banks in the country, and we've had it working in production for four years doing this (before recently open sourcing), having had to make the technical case to many people who are very aware of kdb and what it can do (as far as kdb goes, Morgan Stanley is Mount Doom!).

I mean, I take your point that it's not ubiquitous in the world yet, but in terms of the OP proposition that there are free and technically superior alternatives, it's proof positive.

Post reply on HN