Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kdb+ and Python: EmbedPy and PyQ

kx.com

Post reply on HN
Print view View original

21–30 of 40 posts

Re: Kdb+ and Python: EmbedPy and PyQ

#21

Post by alfiedotwtf »

oper8or wrote: post #15
osrec wrote: post #5

Earlier quoted context omitted.

Your comment made me chuckle. In fact, made me immediately think you were employed by Kx at some point; your profile confirmed this was the case! I think Kx has built a wonderful business, but as an ex MD at a bulge bracket bank, I would not sanction its use any more. It is expensive, and it's difficult to hire good people to work with it (not every guy with kdb on their CV is great). The product is good, but so are…

What are the open source offerings that you referring to? The ones that provide 1) no-copy analytics 2) i/o stack bypass i.e. memory-map.

Everyone is hanging on for OP to deliver

Re: Kdb+ and Python: EmbedPy and PyQ

#22

Post by kthielen »

geocar wrote: post #4
osrec wrote: post #3

I've never quite understood why banks still opt for kdb when (a) hiring staff is ridiculously difficult/expensive and (b) there are so many other cheaper alternatives available. They literally have a strangle hold on every eFX desk in London, and I cannot really figure out why!

Maybe you're wrong about (b)? Sure, I could cobble something about as fast and functional out of postgres, C, awk, and a bunch of other things, but it'd certainly cost more just in developer time than kdb.

https://github.com/Morgan-Stanley/hobbes

Re: Kdb+ and Python: EmbedPy and PyQ

#23

Post by kthielen »

RodgerTheGreat wrote: post #14
chc4 wrote: post #13

Earlier quoted context omitted.

I'm surprised you didn't link an actual c file; everything related to k seems to trigger an exorbitantly high number of "what the fuck"s. https://github.com/kevinlawler/kona/blob/master/src/0.c for example. http://archive.vector.org.uk/art10501320 is one of my favorite articles, though

That article was enough to inspire me to write a K interpreter, and eventually land me a job working with K. If I ever meet Stephen Taylor in person I imagine it'll be an interesting story to tell.

That's great that your interest produced that result! When I made a K interpreter, Kx threatened to sue me and everyone I had worked for.

Re: Kdb+ and Python: EmbedPy and PyQ

#24

Post by kthielen »

oper8or wrote: post #15
osrec wrote: post #5

Earlier quoted context omitted.

Your comment made me chuckle. In fact, made me immediately think you were employed by Kx at some point; your profile confirmed this was the case! I think Kx has built a wonderful business, but as an ex MD at a bulge bracket bank, I would not sanction its use any more. It is expensive, and it's difficult to hire good people to work with it (not every guy with kdb on their CV is great). The product is good, but so are…

What are the open source offerings that you referring to? The ones that provide 1) no-copy analytics 2) i/o stack bypass i.e. memory-map.

The one I linked to in my previous post also produces code that's fast enough to use in the critical path in low-latency trading systems.

As you may know, q's "scalar performance" is not great; similar to Scheme or python due to boxing overhead (as you can see in the linked k.h file below).

Also, the fact that q is untyped has a severe impact on its safe use in large and complex projects.

(ETA: https://github.com/Morgan-Stanley/hobbes)

Re: Kdb+ and Python: EmbedPy and PyQ

#25

Post by e12e »

chc4 wrote: post #13
bkeroack wrote: post #7

I like it when kdb is mentioned because then I can post this link: https://github.com/KxSystems/kdb/blob/master/c/c/k.h#L96 It's one of my all-time favorites. A window into a certain type of mind.

I'm surprised you didn't link an actual c file; everything related to k seems to trigger an exorbitantly high number of "what the fuck"s. https://github.com/kevinlawler/kona/blob/master/src/0.c for example. http://archive.vector.org.uk/art10501320 is one of my favorite articles, though

I was kind of happy to find some java code this time around - who says java needs to be verbose?

https://github.com/KxSystems/kdb/blob/master/c/jdbc.java

Re: Kdb+ and Python: EmbedPy and PyQ

#26

Post by osrec »

oper8or wrote: post #15
osrec wrote: post #5

Earlier quoted context omitted.

Your comment made me chuckle. In fact, made me immediately think you were employed by Kx at some point; your profile confirmed this was the case! I think Kx has built a wonderful business, but as an ex MD at a bulge bracket bank, I would not sanction its use any more. It is expensive, and it's difficult to hire good people to work with it (not every guy with kdb on their CV is great). The product is good, but so are…

What are the open source offerings that you referring to? The ones that provide 1) no-copy analytics 2) i/o stack bypass i.e. memory-map.

There are a few. I've used a modified LMDB source in the past with success, employing similar tricks to kdb for performance (i.e. store daily data as contiguous arrays so that reads are quick etc). Either way, implementing a memory mapped store of arrays and operating on it is not too challenging a problem for any good C++ dev.

Re: Kdb+ and Python: EmbedPy and PyQ

#27

Post by osrec »

ah- wrote: post #10
dilap wrote: post #9

Earlier quoted context omitted.

huh, so what do i need to learn to be one of these ridiculously expensive people working w/ kdb?

I'd start with https://code.kx.com/q4m3/

Also, you need to be lucky enough to somehow move into a team that uses kdb heavily (this usually happens via an internal move within an investment bank). Learning it using the tutorials is not the same as implementing it in a live environment, and the tutorials probably won't be enough to get you a job/contract. If you could say something like "I've worked on the FX desk at JP Morgan, where we used kdb daily", you'll be hired everywhere. Basically, you need a bit of luck in the first instance to get into a kdb focused team that can train you - after that, you're sorted!

Re: Kdb+ and Python: EmbedPy and PyQ

#28

Post by beagle3 »

dilap wrote: post #9
osrec wrote: post #3

I've never quite understood why banks still opt for kdb when (a) hiring staff is ridiculously difficult/expensive and (b) there are so many other cheaper alternatives available. They literally have a strangle hold on every eFX desk in London, and I cannot really figure out why!

huh, so what do i need to learn to be one of these ridiculously expensive people working w/ kdb?

Historically, an APL background was a good indication that you grok the apl/k/j/q mindset, which is very different from the Algol (c/c#/Java/pascal) world and also from the Lisp world.

Re: Kdb+ and Python: EmbedPy and PyQ

#29

Post by beagle3 »

kthielen wrote: post #23
RodgerTheGreat wrote: post #14

Earlier quoted context omitted.

That article was enough to inspire me to write a K interpreter, and eventually land me a job working with K. If I ever meet Stephen Taylor in person I imagine it'll be an interesting story to tell.

That's great that your interest produced that result! When I made a K interpreter, Kx threatened to sue me and everyone I had worked for.

That’s likely because yours was fast enough to threaten their sales, whereas RodgerTheGreat’s is JavaScript and can not.

Nick Nickolov’s one also disappeared off GitHub, though Kevin Lawler’s Kona k3 implementation and Andrey Zholos’s jitted weird dialect are fast and still up; also nils holm’s klong.

Re: Kdb+ and Python: EmbedPy and PyQ

#30

Post by yalph »

osrec wrote: post #5
geocar wrote: post #4

Earlier quoted context omitted.

Maybe you're wrong about (b)? Sure, I could cobble something about as fast and functional out of postgres, C, awk, and a bunch of other things, but it'd certainly cost more just in developer time than kdb.

Your comment made me chuckle. In fact, made me immediately think you were employed by Kx at some point; your profile confirmed this was the case! I think Kx has built a wonderful business, but as an ex MD at a bulge bracket bank, I would not sanction its use any more. It is expensive, and it's difficult to hire good people to work with it (not every guy with kdb on their CV is great). The product is good, but so are…

So what are those alternatives?
Post reply on HN