Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

tech.marksblogg.com

Post reply on HN
Print view View original

41–50 of 104 posts

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#41

Post by RodgerTheGreat »

buckie wrote: post #17

In my experience, kdb+'s k and q (which is broadly speaking a legibility wrapper around k, which again broadly speaking is APL without unicode) are phenomenally fast for dense time series datasets. Though they can struggle (relatively, still pretty fast) with sparse data that's not really what they are built for. They were built for high-performance trading systems, and trading data is dense. If you like writing dens…

I program in K professionally and help maintain a fairly large codebase. It's an unusual-looking language and it takes practice to learn to "skim" K code in the way that most programmers are used to for their favorite curly-bracket language, but it can be learned.

The biggest messes I have to clean up come less from "clever" code than they do from people who try to program in K as if it were some other language. For example, somebody fond of for loops might write the following to apply a function 'f' to pairings of values and their index in a list:

    result:()
    i:0
    do[#v
      r:r,,f[v[i];i]
      i:i+1
    ]
Ugly, complicated, but "close at hand". There is of course a much nicer and more idiomatic way to do the same thing:

    result: f'[v;!#v]
Most of the time conciseness isn't the goal, but you get it as a side effect of writing good code and working "with the grain" of the language.

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#42

Post by hueving »

geocar wrote: post #40
jimbokun wrote: post #39

Earlier quoted context omitted.

Not knowing k, seems like the worst part of the edit.k examples is every variable has a one character name. Or do those single character tokens have special meanings in k? Which would be even worse.

> Not knowing k, seems like the worst part of the edit.k examples is every variable has a one character name. Or do those single character tokens have special meanings in k? Which would be even worse. It can seem that way, but there is value in dense programs: They take up less room on the screen. This means you can see where you repeat yourself, and can often find bugs by just reading your code. I can't count the nu…

Change your scrolling settings?

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#43

Post by geocar »

atemerev wrote: post #32
vegabook wrote: post #30

All lots of fun, but kdb has an eye watering cost of 200k dollars per year per server. Here's hoping some combo of Apache Arrow (also cache aware, much more language stack flexibilty), Aerospike (lua built in), Impala, and others, can finally take on this overpriced product, which has had a lack of serious competitors for 20 years, owing to its (price inelastic) finance client base.

The binary file of the database (all-inclusive and statically linked) is around 300 kilobytes (!), which makes is probably the most expensive (non-custom) software per kilobyte.

kdb is not statically linked.

edit:

    $ file q/l64/q
    q/l64/q: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18,

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#44

Post by geocar »

hueving wrote: post #42
geocar wrote: post #40

Earlier quoted context omitted.

> Not knowing k, seems like the worst part of the edit.k examples is every variable has a one character name. Or do those single character tokens have special meanings in k? Which would be even worse. It can seem that way, but there is value in dense programs: They take up less room on the screen. This means you can see where you repeat yourself, and can often find bugs by just reading your code. I can't count the nu…

Change your scrolling settings?

Not sure what you mean by that.

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#45

Post by gricardo99 »

anonu wrote: post #27

I've been using kdb+/q for a long time (7 years now) - and can attest to its speed. Objects are placed in memory with the intention that you will run vectorized operations over them. So both the memory-model and the language are designed to work together. Lots of people complain about the conciseness of the language and that it is "write-once" code. I tend to disagree. While it might take a while to understand code y…

Somewhat related is Kerf, also by Kevin Lawler.

http://www.kerfsoftware.com/

It seems more like a kdb+ competitor than open-source alternative, and isn't using q.

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#46

Post by shitgoose »

RodgerTheGreat wrote: post #41
buckie wrote: post #17

In my experience, kdb+'s k and q (which is broadly speaking a legibility wrapper around k, which again broadly speaking is APL without unicode) are phenomenally fast for dense time series datasets. Though they can struggle (relatively, still pretty fast) with sparse data that's not really what they are built for. They were built for high-performance trading systems, and trading data is dense. If you like writing dens…

I program in K professionally and help maintain a fairly large codebase. It's an unusual-looking language and it takes practice to learn to "skim" K code in the way that most programmers are used to for their favorite curly-bracket language, but it can be learned. The biggest messes I have to clean up come less from "clever" code than they do from people who try to program in K as if it were some other language. For…

K, not Q? nice! if you don't mind me asking, where? i am curious who is using K these days.

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#47

Post by RodgerTheGreat »

shitgoose wrote: post #46
RodgerTheGreat wrote: post #41

Earlier quoted context omitted.

I program in K professionally and help maintain a fairly large codebase. It's an unusual-looking language and it takes practice to learn to "skim" K code in the way that most programmers are used to for their favorite curly-bracket language, but it can be learned. The biggest messes I have to clean up come less from "clever" code than they do from people who try to program in K as if it were some other language. For…

K, not Q? nice! if you don't mind me asking, where? i am curious who is using K these days.

I work for an NYC-based company called 1010data which specializes in business analytics products. Most of our backend infrastructure is written in K3. And if anyone's curious, yes, we're hiring.

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#49

Post by srpeck »

anonu wrote: post #27

I've been using kdb+/q for a long time (7 years now) - and can attest to its speed. Objects are placed in memory with the intention that you will run vectorized operations over them. So both the memory-model and the language are designed to work together. Lots of people complain about the conciseness of the language and that it is "write-once" code. I tend to disagree. While it might take a while to understand code y…

Some other k-inspired languages to have a look at:

- https://github.com/johnearnest/ok

- https://github.com/zholos/kuc

- https://github.com/ngn/k

- http://t3x.org/klong/

- https://github.com/tlack/xxl

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#50

Post by anonu »

jimbokun wrote: post #39
buckie wrote: post #38

Earlier quoted context omitted.

While I agree that it's possible to write maintainable in kdb+, I'd argue that it takes noticeably more effort; or, as in inverse, if you drop your coding standards things get worse faster. I'd say the same for regex as well (though in a more limited sense). The "characters to clever/unmaintainable" ratios you achieve with k (which most kdb+ platforms end up reaching for at some point) are almost unparalleled. A famo…

Not knowing k, seems like the worst part of the edit.k examples is every variable has a one character name. Or do those single character tokens have special meanings in k? Which would be even worse.

Yes, the terseness of the language itself lends itself to terseness everywhere - even in variable namings. It's not necessary and I usually avoid that when I write k or q. Purposefully obfuscating code like that lends to the "bad rep" that the language gets.
Post reply on HN