Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

tech.marksblogg.com

Post reply on HN
Print view View original

71–80 of 104 posts

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#71

Post by nl »

geocar wrote: post #52
de_Selby wrote: post #28

Earlier quoted context omitted.

Terse syntax isn't an issue, you just need to get used to reading it. Using 1 character variable names is another matter though. There is no reason not to use camel cased variable names and indent functions, if/else blocks etc, and when written this way the code can be perfectly legible even to non q programmers. Something else that leads people into the write-only trap is that the usual way of working with the langu…

> There is no reason not to use camel cased variable names and indent functions, There is: it makes the program bigger. Program source code length is significant, and if you have more lines, you have more opportunities for bugs.

Is this supposed to be a joke? Or maybe related to some weird coding environment?

Long variable names may take more characters, but there is no way they increase bugs.

Indenting(!) doesn't even take many more characters if you use tabs...

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#72

Post by branchless »

buckie wrote: post #17

In my experience, kdb+'s k and q (which is broadly speaking a legibility wrapper around k, which again broadly speaking is APL without unicode) are phenomenally fast for dense time series datasets. Though they can struggle (relatively, still pretty fast) with sparse data that's not really what they are built for. They were built for high-performance trading systems, and trading data is dense. If you like writing dens…

The main problem with Q is the users. Most users are math grads first, programmers second. They don't use version control. They don't comment their code.

Yes Q is dense but it can be annotated with comments.

I also have to maintain some java code written by this lot and it's nearly as unintelligible.

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#73

Post by wott »

geocar wrote: post #70
valarauca1 wrote: post #54

Earlier quoted context omitted.

Bugs don't hide in variable names or white space. Compiler errors do. But you catch those first run. If tab vs 2 spaces vs 3 spaces can induce a logic error that is a purposely obtuse language.

> Bugs don't hide in variable names or white space. Bugs do hide in variable names and whitespace. thisIsALongVariableName = True thisIsaLongVariableName = True is easy to spot when they're side-by-side, but not so much when they're two pages away from each other. Python's tabs and spaces can confuse the indention in cases so that code that "looks" aligned actually isn't. If you haven't run into a problem caused by a…

This is indeed a problem with languages without mandatory variable declaration. Is it the case of K, Q or whatever its name is?

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#74

Post by branchless »

posnet wrote: post #67
RodgerTheGreat wrote: post #47

Earlier quoted context omitted.

I work for an NYC-based company called 1010data which specializes in business analytics products. Most of our backend infrastructure is written in K3. And if anyone's curious, yes, we're hiring.

If you don't mind me asking, how does someone start with Q/Kdb+? Can you learn it with just the free version or realistically does it require working with it in a professional setting and learning from others who already use it. Are there any sources you recommend for a someone just starting out? Thanks.

You can get a kdb/q cookbook here:

http://code.kx.com/wiki/Cookbook

This includes a tutorial with a smartmeter concept which has several queries of increasing complexity:

http://code.kx.com/wsvn/code/cookbook_code/tutorial/

Step through this, look at the function definitions, cross-ref with the kdb reference page. Also read Q for mortals on their site:

http://code.kx.com/wiki/Tutorials

Yes it's not as accessible as other langs with loads of books, but on the plus side it's coherent, unlike c++.

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#75

Post by eggy »

pvitz wrote: post #64

Has somebody here experience with Jd and could comment on the status or the performance? Thanks!

The interpreted Jd is fast, but you need the compiled, commercial license Jd, or you used to, for the speed test.

I love J compared with K, but that is because I found it first, and the differences between J and K are minimal, but a different enough to keep me using J.

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#76

Post by ams6110 »

nl wrote: post #71
geocar wrote: post #52

Earlier quoted context omitted.

> There is no reason not to use camel cased variable names and indent functions, There is: it makes the program bigger. Program source code length is significant, and if you have more lines, you have more opportunities for bugs.

Is this supposed to be a joke? Or maybe related to some weird coding environment? Long variable names may take more characters, but there is no way they increase bugs. Indenting(!) doesn't even take many more characters if you use tabs...

Mathematical formulas don't use long variable names. They use x, y, and various greek letters.

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#77

Post by VodkaHaze »

mtanski wrote: post #65
protomyth wrote: post #63

Earlier quoted context omitted.

Is there a difference in how you program the different Xeon Phi families?

With the new generation there is now a difference. This is because this generation has Phis in both addin card (PCIe) and a bootable CPU package (when you run Linux or windows or whatever on the Phi). Generally with the PCIe one you're running something like OpenCL and with system CPU package you run threads and processes like you normally would. Technically you could run software directly on the old addin cards sinc…

Do you still need the costly intel compiler suite to run some C++ code on it?

Honestly they make it pretty hard to hack with as a device. It could succeed as a device if they let devs easily create cool applications with it.

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#78

Post by Locke1689 »

ams6110 wrote: post #76
nl wrote: post #71

Earlier quoted context omitted.

Is this supposed to be a joke? Or maybe related to some weird coding environment? Long variable names may take more characters, but there is no way they increase bugs. Indenting(!) doesn't even take many more characters if you use tabs...

Mathematical formulas don't use long variable names. They use x, y, and various greek letters.

And despite lambda calculus being written entirely using single letter variable names, Scheme programmers all use proper word-size variable names in their programs.

Mathematics notation is so notoriously awful that mathematicians often can't even read equations from other disciplines without significant extra context.

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#79

Post by maemre »

ams6110 wrote: post #76
nl wrote: post #71

Earlier quoted context omitted.

Is this supposed to be a joke? Or maybe related to some weird coding environment? Long variable names may take more characters, but there is no way they increase bugs. Indenting(!) doesn't even take many more characters if you use tabs...

Mathematical formulas don't use long variable names. They use x, y, and various greek letters.

That might be because:

- you need to write same variable name on each step of derivation of a formula

- paper and blackboards don't have autocomplete

- juxtaposition is used for multiplication, thus SwapBlue can be Swap times Blue or a single variable called SwapBlue, spacing for multiplication is not a good solution when writing to board/paper

Post reply on HN