Live data from Hacker News

1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

tech.marksblogg.com

81–90 of 104 posts

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#81

Earlier quoted context omitted.

K, not Q? nice! if you don't mind me asking, where? i am curious who is using K these days.

I work for an NYC-based company called 1010data which specializes in business analytics products. Most of our backend infrastructure is written in K3. And if anyone's curious, yes, we're hiring.

I interviewed at the NYC office a few months ago and had a good experience. The people I met were smart, friendly, and respected my time.

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#82
It looks like year is extracted from pickup_datetime at ETL, so hence its not a fair comparison against the other databases that do this at runtime in Q3 and Q4. In something like MapD Q3 would be nearly as fast as Q1 (~20ms) without the extract function, which involves relatively complicated math.

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#83

Earlier quoted context omitted.

GDDR5 can typically do 240GB/s access time on a typical GPU, and there are multiple chips on many cards (Tesla K80). The newer cards use HBM2 and can do 732GB/s ( http://www.nvidia.com/object/tesla-p100.html ).

HBM2 is 1024GB/s (256 per stack).

Kind of. Nvidia lowered the voltage on their P100 so it does not hit those rates. Theoretically it can go that high, but the power draw was too large. Next gen we'll likely see that.

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#84
post #65

Earlier quoted context omitted.

With the new generation there is now a difference. This is because this generation has Phis in both addin card (PCIe) and a bootable CPU package (when you run Linux or windows or whatever on the Phi). Generally with the PCIe one you're running something like OpenCL and with system CPU package you run threads and processes like you normally would. Technically you could run software directly on the old addin cards sinc…

Do you still need the costly intel compiler suite to run some C++ code on it? Honestly they make it pretty hard to hack with as a device. It could succeed as a device if they let devs easily create cool applications with it.

Both gcc & llvm can compile binaries targeting the Phi. Since it runs in Linux (as addin PCIe card or host Linux CPU) it can run any kind of ELF x86_64 binary. They also support generating AVX-512 instruction of the Phi.

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#85
post #75
post #64

Has somebody here experience with Jd and could comment on the status or the performance? Thanks!

The interpreted Jd is fast, but you need the compiled, commercial license Jd, or you used to, for the speed test. I love J compared with K, but that is because I found it first, and the differences between J and K are minimal, but a different enough to keep me using J.

I think jd is mostly plain old interpreted J. There are a few shared libraries to add functionality to core J, but it's mostly just J. You can probably get a non-commercial license if you ask nicely.

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#86
post #71
post #52

Earlier quoted context omitted.

> There is no reason not to use camel cased variable names and indent functions, There is: it makes the program bigger. Program source code length is significant, and if you have more lines, you have more opportunities for bugs.

Is this supposed to be a joke? Or maybe related to some weird coding environment? Long variable names may take more characters, but there is no way they increase bugs. Indenting(!) doesn't even take many more characters if you use tabs...

> Is this supposed to be a joke? … there is no way they increase bugs.

No, it's not supposed to be a joke.

Steve McConnell 1993 observed density is proportional to source program length, so this should be obvious to every programmer: If a small program (as measured in source code bytes) is more likely to be a correct program, then this follows.

A major issue with discussing programming is the sheer number of people who believe they know how to program, when any non-programmer could see quite obviously that they don't: A professional bridge-builder doesn't often fail to build a bridge, but a professional CMS programmer seems to unable to get much past hello world without a bug or two; Less code will therefore produce less bugs.

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#87
post #75

Earlier quoted context omitted.

The interpreted Jd is fast, but you need the compiled, commercial license Jd, or you used to, for the speed test. I love J compared with K, but that is because I found it first, and the differences between J and K are minimal, but a different enough to keep me using J.

I think jd is mostly plain old interpreted J. There are a few shared libraries to add functionality to core J, but it's mostly just J. You can probably get a non-commercial license if you ask nicely.

> You can probably get a non-commercial license if you ask nicely.

This was my experience. I got a nice email from Eric Iverson in response along with the activation key.

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#88
post #73
post #70

Earlier quoted context omitted.

> Bugs don't hide in variable names or white space. Bugs do hide in variable names and whitespace. thisIsALongVariableName = True thisIsaLongVariableName = True is easy to spot when they're side-by-side, but not so much when they're two pages away from each other. Python's tabs and spaces can confuse the indention in cases so that code that "looks" aligned actually isn't. If you haven't run into a problem caused by a…

This is indeed a problem with languages without mandatory variable declaration. Is it the case of K, Q or whatever its name is?

> This is indeed a problem with languages without mandatory variable declaration.

Typos affect everyone. It is not a function of the language that causes you to mistype. It is that the program is long and you cannot see the whole thing that causes you to miss it.

> Is it the case of K, Q or whatever its name is?

{foo}[] generates an error because foo is unbound. What's the difference between {foox} and {fooy}? They both might be variables, and yet it is only by reading the program that you can tell which one is correct.

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#89
post #76
post #71

Earlier quoted context omitted.

Is this supposed to be a joke? Or maybe related to some weird coding environment? Long variable names may take more characters, but there is no way they increase bugs. Indenting(!) doesn't even take many more characters if you use tabs...

Mathematical formulas don't use long variable names. They use x, y, and various greek letters.

This is generally (amongst mathematicians) considered to be a problem, not a good thing (Source: work with university Math departments).

Edit, see http://mathoverflow.net/questions/8295/origins-of-mathematic... for the confusion this causes..

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#90
post #86
post #71

Earlier quoted context omitted.

Is this supposed to be a joke? Or maybe related to some weird coding environment? Long variable names may take more characters, but there is no way they increase bugs. Indenting(!) doesn't even take many more characters if you use tabs...

> Is this supposed to be a joke? … there is no way they increase bugs. No, it's not supposed to be a joke. Steve McConnell 1993 observed density is proportional to source program length, so this should be obvious to every programmer: If a small program (as measured in source code bytes) is more likely to be a correct program, then this follows. A major issue with discussing programming is the sheer number of people w…

You are completely misrepresenting what McConnell says.

There is a summary of his advice here[1], but just to highlight

Describe everything the routine does And we mean literally everything. If that makes the name ridiculously long, the name isn't the problem. Your routine is.

And

Make names as long as necessary According to McConnell, the optimum name length for a variable is 9 to 15 characters; routines tend to be more complex and therefore deserve longer names. Make your names as long as they need to be in order to make them understandable.

I've read McConnell, and your claims are so completely the opposite of what he recommends that I'm still unconvinced you aren't trolling.

[1] https://blog.codinghorror.com/i-shall-call-it-somethingmanag...

Post reply on HN