Live data from Hacker News

1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

tech.marksblogg.com

91–100 of 104 posts

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#92
post #17

In my experience, kdb+'s k and q (which is broadly speaking a legibility wrapper around k, which again broadly speaking is APL without unicode) are phenomenally fast for dense time series datasets. Though they can struggle (relatively, still pretty fast) with sparse data that's not really what they are built for. They were built for high-performance trading systems, and trading data is dense. If you like writing dens…

"If you like writing..."

For me it is the opposite. I like not having to type any more than necessary. I like writing as little code as possible.

I also like being able to read through a program listing without having to wade through page upon page of long, camel case function names, and trying to follow five levels of indentation.

Not because I think that is the "wrong" approach for everyone but because I personally struggle with overcoming language and documentation verbosity in order to make any progress. It is the wrong approach for me. I wonder if perhaps this is how it feels for the typical programmer, in the OP's comment, who might struggle with a lack of verbosity.

Sometimes I streamedit (search and replace) blocks of code to remove indentation and shorten long function names to two letter labels, just so I can read without distraction.

What are the chances of finding someone who has the same preference for terseness and who is as skilled as Arthur Whitney?

q.k is about 240 lines in the version I am using. Most of the functions in .q, .Q and the other namespaces fit on a single line. This is a thing of beauty, IMO. For me, this makes studying the language manageable.

Someone in this thread posted a fizzbuzz solution last week that I thought illustrated how one can "work outward", incrementally adding primitives to the left or right to keep modifying output until the desired result is reached.

I use kdb+ daily but I'm a slow learner and still not very good with composing programs from scratch in k.

What little I have written was also done by "working outward", so the fizzbuzz example gave me some hope maybe I'm not too far off track. Thank you for that example. I hope we see some more.

I am thankful k exists, even if the intepreter is not open source and there's no BSD port.

As someone else commented somewhere, perhaps the most awkward aspect of k is getting it to interface with anything else. I think this is what keeps me from using it more frequently for more daily tasks.

But this may be more of a problem with everything else, and not necessarily with k.

I wish a more competent C programmer than I would write a lightweight replacement using linenoise or libedit to substitute for "rlwrap".

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#93
post #88
post #73

Earlier quoted context omitted.

This is indeed a problem with languages without mandatory variable declaration. Is it the case of K, Q or whatever its name is?

> This is indeed a problem with languages without mandatory variable declaration. Typos affect everyone. It is not a function of the language that causes you to mistype. It is that the program is long and you cannot see the whole thing that causes you to miss it. > Is it the case of K, Q or whatever its name is? {foo}[] generates an error because foo is unbound. What's the difference between {foox} and {fooy}? They b…

> Typos affect everyone. It is not a function of the language that causes you to mistype. It is that the program is long and you cannot see the whole thing that causes you to miss it.

In any language with mandatory variable declaration the code wouldn't even compile. And any decent IDE would even flag this while writing the code.

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#94
post #14

Earlier quoted context omitted.

For a columnar database, that's a continuous chunk of memory. Assuming 32bit q defaults to 32bit int, 1.1 billion integers across four machines means each 64-core (with 4 threads/core) KNL chip is averaging over 275M elements of int array, or 1.1M 32bit int operations per thread. Now think again whether that's amazing or not.

You're not accounting for the memory bandwidth at all. Yes, that's still amazing. Try doing that in opentsdb.

~4.4 GB in 150 ms are just about 30 GB/s.

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#95

Earlier quoted context omitted.

Why not? Sounds like the initial data load and indexing are done up front. Once you get past that to run the benchmark it's not clear that a quad channel ddr4 system would be saturated.

A current-generation Xeon E7 has a memory bandwidth of up to 102 GB/s. The article says 90, which is probably a realistic estimate of achievable throughput. But the data is 500GB. So the problem is not with loading it from disk (that can be done beforehand), but getting it from CPU to RAM fast enough.

Inaccurate; the very point of columnar stores is that if I'm only interested in a column I only expend the memory bandwidth required for that very column and nothing else. Hence typical queries to a columnar store would never stream the whole 500 GB for processing.

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#96
post #20
post #6

How does Phi's MCDRAM compare to GDDR5 (wrt throughput)?

According to wikipedia, the fastest GDDR5 can do 256 Gbit per chip[0]. I don't know how many chips are typically used. MCDRAM in the article does 400 GB/s, or 3,200 Gb/s. That would require 12.5 of those GDDR5 chips, assuming they scale linearly. [0]: https://en.wikipedia.org/wiki/GDDR5_SDRAM#Commercial_impleme...

Speed and size are somewhat independent. Speed is set by the bus frequency and how many chips you use - every chip has a fixed data bus width, typically 32 bits, while overall size is then how many chips you have times how big each chip is.

Unlike typical computer memory architectures, where the memory bus connects multiple chips or modules to one controller, GDDR doesn't do that; every slice of the memory controller only speaks with a single chip, strictly point-to-point. (Reducing bus load and layout issues and thus allowing higher clock rates).

That's why, with GPUs, it's usually sufficient to say how wide the bus is (often 64 - 128 - 256 - 384 - 512 bits) to get a rough idea of it's performance, since memory clock frequencies occupy a rather narrow range. (However, narrow-bus, lower-end GPUs often don't use the same technology as higher-end GPUs, eg. DDR3 instead of GDDR5)

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#97
post #90
post #86

Earlier quoted context omitted.

> Is this supposed to be a joke? … there is no way they increase bugs. No, it's not supposed to be a joke. Steve McConnell 1993 observed density is proportional to source program length, so this should be obvious to every programmer: If a small program (as measured in source code bytes) is more likely to be a correct program, then this follows. A major issue with discussing programming is the sheer number of people w…

You are completely misrepresenting what McConnell says. There is a summary of his advice here[1], but just to highlight Describe everything the routine does And we mean literally everything. If that makes the name ridiculously long, the name isn't the problem. Your routine is. And Make names as long as necessary According to McConnell, the optimum name length for a variable is 9 to 15 characters; routines tend to be…

I don't agree with everything McConnell says, but if you have it handy, note at page 173:

"A study from Basili and Perricone found that routine size was inversely correlated with errors: as the size of routines increased (up to 200 lines of code), the number of errors per line decreased (Basili and Perricone 1984).

And the conclusion (on page 174): …None of the studies that reported decreased cost, decreased error rates, or both with larger routines distinguished among sizes larger than 200 lines, and you're bound to run into an upper limit of understandability as you pass 200 lines of code

With that in mind, consider that KDB's SQL92 interface is 35 lines (the parser is 14 lines). It may be an extreme example of what McConnell is observing, and yet was himself unable to learn from.

> I'm still unconvinced you aren't trolling.

Look at it this way: Here is software that is faster than what you can write (or maybe you want to write your own taxi problem implementation), and if you don't try hard, you will miss out in finding out how to do something that you can't do.

That downvote button is so easy, that internet person is just a troll, I've been programming for ages, so of course I know what I'm talking about, but if you look through my comment history, you might find it easier to convince yourself at least that I believe that program length matters and permit yourself a discussion about it. You might learn something.

* https://news.ycombinator.com/item?id=8476294

* https://news.ycombinator.com/item?id=8477064

* https://news.ycombinator.com/item?id=10872209

and so on.

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#98
post #93
post #88

Earlier quoted context omitted.

> This is indeed a problem with languages without mandatory variable declaration. Typos affect everyone. It is not a function of the language that causes you to mistype. It is that the program is long and you cannot see the whole thing that causes you to miss it. > Is it the case of K, Q or whatever its name is? {foo}[] generates an error because foo is unbound. What's the difference between {foox} and {fooy}? They b…

> Typos affect everyone. It is not a function of the language that causes you to mistype. It is that the program is long and you cannot see the whole thing that causes you to miss it. In any language with mandatory variable declaration the code wouldn't even compile. And any decent IDE would even flag this while writing the code.

> In any language with mandatory variable declaration the code wouldn't even compile.

Nonsense. memmove and memcpy are both defined, and if you choose the wrong one your program compiles and runs, but it simply produces the wrong result.

> And any decent IDE would even flag this while writing the code.

There is no IDE that will flag this. Advanced whole-program static analysis has a hard time with this problem and I'm not aware of any general solutions.

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#100

Earlier quoted context omitted.

A current-generation Xeon E7 has a memory bandwidth of up to 102 GB/s. The article says 90, which is probably a realistic estimate of achievable throughput. But the data is 500GB. So the problem is not with loading it from disk (that can be done beforehand), but getting it from CPU to RAM fast enough.

Inaccurate; the very point of columnar stores is that if I'm only interested in a column I only expend the memory bandwidth required for that very column and nothing else. Hence typical queries to a columnar store would never stream the whole 500 GB for processing.

Right, a columnar store would touch only a couple (1.1bn * three numerical columns) of those 500 GB. But the question was about Redis :)

The author has an overview of the benchmarks with various systems at http://tech.marksblogg.com/benchmarks.html – but one can't just compare rows as many factors vary (esp. hardware).

Post reply on HN