Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

tech.marksblogg.com

Post reply on HN
Print view View original

31–40 of 104 posts

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#31

Post by AlphaSite »

shaklee3 wrote: post #26
loeg wrote: post #20

Earlier quoted context omitted.

According to wikipedia, the fastest GDDR5 can do 256 Gbit per chip[0]. I don't know how many chips are typically used. MCDRAM in the article does 400 GB/s, or 3,200 Gb/s. That would require 12.5 of those GDDR5 chips, assuming they scale linearly. [0]: https://en.wikipedia.org/wiki/GDDR5_SDRAM#Commercial_impleme...

GDDR5 can typically do 240GB/s access time on a typical GPU, and there are multiple chips on many cards (Tesla K80). The newer cards use HBM2 and can do 732GB/s ( http://www.nvidia.com/object/tesla-p100.html ).

HBM2 is 1024GB/s (256 per stack).

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#32

Post by atemerev »

vegabook wrote: post #30

All lots of fun, but kdb has an eye watering cost of 200k dollars per year per server. Here's hoping some combo of Apache Arrow (also cache aware, much more language stack flexibilty), Aerospike (lua built in), Impala, and others, can finally take on this overpriced product, which has had a lack of serious competitors for 20 years, owing to its (price inelastic) finance client base.

The binary file of the database (all-inclusive and statically linked) is around 300 kilobytes (!), which makes is probably the most expensive (non-custom) software per kilobyte.

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#33

Post by kpierre »

vegabook wrote: post #30

All lots of fun, but kdb has an eye watering cost of 200k dollars per year per server. Here's hoping some combo of Apache Arrow (also cache aware, much more language stack flexibilty), Aerospike (lua built in), Impala, and others, can finally take on this overpriced product, which has had a lack of serious competitors for 20 years, owing to its (price inelastic) finance client base.

> Apache Arrow (also cache aware, much more cross platform)

kdb+ is available for raspberry pi, is that cross platform enough?

https://kx.com/2016/06/08/kx-releases-raspberry-pi-build-wit...

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#34

Post by vegabook »

kpierre wrote: post #33
vegabook wrote: post #30

All lots of fun, but kdb has an eye watering cost of 200k dollars per year per server. Here's hoping some combo of Apache Arrow (also cache aware, much more language stack flexibilty), Aerospike (lua built in), Impala, and others, can finally take on this overpriced product, which has had a lack of serious competitors for 20 years, owing to its (price inelastic) finance client base.

> Apache Arrow (also cache aware, much more cross platform) kdb+ is available for raspberry pi, is that cross platform enough? https://kx.com/2016/06/08/kx-releases-raspberry-pi-build-wit...

32 bit. Please be serious. You know full well that for all non-toy work kdb is exhorbitantly expensive.

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#36

Post by picodoc »

svan99 wrote: post #10

Nice writeup. If you are interested in learning KDB/Q, please take a look at this book: http://code.kx.com/mkdocs/qformortals3/

or for a really quick high level overview you can use this: https://learnxinyminutes.com/docs/kdb+/

it's a really beautiful little language once you get into it :-)

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#38

Post by buckie »

de_Selby wrote: post #23
buckie wrote: post #17

In my experience, kdb+'s k and q (which is broadly speaking a legibility wrapper around k, which again broadly speaking is APL without unicode) are phenomenally fast for dense time series datasets. Though they can struggle (relatively, still pretty fast) with sparse data that's not really what they are built for. They were built for high-performance trading systems, and trading data is dense. If you like writing dens…

That's more on you than it is the language though. You can write obtuse write-only code in any language. I will concede that there is a culture of trying to be a bit too clever on the k4 mailing list but it's perfectly possible to write maintainable code in kdb+

While I agree that it's possible to write maintainable in kdb+, I'd argue that it takes noticeably more effort; or, as in inverse, if you drop your coding standards things get worse faster. I'd say the same for regex as well (though in a more limited sense).

The "characters to clever/unmaintainable" ratios you achieve with k (which most kdb+ platforms end up reaching for at some point) are almost unparalleled. A famous example of how awesome/powerful/ridiculous k can be is the "4 lines of K" text editor: http://www.kparc.com/$/edit.k

I guess my point is that letting your guard down, for even a single line, can be orders of magnitude more trouble than it would be in most languages and for that reason I wouldn't base a new stack on it. Also, the developers for it are rare-ish and expensive + it has a serious learning curve for those unfamiliar with FP or Lisp.

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#39

Post by jimbokun »

buckie wrote: post #38
de_Selby wrote: post #23

Earlier quoted context omitted.

That's more on you than it is the language though. You can write obtuse write-only code in any language. I will concede that there is a culture of trying to be a bit too clever on the k4 mailing list but it's perfectly possible to write maintainable code in kdb+

While I agree that it's possible to write maintainable in kdb+, I'd argue that it takes noticeably more effort; or, as in inverse, if you drop your coding standards things get worse faster. I'd say the same for regex as well (though in a more limited sense). The "characters to clever/unmaintainable" ratios you achieve with k (which most kdb+ platforms end up reaching for at some point) are almost unparalleled. A famo…

Not knowing k, seems like the worst part of the edit.k examples is every variable has a one character name.

Or do those single character tokens have special meanings in k? Which would be even worse.

Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs

#40

Post by geocar »

jimbokun wrote: post #39
buckie wrote: post #38

Earlier quoted context omitted.

While I agree that it's possible to write maintainable in kdb+, I'd argue that it takes noticeably more effort; or, as in inverse, if you drop your coding standards things get worse faster. I'd say the same for regex as well (though in a more limited sense). The "characters to clever/unmaintainable" ratios you achieve with k (which most kdb+ platforms end up reaching for at some point) are almost unparalleled. A famo…

Not knowing k, seems like the worst part of the edit.k examples is every variable has a one character name. Or do those single character tokens have special meanings in k? Which would be even worse.

> Not knowing k, seems like the worst part of the edit.k examples is every variable has a one character name. Or do those single character tokens have special meanings in k? Which would be even worse.

It can seem that way, but there is value in dense programs: They take up less room on the screen. This means you can see where you repeat yourself, and can often find bugs by just reading your code.

I can't count the number of times I scrolled over some code and missed a bug.

Post reply on HN