Pretty wide array of technologies covered in these benchmarks. I wonder how ClickHouse will fare, should be very competitive.
1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs
51–60 of 104 posts
Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs
#52Earlier quoted context omitted.
But from what I recall, the syntax is terse by design - this is not inherently bad, though. In other mainstream languages, you have to go out of your way to write obtuse code (e.g, code golf). I'm guessing that best way to address this issue is through liberal use of explanatory comments.
Terse syntax isn't an issue, you just need to get used to reading it. Using 1 character variable names is another matter though. There is no reason not to use camel cased variable names and indent functions, if/else blocks etc, and when written this way the code can be perfectly legible even to non q programmers. Something else that leads people into the write-only trap is that the usual way of working with the langu…
> There is no reason not to use camel cased variable names and indent functions,
There is: it makes the program bigger. Program source code length is significant, and if you have more lines, you have more opportunities for bugs.
Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs
#53Earlier quoted context omitted.
I program in K professionally and help maintain a fairly large codebase. It's an unusual-looking language and it takes practice to learn to "skim" K code in the way that most programmers are used to for their favorite curly-bracket language, but it can be learned. The biggest messes I have to clean up come less from "clever" code than they do from people who try to program in K as if it were some other language. For…
K, not Q? nice! if you don't mind me asking, where? i am curious who is using K these days.
Bank of America uses it to quite an extent - especially in eFX.
Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs
#54Earlier quoted context omitted.
Terse syntax isn't an issue, you just need to get used to reading it. Using 1 character variable names is another matter though. There is no reason not to use camel cased variable names and indent functions, if/else blocks etc, and when written this way the code can be perfectly legible even to non q programmers. Something else that leads people into the write-only trap is that the usual way of working with the langu…
> There is no reason not to use camel cased variable names and indent functions, There is: it makes the program bigger. Program source code length is significant, and if you have more lines, you have more opportunities for bugs.
Bugs don't hide in variable names or white space.
Compiler errors do. But you catch those first run.
If tab vs 2 spaces vs 3 spaces can induce a logic error that is a purposely obtuse language.
Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs
#55Earlier quoted context omitted.
Great, and btw I think it's cool and I enjoyed reading it. But what useful conclusions can be drawn from it?
All conclusions are only valid for similar workloads, but each of MapD and GPUs, Q/kdb+ and Xeon Phi, Redshift, Athena, Big Query, Presto, and Elasticsearch claim to be fast, inexpensive, easy to work with, and otherwise great for Big Data. Which ones really are fast? How fast is fast? How much is this going to cost? Do I need 5 nodes or 50? A few examples of some useful conclusions: - Just because a relatively well-…
>Just because a relatively well-optimized PostgreSQL database on a regular workstation takes 5 minutes to run a query doesn't mean you can't get special hardware to run that query faster than you can type.
This is actually my biggest complaint with the article. He used cstore_fdw with Postgres, which doesn't allow much real indexing, and as far as I can tell (knowing only a little bit about it) he didn't really use any of the benefits of cstore_fdw.
I'd be interested to see how plain Postgres, possibly on a compressed filesystem, with properly-indexed tables stacks up.
Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs
#56Earlier quoted context omitted.
Great, and btw I think it's cool and I enjoyed reading it. But what useful conclusions can be drawn from it?
All conclusions are only valid for similar workloads, but each of MapD and GPUs, Q/kdb+ and Xeon Phi, Redshift, Athena, Big Query, Presto, and Elasticsearch claim to be fast, inexpensive, easy to work with, and otherwise great for Big Data. Which ones really are fast? How fast is fast? How much is this going to cost? Do I need 5 nodes or 50? A few examples of some useful conclusions: - Just because a relatively well-…
These conclusions don't seem very useful because either they are already well established or are not valid. Some examples:
Just because a relatively well-optimized PostgreSQL database on a regular workstation takes 5 minutes to run a query doesn't mean you can't get special hardware to run that query faster than you can type.
Already well established for years with systems like redis, and more recently with gpu databases, and other techniques posted on HN regularly.
Spark + S3 + Amazon Elastic Map Reduce...is pretty slow compared to better tools, and even compared to plain PostgreSQL.
Not valid because it doesn't generalize. It so much depends on type of work being done, system architecture, etc, that you can only say it may or may not be true.
HDFS really is a lot faster than S3.
This is already well established, Amazon states aa much right in the docs: http://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-pl...
Performance of an Xeon Phi 64-core CPU is within an order of magnitude to an NVidia Titan X.
Not precise enough to matter because getting within 10x difference is not close to being competitive.*
Loading 104 GB of compressed data into Q/kdb+ expands to 125 GB with and takes about 30 minutes, but on Redshift expands to 2 TB and takes many hours to upload on a normal connection, plus 4 hours to actually import!
I don't see how it's possible for 104GB of csv text data to decompress into only 125GB. For cvs to compress only ~20%...doesn't make sense.
It might cost $5000 to custom-build a GPU-based supercomputer that can do these queries in under a second
No, two problems here. The hardware in question could have used 1 cheap CPU instead of two expensive Xeons and been much less expensive. Bigger problem: The MapD software itself will be $50,000.
Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs
#57All lots of fun, but kdb has an eye watering cost of 200k dollars per year per server. Here's hoping some combo of Apache Arrow (also cache aware, much more language stack flexibilty), Aerospike (lua built in), Impala, and others, can finally take on this overpriced product, which has had a lack of serious competitors for 20 years, owing to its (price inelastic) finance client base.
Are you sure about that? I think it's significantly lower than 200k
Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs
#58If the combination of such languages, high-performance hardware, and large scale compute problems is interesting.. the startup I work for in Mountain View is hiring...
Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs
#59This was a good idea for a test. I'll definitely check out the author's other stuff. Commenting briefly on cost: while the article mentions the free 32 bit version early on, the actual benchmarks were done using the commercial version. I've had the impression the comercial version was cost prohibitive for us poor folks. For those interested in experimenting with Xeon Phi though, it looks like you can get started for…
If you just want to meddle with a Xeon Phi, you can get some as cheap as $300: https://www.amazon.com/Intel-BC31S1P-Xeon-31S1P-Coprocessor/... , though that's from the 3100 rather than 7200 family, and so won't perform as fast. That kind of money puts it more into the hobby territory, though.
Re: 1.1B Taxi Rides on Kdb+/q and 4 Xeon Phi CPUs
#60Earlier quoted context omitted.
A single machine doesn't have the required memory bandwidth to do this in the same time.
Why not? Sounds like the initial data load and indexing are done up front. Once you get past that to run the benchmark it's not clear that a quad channel ddr4 system would be saturated.
A current-generation Xeon E7 has a memory bandwidth of up to 102 GB/s. The article says 90, which is probably a realistic estimate of achievable throughput. But the data is 500GB. So the problem is not with loading it from disk (that can be done beforehand), but getting it from CPU to RAM fast enough.