41–50 of 66 posts
I chuckled when I read "We have a legacy data warehouse based in Hive and Elastic MapReduce, with backing data stores in S3.". I guess things have come full circle. It wasn't long ago that a relational database solution would have been "legacy".
These type of articles baffle me, you're comparing a high-performance analytical database to a batch-orientated SQL engine. The whole point behind these query engines on Hadoop (Hive, Presto, Impala, etc) is to separate the database from the query engine. With these engines you can project schemas over raw data in its original form, without having to load it into a table. With Redshift, or other similar analytical da…
Redshift is an especially limited SQL engine considering it doesn't support UDFs. It is wicked fast, but what you get in speed you lose in flexibility. Current (well, February, but fairly current) benchmarks[0] place Impala and Shark (SQL on top of Spark) within grasp of Redshift while pulling data from disk and, for certain workloads, on par or faster than Redshift. This is without using a columnar file format. Impa…
Earlier quoted context omitted.
In terms of time, which is by far the biggest cost in most development, CouchDB is the better option. For a single, one-off utility, sure. For anything that you ever planned for production, that would be crazy . Just to be clear, the mentality that onion proposes (at least from my interpretation, though I apologize if I'm misunderstanding), usually justified under a gross misinterpretation of the "premature optimizat…
That's just like your opinion though. You ever used CouchDB in production before?
These type of articles baffle me, you're comparing a high-performance analytical database to a batch-orientated SQL engine. The whole point behind these query engines on Hadoop (Hive, Presto, Impala, etc) is to separate the database from the query engine. With these engines you can project schemas over raw data in its original form, without having to load it into a table. With Redshift, or other similar analytical da…
Redshift is an especially limited SQL engine considering it doesn't support UDFs. It is wicked fast, but what you get in speed you lose in flexibility. Current (well, February, but fairly current) benchmarks[0] place Impala and Shark (SQL on top of Spark) within grasp of Redshift while pulling data from disk and, for certain workloads, on par or faster than Redshift. This is without using a columnar file format. Impa…
The queries and data set Berkeley chooses are bizarre. TPC-DS or TPC-H are much more representative of real-world performance, and the differences are more pronounced as the queries get more complex.
edit: I also don't understand why the Spark team is reinventing the wheel for Spark SQL when Hive running on Tez will produce very similar query plans. The two projects are converging to the same place, but they insist on having divergent code bases ;)
When you were on EMR, you could have used Mahout's distributed collaborative filtering, which has the benefits of being correct, and requiring zero coding.
Wikipedia explains here: http://en.wikipedia.org/wiki/Cosine_similarity
Earlier quoted context omitted.
In terms of time, which is by far the biggest cost in most development, CouchDB is the better option. For a single, one-off utility, sure. For anything that you ever planned for production, that would be crazy . Just to be clear, the mentality that onion proposes (at least from my interpretation, though I apologize if I'm misunderstanding), usually justified under a gross misinterpretation of the "premature optimizat…
At least for the startup world, it's about prioritisation of concerns. Will that disaster implementation take me to my next(or first) round of funding? If yes, I'll happily go with it. After that, I can throw money at the problem.
Earlier quoted context omitted.
Redshift is an especially limited SQL engine considering it doesn't support UDFs. It is wicked fast, but what you get in speed you lose in flexibility. Current (well, February, but fairly current) benchmarks[0] place Impala and Shark (SQL on top of Spark) within grasp of Redshift while pulling data from disk and, for certain workloads, on par or faster than Redshift. This is without using a columnar file format. Impa…
Although I have a lot of respect for the amplab, they did not do their due diligence with that benchmark. Mainly for a few reasons, they didn't test using columnar storage in Hadoop (ORC / Parquet), which is what Redshift is using underneath (a proprietary columnar store). Also, the most complicated query they ran was a two table join, and from what I can tell, there wasn't any concurrent workload testing. (disclaime…
Earlier quoted context omitted.
In terms of time, which is by far the biggest cost in most development, CouchDB is the better option. For a single, one-off utility, sure. For anything that you ever planned for production, that would be crazy . Just to be clear, the mentality that onion proposes (at least from my interpretation, though I apologize if I'm misunderstanding), usually justified under a gross misinterpretation of the "premature optimizat…
At least for the startup world, it's about prioritisation of concerns. Will that disaster implementation take me to my next(or first) round of funding? If yes, I'll happily go with it. After that, I can throw money at the problem.
Another commenter pointed this out, but what you're trying to compute is cosine similarity, in which case you're missing the normalizing part in the denominator (the product of the magnitude of both vectors). In other words, two items which both occur frequently will score higher than two items which occur infrequently, but which co-occur higher than usual. This leads to a tendency to over-recommend popular items. Wh…
Earlier quoted context omitted.
In terms of time, which is by far the biggest cost in most development, CouchDB is the better option. For a single, one-off utility, sure. For anything that you ever planned for production, that would be crazy . Just to be clear, the mentality that onion proposes (at least from my interpretation, though I apologize if I'm misunderstanding), usually justified under a gross misinterpretation of the "premature optimizat…
At least for the startup world, it's about prioritisation of concerns. Will that disaster implementation take me to my next(or first) round of funding? If yes, I'll happily go with it. After that, I can throw money at the problem.
I've been involved in companies that went 14 years without a disaster. Another company I was involved with had 2 in a span of 2 months, each taking between 2 and 3 days to recover from.
Regardless of whether I need it or not, I sleep better at night knowing a decent plan is in place. Which means I can perform better during the day.