61–66 of 66 posts
Is it me or are people switching to non-relational data warehouse architectures simply because it's en vogue? How many companies do you know that have enough data where a non-relational DW would actually make sense? I wonder, have we really pushed relational databases to their breaking point?
I'm a big fan of compressed denormalized data.
Earlier quoted context omitted.
Disaster recovery is easy to put off forever because you don't need it until you do. When it happens it can also kill off your company. I've been involved in companies that went 14 years without a disaster. Another company I was involved with had 2 in a span of 2 months, each taking between 2 and 3 days to recover from. Regardless of whether I need it or not, I sleep better at night knowing a decent plan is in place.…
Yeah, but was it your choice of DB that killed you or something else? That something else is always more likely to happen and more dangerous than 'oh noes all my data is gone stupid mongo/couch!' as if that ever really happens.
Earlier quoted context omitted.
Disaster recovery is easy to put off forever because you don't need it until you do. When it happens it can also kill off your company. I've been involved in companies that went 14 years without a disaster. Another company I was involved with had 2 in a span of 2 months, each taking between 2 and 3 days to recover from. Regardless of whether I need it or not, I sleep better at night knowing a decent plan is in place.…
Yeah, but was it your choice of DB that killed you or something else? That something else is always more likely to happen and more dangerous than 'oh noes all my data is gone stupid mongo/couch!' as if that ever really happens.
I would need a hell of a lot more than a time savings of 1-2 days to add a whole new database technology to my production environment. Even if I know the tech the installation, configuration, automated backups, and automated validation of backups will likely consume more than 1-2 days to get setup. Then add on the learning curve aspect if I've never used it in a real production environment. Then add on the learning curve for any team members who might not be familiar with it.
Weeks, maybe. A day or two: not worth it.
Earlier quoted context omitted.
Yeah, but was it your choice of DB that killed you or something else? That something else is always more likely to happen and more dangerous than 'oh noes all my data is gone stupid mongo/couch!' as if that ever really happens.
Granted, I was making some assumptions about the original post. For me it's not about the choice of DB, it's about having the proper knowledge, time, and team to be able to setup a production environment that isn't seriously flawed in one or more ways. I would need a hell of a lot more than a time savings of 1-2 days to add a whole new database technology to my production environment. Even if I know the tech the inst…
I think if your architecture is likely to collapse when you shop around for databases, then you built the whole thing wrong to begin with.
Earlier quoted context omitted.
Although Impala is still a fairly new product, my team has been using it internally at Cloudera in production for over a year for real-time log analysis to our support engineers ( http://bit.ly/USFQdh ), among other ad-hoc BI analytics. We also have a bunch of customers who are using Impala to power very critical interactive workloads. What about Impala makes you feel like it's not production ready? Good question abo…
I knew I smelled me some Cloudera... :) HBase: I think HBase (based on the sorting of qualifiers within rows) would be suited toward the "ranking" problem, that's why I brought it up. I see this as being a map-only job (and possibly suited toward streaming, or not even using Hadoop at all). It would just be a quick scan/filter/pagination and then a quick ranking algo in some sort of API middle layer (how I envision t…
As for Parquet, that file format is not designed for streaming, but instead is like you mentioned, it's meant for converting large datasets that you plan on running analytics on. Queries against data in parquet is _fast_, like really fast...I've seen queries go from 200 seconds down to 5 seconds by just converting the dataset to Parquet from text.
Concurrency in Impala is actually pretty good, and has always been a design goal from the beginning. I wouldn't compare Impala to Vertica or other analytical databases just yet, there's still a lot of room for improvement, but concurrency in Impala is much better than the other SQL on Hadoop engines (Hive, Presto, etc), and we've demonstrated that on our latest rounds of benchmarks.
BTW #1 - As I mentioned, HBase support Impala is pretty minimal at the moment, but still works fine for ad-hoc queries over small key spaces.
BTW #2 - hehe! I'll let our marketing team know :P
BTW #3 - I'm not sure what you mean here, I'll ask around to see if someone knows.
Thanks for your kind words! Sorry for the late response, I just saw that I had a response :)