Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big Data Debate: HBase

informationweek.com

Post reply on HN
Print view View original

31–40 of 56 posts

Re: Big Data Debate: HBase

#31

Post by jeremiahjordan »

monstrado wrote: post #25
trun wrote: post #20

Linkbait aside there are some reasonable points being made here, specifically "Failover means downtime" about HBase. We run into this pretty much every day in one of our customer facing applications or APIs and it's quite frustrating to have to explain that there's very little you can do to prevent it. I haven't looked too closely at MapR yet, but "instant recovery, seamless sharding and high availability" are impres…

Which version of HBase are you running? If we have a node or two go down in our cluster, HBase is completely unaffected. Recoveries can take up to a few minutes, in the old ages it took hours. I know that MTTR (mean time to recovery) is being worked on together by several large companies to get down to seconds.

"Minutes" is not "unaffected" if your site is now down.

Re: Big Data Debate: HBase

#32

Post by monstrado »

m0nastic wrote: post #30
monstrado wrote: post #18

Earlier quoted context omitted.

Disclaimer: I work at Cloudera as a Tools Developer What do you mean by unstructured? Do you mean the data has yet to be parsed into a format which could be logically grouped into columns? Or do you mean that it's deeply nested? Since log data doesn't really change, it might be overkill to use something like HBase (or any database for that matter). On the tools team at Cloudera, we've found that writing the data into…

Thanks, I've been testing out a bunch of hare-brained schemes and it didn't even occur to me to just use HDFS directly (and seeing you and karterk both suggest it helps). Basically, at a high level, the system I'm working on aggregates and processes security information (It's a SIEM, if that product category means anything to you). At the point the logs get ingested, the server determines if they're "actionable" (whi…

No problem, glad I could help. Your use case sounds pretty interesting, HDFS should fit the bill for sure.

You should take a look at Parquet (http://parquet.io/) for storing your data. It's an open source columnar format that was designed for Hadoop, it even supports nesting (https://github.com/Parquet/parquet-mr/wiki/The-striping-and-... Good luck!

Re: Big Data Debate: HBase

#33

Post by monstrado »

jeremiahjordan wrote: post #31
monstrado wrote: post #25

Earlier quoted context omitted.

Which version of HBase are you running? If we have a node or two go down in our cluster, HBase is completely unaffected. Recoveries can take up to a few minutes, in the old ages it took hours. I know that MTTR (mean time to recovery) is being worked on together by several large companies to get down to seconds.

"Minutes" is not "unaffected" if your site is now down.

Sorry, I should have clarified. When a node goes down, there isn't any sort of "minute" interruption...When there's a full scale outage and you need to perform an actual recovery, that can take minutes.

Re: Big Data Debate: HBase

#34

Post by duaneb »

monstrado wrote: post #22
duaneb wrote: post #19

Earlier quoted context omitted.

How are multi-row transactions these days? That was the largest problem when I looked last... A data store does not a database make.

I don't think that's something they are trying to solve right now. Although HBase guarantees write consistency, I think http://research.google.com/pubs/pub36726.html is the closest paper on how to go about it, but I could be wrong. Like you mentioned, a lot of people use HBase as a data-store, it's incredibly good at that.

> Like you mentioned, a lot of people use HBase as a data-store, it's incredibly good at that.

This is exactly what confuses me about NoSQL. It's not really comparable to relational databases on the CA[P] spectrum.

Re: Big Data Debate: HBase

#35

Post by monstrado »

duaneb wrote: post #34
monstrado wrote: post #22

Earlier quoted context omitted.

I don't think that's something they are trying to solve right now. Although HBase guarantees write consistency, I think http://research.google.com/pubs/pub36726.html is the closest paper on how to go about it, but I could be wrong. Like you mentioned, a lot of people use HBase as a data-store, it's incredibly good at that.

> Like you mentioned, a lot of people use HBase as a data-store, it's incredibly good at that. This is exactly what confuses me about NoSQL. It's not really comparable to relational databases on the CA[P] spectrum.

HBase doesn't claim to be some NoSQL database or equivalent to a relational database, anyone who thinks otherwise hasn't actually read into HBase.

Just look at HBase's website which describe it exactly as

> Apache HBase is the Hadoop database, a distributed, scalable, big data store.

Re: Big Data Debate: HBase

#36

Post by mh- »

threeseed wrote: post #26
duaneb wrote: post #19

Earlier quoted context omitted.

How are multi-row transactions these days? That was the largest problem when I looked last... A data store does not a database make.

I don't know about HBase but for Cassandra it should be available in the next minor update. Single row transactions have just recently been added: http://www.datastax.com/dev/blog/lightweight-transactions-in...

FWIW, Cassandra 1.2 is the latest production/stable version.

Re: Big Data Debate: HBase

#37

Post by samspenc »

monstrado wrote: post #25
trun wrote: post #20

Linkbait aside there are some reasonable points being made here, specifically "Failover means downtime" about HBase. We run into this pretty much every day in one of our customer facing applications or APIs and it's quite frustrating to have to explain that there's very little you can do to prevent it. I haven't looked too closely at MapR yet, but "instant recovery, seamless sharding and high availability" are impres…

Which version of HBase are you running? If we have a node or two go down in our cluster, HBase is completely unaffected. Recoveries can take up to a few minutes, in the old ages it took hours. I know that MTTR (mean time to recovery) is being worked on together by several large companies to get down to seconds.

Also, if you use the Thrift service as an interface to HBase (we do), you can have it running on several machines on the cluster and be able to set them up using a failover architecture - as long as the cluster itself is healthy, Thrift should work from any machine you have it running on.

Re: Big Data Debate: HBase

#38

Post by duaneb »

monstrado wrote: post #35
duaneb wrote: post #34

Earlier quoted context omitted.

> Like you mentioned, a lot of people use HBase as a data-store, it's incredibly good at that. This is exactly what confuses me about NoSQL. It's not really comparable to relational databases on the CA[P] spectrum.

HBase doesn't claim to be some NoSQL database or equivalent to a relational database, anyone who thinks otherwise hasn't actually read into HBase. Just look at HBase's website which describe it exactly as > Apache HBase is the Hadoop database, a distributed, scalable, big data store.

I mean, I understand this, but plenty of people identify it as NoSQL. I'm sure the people on HBase are intelligent enough to understand it's a meaningless phrase that would make them look silly.

EDIT: By which I mean, "NoSQL" doesn't even make sense as an approach by name. ACID relational databases and high availability data stores both have their places so evangelizing on either side is just silly.

Though I would like to point people toward http://research.google.com/pubs/pub38125.html.

Re: Big Data Debate: HBase

#39

Post by samspenc »

Heavy HBase user here. My two cents, FWIW.

I totally identify with both sides of this article, but if we had to do this again, we would probably go for HBase again. Its quite a pain to manage unless you have someone on your team with a PhD in HBase, but:

1. The main HBase committers are also the ones who contribute to the Hadoop project, so its in a forward trajectory with good velocity, since its fairly coupled to the Hadoop ecosystem. Cloudera is a big contributor, as are Facebook and Salesforce.

2. Facebook, Adobe and other BigTM companies use it in production and at scale. (Granted they have armies of smart people to maintain HBase.)

3. For all the pain it is, its reasonably documented and has a growing community that fills in the gap. I could be totally totally wrong about this, but Cassandra doesn't have that same level of community as HBase does.

Re: Big Data Debate: HBase

#40

Post by linuxhansl »

Oh please.

On the one side we have a commercially competing entity arguing against HBase and on the other side we have an individual "defending" HBase who never contributed a single line of code to open source HBase promoting their own closed source solution.

HBase is a "Sparse, Consistent, Distributed, Multidimensional, Sorted map" and at that it is pretty good.

Other stores are either not consistent or not sorted by default (which means you cannot do range scans). Some can be configured to do that, but then suddenly all those nice claims made by the commercial entities backing them just vanish.

If you do not need consistency and range scans, then do not use HBase. If you do need those HBase will be a excellent fit.

Some of the most heavyweight entities on this planet have committers to HBase (Facebook, Intel, Salesforce, to some extend Twitter, etc) as well as commercial backers such as Cloudera and Hortonworks.

Disclaimer: HBase committer here.

Post reply on HN