Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Big Data Debate: HBase

informationweek.com

Post reply on HN
Print view View original

21–30 of 56 posts

Re: Big Data Debate: HBase

#21

Post by karterk »

m0nastic wrote: post #16

At the risk of hijacking what seems like a linkbait article, I'll ask folks here a question: Does anyone have good experiences/recommendations for storing a "reasonable" amount of unstructured logs/pcap files. By "reasonable", I mean, not petabytes (maybe a couple terabytes, over time). I ask, because I keep thinking something like HBase is overkill (although one of my alternate solutions is to run an internal Openst…

Store it in HDFS as sequence files. HBase (as the article mentions) is a beast to manage with many moving parts. But, HDFS is just rock solid and very stable. If you want to analyze logs, you can do so in a batch mode (I recommend using Scalding or Pig). If you want it to be more "searchable", then take a fraction of it using a MR job and index it in Solr or ElasticSearch.

Re: Big Data Debate: HBase

#22

Post by monstrado »

duaneb wrote: post #19
monstrado wrote: post #17

Earlier quoted context omitted.

If you're comparing this to your riak, redis, tokyo cabinet, ... database than most likely. With HBase, one of it's bread and butter operations is the start and stop row scan (otherwise known as a range scan). The only thing equivalent I know of in MySQL is using windowing functions, and even then I don't think that's an appropriate comparison. I hear you though, most people equate NoSQL to some sort of KeyValue stor…

How are multi-row transactions these days? That was the largest problem when I looked last... A data store does not a database make.

I don't think that's something they are trying to solve right now. Although HBase guarantees write consistency, I think http://research.google.com/pubs/pub36726.html is the closest paper on how to go about it, but I could be wrong.

Like you mentioned, a lot of people use HBase as a data-store, it's incredibly good at that.

Re: Big Data Debate: HBase

#23

Post by karterk »

We use HBase pretty extensively and I have mixed feelings about it. On one hand, it's very clunky. Though the documentation is pretty good these days, setting up and managing a HBase cluster on production and in scale involves tackling many moving parts. Setting up Cassandra is almost a joke in comparison.

HBase-Hadoop integration is great, but Cassandra has caught up significantly on that front. If you want strong consistency, apart from HBase there is no other (non-sql) solution that's really battle tested. If you're okay with eventual consistency, you should take a hard look at Cassandra.

Re: Big Data Debate: HBase

#24

Post by threeseed »

karterk wrote: post #23

We use HBase pretty extensively and I have mixed feelings about it. On one hand, it's very clunky. Though the documentation is pretty good these days, setting up and managing a HBase cluster on production and in scale involves tackling many moving parts. Setting up Cassandra is almost a joke in comparison. HBase-Hadoop integration is great, but Cassandra has caught up significantly on that front. If you want strong c…

I don't understand why people keep thinking that Cassandra is only eventually consistent.

It can be set to any consistency level you like for both reads and writes: http://www.datastax.com/docs/1.1/dml/data_consistency

Re: Big Data Debate: HBase

#25

Post by monstrado »

trun wrote: post #20

Linkbait aside there are some reasonable points being made here, specifically "Failover means downtime" about HBase. We run into this pretty much every day in one of our customer facing applications or APIs and it's quite frustrating to have to explain that there's very little you can do to prevent it. I haven't looked too closely at MapR yet, but "instant recovery, seamless sharding and high availability" are impres…

Which version of HBase are you running? If we have a node or two go down in our cluster, HBase is completely unaffected. Recoveries can take up to a few minutes, in the old ages it took hours.

I know that MTTR (mean time to recovery) is being worked on together by several large companies to get down to seconds.

Re: Big Data Debate: HBase

#26

Post by threeseed »

duaneb wrote: post #19
monstrado wrote: post #17

Earlier quoted context omitted.

If you're comparing this to your riak, redis, tokyo cabinet, ... database than most likely. With HBase, one of it's bread and butter operations is the start and stop row scan (otherwise known as a range scan). The only thing equivalent I know of in MySQL is using windowing functions, and even then I don't think that's an appropriate comparison. I hear you though, most people equate NoSQL to some sort of KeyValue stor…

How are multi-row transactions these days? That was the largest problem when I looked last... A data store does not a database make.

I don't know about HBase but for Cassandra it should be available in the next minor update.

Single row transactions have just recently been added: http://www.datastax.com/dev/blog/lightweight-transactions-in...

Re: Big Data Debate: HBase

#27

Post by jbellis »

threeseed wrote: post #24
karterk wrote: post #23

We use HBase pretty extensively and I have mixed feelings about it. On one hand, it's very clunky. Though the documentation is pretty good these days, setting up and managing a HBase cluster on production and in scale involves tackling many moving parts. Setting up Cassandra is almost a joke in comparison. HBase-Hadoop integration is great, but Cassandra has caught up significantly on that front. If you want strong c…

I don't understand why people keep thinking that Cassandra is only eventually consistent. It can be set to any consistency level you like for both reads and writes: http://www.datastax.com/docs/1.1/dml/data_consistency

Even lightweight transactions in 2.0 for linearizability: http://www.datastax.com/dev/blog/lightweight-transactions-in...

Re: Big Data Debate: HBase

#30

Post by m0nastic »

monstrado wrote: post #18
m0nastic wrote: post #16

At the risk of hijacking what seems like a linkbait article, I'll ask folks here a question: Does anyone have good experiences/recommendations for storing a "reasonable" amount of unstructured logs/pcap files. By "reasonable", I mean, not petabytes (maybe a couple terabytes, over time). I ask, because I keep thinking something like HBase is overkill (although one of my alternate solutions is to run an internal Openst…

Disclaimer: I work at Cloudera as a Tools Developer What do you mean by unstructured? Do you mean the data has yet to be parsed into a format which could be logically grouped into columns? Or do you mean that it's deeply nested? Since log data doesn't really change, it might be overkill to use something like HBase (or any database for that matter). On the tools team at Cloudera, we've found that writing the data into…

Thanks, I've been testing out a bunch of hare-brained schemes and it didn't even occur to me to just use HDFS directly (and seeing you and karterk both suggest it helps).

Basically, at a high level, the system I'm working on aggregates and processes security information (It's a SIEM, if that product category means anything to you). At the point the logs get ingested, the server determines if they're "actionable" (which is determined by rules I load into Redis), in which case it parses them and stores them in a Postgres event table; or "not individually actionable, but may cause an action in conjunction with some other log" that I want to just store somewhere for batch processing.

I don't really need to tokenize those logs, as at the point I care about them I'm just going to be searching through them. So, they're "unstructured" in the sense that there's about 15 different collection points, each with it's own format (many just an ugly facsimile of syslog with some JSON in the middle).

So, I think your suggestion will work out very well.

Thanks again.

Post reply on HN