Live data from Hacker News

Hadoop Reaches 1.0

hadoop.apache.org

11–20 of 31 posts

Re: Hadoop Reaches 1.0

#11
Hadoop versioning has always been a little confusing to me:

* 0.23.0: 11 November, 2011

* 0.22.0: 10 December, 2011

Now we have 1.0, but it's based on 0.20, not any of the more recent releases?

The 1.0 release notes are pretty useless--it's just a list of issues. Is there a summary anywhere?

Re: Hadoop Reaches 1.0

#12

Hadoop reaches 1.0 and my understanding of how to use it is still in development. Does anyone have a high level resource of how MapReduce works for mediocre programmers like myself that are late to the game? I know she's not ready to have my babies, but surely I could get to know her a little, maybe just be friends? I grabbed a Hadoop pre-made virtual machine the other month and was surely so far over my head that I…

I found "Hadoop: The Definitive Guide" to be excellent (hah, I just noticed there's a quote from me on Twitter on the book's homepage): http://www.hadoopbook.com/

Re: Hadoop Reaches 1.0

#13

Hadoop reaches 1.0 and my understanding of how to use it is still in development. Does anyone have a high level resource of how MapReduce works for mediocre programmers like myself that are late to the game? I know she's not ready to have my babies, but surely I could get to know her a little, maybe just be friends? I grabbed a Hadoop pre-made virtual machine the other month and was surely so far over my head that I…

The Hadoop and HBase books from O'Reilly Media are quite good. They gave me a good overview and got me up to speed enough that I was comfortable stepping into a new personal project using them.

Re: Hadoop Reaches 1.0

#14
Congrats on the milestone to those involved - it's great to have something like this available to everyone for free.

On a side note, and not to take anything away from the H-team, I'm pretty curious on how it compares to Google's GFS and the rest of their distributed computing stack (MR, Chubby, etc.). It would be sweet if Google released some or all of these some day.

Re: Hadoop Reaches 1.0

#15

It was already prod ready in my opinion. I think this release is more of a "polish" thing since some people are timid to run "0.20" code in prod.

Agreed. Working with hadoop a few years ago was a pain in the ass, what really made it ready (at least for me) was the packaging done by Cloudera.

There's now an Apache effort for producing a fully packaged, validated and deployable stack of Hadoop components. The project is called Apache Bigtop (incubating) and the relationship with Cloudera's CDH is like a relationship between Debian and Ubuntu. We make it super easy for folks to deploy the released versions of Bigtop distribution either via packages: http://bit.ly/rHpybV or VMs: http://bit.ly/tBGmNt

Re: Hadoop Reaches 1.0

#16
I agree with an earlier comment. Big Data, Hadoop etc. are keywords that are supposed to get big in 2012, however, as a regular web dev, it's hard for me to grasp what it can do unless you have gigantic data stores

Re: Hadoop Reaches 1.0

#17

Can someone describe differences from previous version? Or just this means Hadoop is now "production ready"?

The 1.0.0 release is actually formerly known as the 0.20.205.1 release -- ie just bugfixes since 0.20.205.

Hadoop's been "production ready" for years - there are hundreds of companies running it in business critical applications. But some people want to see "1.0" before they move to production :) So we recently decided to call it 1.0 so that the version numbering matches the maturity Hadoop has already achieved.

-Todd (Hadoop PMC)

Re: Hadoop Reaches 1.0

#18

Hadoop reaches 1.0 and my understanding of how to use it is still in development. Does anyone have a high level resource of how MapReduce works for mediocre programmers like myself that are late to the game? I know she's not ready to have my babies, but surely I could get to know her a little, maybe just be friends? I grabbed a Hadoop pre-made virtual machine the other month and was surely so far over my head that I…

Don't write MapReduce. Check out my tutorials of late on http://datasyndrome.com and check out Apache Pig. That you can understand easily. It will MapReduce for you. Try Amazon Elastic MapReduce wih Pig on some logs.

Re: Hadoop Reaches 1.0

#19

Hadoop reaches 1.0 and my understanding of how to use it is still in development. Does anyone have a high level resource of how MapReduce works for mediocre programmers like myself that are late to the game? I know she's not ready to have my babies, but surely I could get to know her a little, maybe just be friends? I grabbed a Hadoop pre-made virtual machine the other month and was surely so far over my head that I…

Hadoop is much more than MapReduce. It's a fault tolerant, highly scalable file system where you can store a shitload (technical term) of data. That alone is remarkable.

Once your data is there, then you get your map/reduce on. And the best way to get started there is to look into Pig or Hive (high level map/reduce abstractions). Either of those will take you a long way.

Re: Hadoop Reaches 1.0

#20

Hadoop reaches 1.0 and my understanding of how to use it is still in development. Does anyone have a high level resource of how MapReduce works for mediocre programmers like myself that are late to the game? I know she's not ready to have my babies, but surely I could get to know her a little, maybe just be friends? I grabbed a Hadoop pre-made virtual machine the other month and was surely so far over my head that I…

A good introduction to MapReduce is probably CouchDB, where you use it for database views instead of SQL-style queries. The basic concepts are: - The "Map" phase takes a key/value pair of input and produces as many other key/value pairs of output as it wants. This can be zero, it can be one, or it can be over 9000. Each Map over a piece of input data operates in isolation. - The "Reduce" phase takes a bunch of values…

I have a question. I have read somewhere that map-reduce can leverage parallelism. So if I map a function to an array every element in the array is mapped with that function so that they can be executed parallely because they have no dependency on each other. But how do reduce leverage parallelism? As far as I understand output of the reduce function is dependent on the previous value.
Post reply on HN