Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hadoop Reaches 1.0

hadoop.apache.org

Post reply on HN
Print view View original

21–30 of 31 posts

Re: Hadoop Reaches 1.0

#21

Post by jshen »

tlipcon wrote: post #17
paraschopra wrote: post #4

Can someone describe differences from previous version? Or just this means Hadoop is now "production ready"?

The 1.0.0 release is actually formerly known as the 0.20.205.1 release -- ie just bugfixes since 0.20.205. Hadoop's been "production ready" for years - there are hundreds of companies running it in business critical applications. But some people want to see "1.0" before they move to production :) So we recently decided to call it 1.0 so that the version numbering matches the maturity Hadoop has already achieved. -Tod…

Have they figured out which API they are using? You know, the old deprecated vs the new one, which last time I useed hadoop was missing features that required me to use the old API. Even though they had @deprecated all over the old API.

Re: Hadoop Reaches 1.0

#22

Post by cbsmith »

tlipcon wrote: post #17
paraschopra wrote: post #4

Can someone describe differences from previous version? Or just this means Hadoop is now "production ready"?

The 1.0.0 release is actually formerly known as the 0.20.205.1 release -- ie just bugfixes since 0.20.205. Hadoop's been "production ready" for years - there are hundreds of companies running it in business critical applications. But some people want to see "1.0" before they move to production :) So we recently decided to call it 1.0 so that the version numbering matches the maturity Hadoop has already achieved. -Tod…

I think webhdfs and the option for datanode bypass on the local clients qualifies as major new features though. hftp was such a PITA and a lot of mappers are now going to go a LOT faster.

Re: Hadoop Reaches 1.0

#23

Post by scott_s »

mun2mun wrote: post #20
LeafStorm wrote: post #8

Earlier quoted context omitted.

A good introduction to MapReduce is probably CouchDB, where you use it for database views instead of SQL-style queries. The basic concepts are: - The "Map" phase takes a key/value pair of input and produces as many other key/value pairs of output as it wants. This can be zero, it can be one, or it can be over 9000. Each Map over a piece of input data operates in isolation. - The "Reduce" phase takes a bunch of values…

I have a question. I have read somewhere that map-reduce can leverage parallelism. So if I map a function to an array every element in the array is mapped with that function so that they can be executed parallely because they have no dependency on each other. But how do reduce leverage parallelism? As far as I understand output of the reduce function is dependent on the previous value.

In principle, reductions can often be staged, since there's no ordering requirements. Imagine a tree of reductions. But you are correct, the reduce phase is what will limit parallelism. If you have a cheap map operation, but a really expensive reduction, you may not see much scalability. (Where "scalable" is a way of saying "performance improves as available hardware increases because more parallelism inherent in the application is exploitable.")

Re: Hadoop Reaches 1.0

#25

Post by stingraycharles »

firemanx wrote: post #9

Awesome! Now if we can just get HBase to update it's prereqs and bump it's version, I can have some symmetry in my life! On a more serious note - is anyone using HDFS for something like the WebHDFS stuff was designed? We're currently looking at HDFS right now for an Event Store mechanism, but it appears to me to be pretty large file / stream oriented, and I'm wondering how it will stack up if we want to do something…

If you settle with Java or a bit of java extensions, you can probably write your own TaskSplitter and define a way that hadoop should distribute your jobs into smaller tasks. Be aware: you might end up either having a lot of trouble getting the 'optimal splits', or you'll lose one of Hadoop's major advantages, data (calculation) locality (for example, when you decide to combine 10 smaller files into a single task, and you have 10 different DataNodes, chances are small that all files are stored on the machine that's performing the MapReduce task).

One thing to note, though: HDFS is indeed very stream oriented. It works in blocks of 64 MB (by default), and only sends data upstream when you either close a file or a full block is available to be written. So, when your servers crashes at 63MB, and you have unrecoverable data, you'll have lost all 63MB of data. That was one of the big caveats we had to work around for our own problems we solve with Hadoop.

Re: Hadoop Reaches 1.0

#26

Post by tlipcon »

stingraycharles wrote: post #25
firemanx wrote: post #9

Awesome! Now if we can just get HBase to update it's prereqs and bump it's version, I can have some symmetry in my life! On a more serious note - is anyone using HDFS for something like the WebHDFS stuff was designed? We're currently looking at HDFS right now for an Event Store mechanism, but it appears to me to be pretty large file / stream oriented, and I'm wondering how it will stack up if we want to do something…

If you settle with Java or a bit of java extensions, you can probably write your own TaskSplitter and define a way that hadoop should distribute your jobs into smaller tasks. Be aware: you might end up either having a lot of trouble getting the 'optimal splits', or you'll lose one of Hadoop's major advantages, data (calculation) locality (for example, when you decide to combine 10 smaller files into a single task, an…

This isn't quite true - data is streamed from the client through a pipeline made up of all of the replicas, as it's written. It's true you'll lose data if you crash in the middle of a block, _unless_ you call the sync() function which makes sure the data has been fully replicated to all of the nodes.

Re: Hadoop Reaches 1.0

#27

Post by tlipcon »

jshen wrote: post #21
tlipcon wrote: post #17

Earlier quoted context omitted.

The 1.0.0 release is actually formerly known as the 0.20.205.1 release -- ie just bugfixes since 0.20.205. Hadoop's been "production ready" for years - there are hundreds of companies running it in business critical applications. But some people want to see "1.0" before they move to production :) So we recently decided to call it 1.0 so that the version numbering matches the maturity Hadoop has already achieved. -Tod…

Have they figured out which API they are using? You know, the old deprecated vs the new one, which last time I useed hadoop was missing features that required me to use the old API. Even though they had @deprecated all over the old API.

Both APIs are available and will continue to be available for the foreseeable future.

-Todd

Re: Hadoop Reaches 1.0

#28

Post by fletchowns »

digitalsushi wrote: post #5

Hadoop reaches 1.0 and my understanding of how to use it is still in development. Does anyone have a high level resource of how MapReduce works for mediocre programmers like myself that are late to the game? I know she's not ready to have my babies, but surely I could get to know her a little, maybe just be friends? I grabbed a Hadoop pre-made virtual machine the other month and was surely so far over my head that I…

Here's a nice explanation: http://ayende.com/blog/4435/map-reduce-a-visual-explanation

And a nice simple little hadoop setup: http://hadoop.apache.org/common/docs/current/single_node_set...

Re: Hadoop Reaches 1.0

#29

Post by jshen »

tlipcon wrote: post #27
jshen wrote: post #21

Earlier quoted context omitted.

Have they figured out which API they are using? You know, the old deprecated vs the new one, which last time I useed hadoop was missing features that required me to use the old API. Even though they had @deprecated all over the old API.

Both APIs are available and will continue to be available for the foreseeable future. -Todd

My point is that I would assume a 1.0 release would have a clear "right way". If I'm starting a fresh project is the new API the right one?

Re: Hadoop Reaches 1.0

#30

Post by stingraycharles »

tlipcon wrote: post #26
stingraycharles wrote: post #25

Earlier quoted context omitted.

If you settle with Java or a bit of java extensions, you can probably write your own TaskSplitter and define a way that hadoop should distribute your jobs into smaller tasks. Be aware: you might end up either having a lot of trouble getting the 'optimal splits', or you'll lose one of Hadoop's major advantages, data (calculation) locality (for example, when you decide to combine 10 smaller files into a single task, an…

This isn't quite true - data is streamed from the client through a pipeline made up of all of the replicas, as it's written. It's true you'll lose data if you crash in the middle of a block, _unless_ you call the sync() function which makes sure the data has been fully replicated to all of the nodes.

Hadoop only writes a block from a client to a DataNode when a whole block is available. This is to minimize the amount of "open connections" in the datanodes (it can take a long time for the client to generate 64MB of data, while distributing the block over the replicas can occur in a relatively short time).

For more information about this, see: http://hadoop.apache.org/common/docs/current/hdfs_design.htm... and http://hadoop.apache.org/common/docs/current/hdfs_design.htm...

Post reply on HN