A Pythonic way to do MapReduce using hadoop

Post by jcsalterego » Sun, Feb 20, 2011, 10:13 PM UTC

  for line in sys.stdin:
      name, marks = line.rstrip().split('\t')
      try:
          if agregatedmarks.get(name):
              agregatedmarks.get(name).append(marks)
          else:
              agregatedmarks[name] = [marks]
      except ValueError:
          pass

  for line in sys.stdin:
      name, marks = line.rstrip().split('\t')
      if name in aggregatedmarks:
          aggregatedmarks.append(marks)
      else:
          aggregatedmarks[name] = [marks]

Post by ferdous » Mon, Feb 21, 2011, 2:57 PM UTC

beagle3 wrote: post #3

jcsalterego wrote: post #2

Great overview, from top to bottom. Just wondering, is it better performance-wise to explicitly check for key membership than to rely on exceptions? Existing: for line in sys.stdin: name, marks = line.rstrip().split('\t') try: if agregatedmarks.get(name): agregatedmarks.get(name).append(marks) else: agregatedmarks[name] = [marks] except ValueError: pass Or for line in sys.stdin: name, marks = line.rstrip().split('\t'…

aggregatedmarks.setdefault(name,[]).append(marks) See http://docs.python.org/release/2.5.2/lib/typesmapping.html