Live data from Hacker News

The Australian Square Kilometre Array Pathfinder hits the big-data highway

blog.csiro.au

51–52 of 52 posts

Re: The Australian Square Kilometre Array Pathfinder hits the big-data highway

#51
post #19

How do you even begin to deal with that much data per second? What do they do with it? Storing more than a few minutes of data isn't practical is it? Do they process it in realtime?

You're right in the last point, the raw data has such a high rate that they have to process it on the fly, which has its drawbacks. The reason that these instruments output so much data is that the process of correlating the signal between each unique pair of antennas is O(n^2) in the number of antennas. Now 36 antennas (when complete) is not very large, that's approximately the size of the VLA[1], but because of the…

Whoops! I misremembered in my post above, data rates on the VLA should have been around 20 MB/s not 10 GB/s, but I can't seem to edit my response above now.

Re: The Australian Square Kilometre Array Pathfinder hits the big-data highway

#52

Earlier quoted context omitted.

Or inside of the software world, for that matter.

Huh? 1024^4 is how Windows defines a terabyte, accordingly that's what most people think a terabyte is. OS X was the first OS to switch to 1000^4 with Snow Leopard (2009). Different Linux system tools use either standard.

Referring the abbreviation of "TiB" versus "TB" for 1024^4.
Post reply on HN