Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Australian Square Kilometre Array Pathfinder hits the big-data highway

blog.csiro.au

Post reply on HN
Print view View original

31–40 of 52 posts

Re: The Australian Square Kilometre Array Pathfinder hits the big-data highway

#31

Post by sandworm101 »

discodave wrote: post #17
wyager wrote: post #15

Earlier quoted context omitted.

> A large chunk of that capacity will be for private use, What private use requires that much bandwidth?

Inter region traffic of large cloud providers.

Id call the inner movements of data within google/facebook private networks part of the internet. When i do a google search, or watch a youtube vid, i know i am generating more traffic than the http requests and answer from the edge of google's network. The inner workings of the largest private networks often service the internet, if not by name, and therefore should be included in its total bandwidth. But good luck measuring it.

Re: The Australian Square Kilometre Array Pathfinder hits the big-data highway

#32

Post by Infernal »

nickspacek wrote: post #28
tangue wrote: post #22

Earlier quoted context omitted.

Cisco went to court... and lost. Here's an article where I 've learned that there's a stylized version of the Golden Gate in Cisco's logo and that CSIRO's logo is the shape of Australia. http://www.itnews.com.au/news/csiro-beats-cisco-in-fight-ove...

I'm embarrassed to admit that I never realized that "Cisco" was, in addition to the company's name, short/slang for San Francisco, nor that their logo was meant to evoke the Golden Gate Bridge.

Seconded. Never really thought about the word "Cisco", and assumed the logo was intended to evoke RF on an oscilloscope or some other visualization of electronic signaling.

Re: The Australian Square Kilometre Array Pathfinder hits the big-data highway

#33

Post by wlesieutre »

snerbles wrote: post #16
semi-extrinsic wrote: post #8

Earlier quoted context omitted.

Are those Metric terabytes (1000^4), or in freedom units (1024^4)?

Should be tebibytes (TiB) for the latter, though binary prefixes don't have much adoption outside of the software world. http://www.physics.nist.gov/cuu/Units/binary.html

Or inside of the software world, for that matter.

Re: The Australian Square Kilometre Array Pathfinder hits the big-data highway

#34

Post by ChuckMcM »

angrygoat wrote: post #30
satysin wrote: post #19

How do you even begin to deal with that much data per second? What do they do with it? Storing more than a few minutes of data isn't practical is it? Do they process it in realtime?

This from CSIRO has a lot of detail answering your question – CSIRO ASKAP Science Data Archive: Overview, Requirements and Use Cases http://www.atnf.csiro.au/projects/askap/ASKAP_SW_0017_v1.0_d...

Perfect, and it shows how 36 antenna get correlated down to 2.5 gigabytes per second.

Re: The Australian Square Kilometre Array Pathfinder hits the big-data highway

#35

Post by ismail »

privong wrote: post #2

Please note, this is for a Square Kilometer Array pathfinder (the Australian Square Kilometre Array Pathfinder, ASKAP), not the Square Kilometer Array. Construction hasn't even begun on the latter, AFAIK.

References:

https://en.wikipedia.org/wiki/Square_Kilometre_Array

https://en.wikipedia.org/wiki/Australian_Square_Kilometre_Ar...

Re: The Australian Square Kilometre Array Pathfinder hits the big-data highway

#36

Post by autocorr »

satysin wrote: post #19

How do you even begin to deal with that much data per second? What do they do with it? Storing more than a few minutes of data isn't practical is it? Do they process it in realtime?

You're right in the last point, the raw data has such a high rate that they have to process it on the fly, which has its drawbacks. The reason that these instruments output so much data is that the process of correlating the signal between each unique pair of antennas is O(n^2) in the number of antennas. Now 36 antennas (when complete) is not very large, that's approximately the size of the VLA[1], but because of the very unique phased array feeds on ASKAP mentioned in the article, each antenna has not just one receiver, but a little camera of receivers. So if it's an 8x8 feed/receiver array, that's 64 times the data rate of a 36 antenna system. For comparison, I've had projects on the VLA that are about 10 GB/S, but they have only a single feed/receiver. At ASKAP I believe they have to process/"reduce" the data on the fly, which in the most extreme case (likely necessary for the real Square Kilometer Array) is to not store the raw correlations between antennas at all but to convert it to images immediately, which is lossy. Normally this reduction to images is optimized by carefully tuning the imaging parameters and iterating to find the best, but this can't be done if the "visibilities"/correlations have been thrown away.

[1] Edit: saying that the Very Large Array does not have a very large number of antennas is actually pretty ironic.

Re: The Australian Square Kilometre Array Pathfinder hits the big-data highway

#37

Post by autocorr »

ChuckMcM wrote: post #29
satysin wrote: post #19

How do you even begin to deal with that much data per second? What do they do with it? Storing more than a few minutes of data isn't practical is it? Do they process it in realtime?

In parallel :-) Divide that number by the number of telescopes in operation (in this case 12) and you're fetching 433 GB/sec on a telescope. Imagine that is I/Q data coming from the antenna. Now you're only looking for neutral hydrogen which has a much narrower spectrum, so you process that data with a band pass decimating filter, that takes you down to a few GB/sec. Then take the redshift into account and process ev…

The example of being able to reduce the data rate by only configuring the system to look in a narrow frequency range in the spectra is perceptive! Although unfortunately you cannot divide the data rate down by the number of antennas by processing each antenna in parallel. Interferometers such as this must correlate the signals between each unique pair of antennas and in approximate real time (1 sec to 1 min range). The data rate comes from sending the signal of each individual antenna by fiber or waveguide to a special FPGA/ASIC super computer called a correlator that computes the correlated signal between each unique pair of antennas. So it all goes into the hopper at once.

Now what you're saying can actually be done, and is done on very long baseline facilities like the VLBA or Event Horizon Telescope (EHN). As an example, the EHN uses hydrogen atomic clocks to essentially time stamp the signal from a single antenna into a 50 TB disk pack, where each of the antennas are off by themselves from Hawaii, Spain, to the South Pole. They then bring all the disk packs to MIT where they then do the correlation from the time stamped data streams. This process is actually very complex however and it requires more total processing in the end. The higher computation comes from having to repeatedly process the data searching for subtle time offsets and antenna position offsets to find the correlation (this is how they use interferometers to measure those 1 cm/year continental drifts).

Re: The Australian Square Kilometre Array Pathfinder hits the big-data highway

#38

Post by TeMPOraL »

sandworm101 wrote: post #27
tangue wrote: post #22

Earlier quoted context omitted.

Cisco went to court... and lost. Here's an article where I 've learned that there's a stylized version of the Golden Gate in Cisco's logo and that CSIRO's logo is the shape of Australia. http://www.itnews.com.au/news/csiro-beats-cisco-in-fight-ove...

Honestly, i never linked the cisco logo to the goldengate. It looked like meters of an audio output or something else electronic, an o-scope. The proportions seem off, probably because i never see the bridge from the side. Im betting they want to play down the link to the city name. "Frisco" is a week trademark because of its common use. Cisco is akin to the "Syfy" channel. You might win on the letters, but not the s…

You win on letters, you lose on i18n. "Syfy" in Polish is plural of "syf", which means disorder, dirt, something ugly, or (probably also the origin of the word) syphilis.

Obviously, this made telling some friends about some shows I watched rather awkward. ;).

Re: The Australian Square Kilometre Array Pathfinder hits the big-data highway

#39

Post by semi-extrinsic »

wlesieutre wrote: post #33
snerbles wrote: post #16

Earlier quoted context omitted.

Should be tebibytes (TiB) for the latter, though binary prefixes don't have much adoption outside of the software world. http://www.physics.nist.gov/cuu/Units/binary.html

Or inside of the software world, for that matter.

Huh? 1024^4 is how Windows defines a terabyte, accordingly that's what most people think a terabyte is. OS X was the first OS to switch to 1000^4 with Snow Leopard (2009). Different Linux system tools use either standard.

Re: The Australian Square Kilometre Array Pathfinder hits the big-data highway

#40

Post by billiam »

sandworm101 wrote: post #31
discodave wrote: post #17

Earlier quoted context omitted.

Inter region traffic of large cloud providers.

Id call the inner movements of data within google/facebook private networks part of the internet. When i do a google search, or watch a youtube vid, i know i am generating more traffic than the http requests and answer from the edge of google's network. The inner workings of the largest private networks often service the internet, if not by name, and therefore should be included in its total bandwidth. But good luck…

Nah, it is mostly traffic to manage the computing activity of large enterprises, not directly related to what we consider the Internet.
Post reply on HN