Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Australian Square Kilometre Array Pathfinder hits the big-data highway

blog.csiro.au

Post reply on HN
Print view View original

41–50 of 52 posts

Re: The Australian Square Kilometre Array Pathfinder hits the big-data highway

#41

Post by sandworm101 »

TeMPOraL wrote: post #38
sandworm101 wrote: post #27

Earlier quoted context omitted.

Honestly, i never linked the cisco logo to the goldengate. It looked like meters of an audio output or something else electronic, an o-scope. The proportions seem off, probably because i never see the bridge from the side. Im betting they want to play down the link to the city name. "Frisco" is a week trademark because of its common use. Cisco is akin to the "Syfy" channel. You might win on the letters, but not the s…

You win on letters, you lose on i18n. "Syfy" in Polish is plural of "syf", which means disorder, dirt, something ugly, or (probably also the origin of the word) syphilis. Obviously, this made telling some friends about some shows I watched rather awkward. ;).

And that is why i read hn.

Re: The Australian Square Kilometre Array Pathfinder hits the big-data highway

#42

Post by [deleted] »

jankey wrote: post #9
nrki wrote: post #4

Actually, it generates 5.2T B /sec. "Its antennas are now churning out 5.2 terabytes of data per second"

Either case, I doubt it is "15% of the internets current data rate" as they claim. E.g. the newish submarine cable between the US and Japan has 60 Tb capacity. And that's just one cable.

[deleted]

Re: The Australian Square Kilometre Array Pathfinder hits the big-data highway

#43

Post by Frqy3 »

jankey wrote: post #9
nrki wrote: post #4

Actually, it generates 5.2T B /sec. "Its antennas are now churning out 5.2 terabytes of data per second"

Either case, I doubt it is "15% of the internets current data rate" as they claim. E.g. the newish submarine cable between the US and Japan has 60 Tb capacity. And that's just one cable.

Also, that is the raw output from the antenna array. This is then analysed before storing, compressing it down to a much more managable few GB/s.

Re: The Australian Square Kilometre Array Pathfinder hits the big-data highway

#44

Post by NamTaf »

Fun fact: the CSIRO is the only Australian organisation to have a 2nd level domain name, lacking the .com and just using .au. This is because they were the first organisation to use a domain name in Australia and got in before the regulation specified it to be .{com|gov|edu|etc}.au

Re: The Australian Square Kilometre Array Pathfinder hits the big-data highway

#45

Post by plugger »

Frqy3 wrote: post #43
jankey wrote: post #9

Earlier quoted context omitted.

Either case, I doubt it is "15% of the internets current data rate" as they claim. E.g. the newish submarine cable between the US and Japan has 60 Tb capacity. And that's just one cable.

Also, that is the raw output from the antenna array. This is then analysed before storing, compressing it down to a much more managable few GB/s.

It's analysed and stored in Melbourne, Victoria. The SKA Pathfinder is in Western Australia outside of Geraldton. Hence the data transfer statistics as the two sites are roughly 4,000km from one another.

Re: The Australian Square Kilometre Array Pathfinder hits the big-data highway

#46

Post by zer0t3ch »

NamTaf wrote: post #44

Fun fact: the CSIRO is the only Australian organisation to have a 2nd level domain name, lacking the .com and just using .au. This is because they were the first organisation to use a domain name in Australia and got in before the regulation specified it to be .{com|gov|edu|etc}.au

That is a surprisingly fun fact.

Re: The Australian Square Kilometre Array Pathfinder hits the big-data highway

#47

Post by discodave »

sandworm101 wrote: post #31
discodave wrote: post #17

Earlier quoted context omitted.

Inter region traffic of large cloud providers.

Id call the inner movements of data within google/facebook private networks part of the internet. When i do a google search, or watch a youtube vid, i know i am generating more traffic than the http requests and answer from the edge of google's network. The inner workings of the largest private networks often service the internet, if not by name, and therefore should be included in its total bandwidth. But good luck…

While it may be that a good chunk of private traffic is in the service of internet services, I think calling it 'the internet' is wrong technically and in spirit.

The spirit and technical definition is that fundamentally the internet is a connection between private networks. Initially it was some universities, ARPA and so on. Now it's Google, Facebook, Verizon, Amazon, AT&T and so on.

What happens on these networks is private network traffic. If some of that traffic travels long distances it doesn't make it less private. Why count inter-region traffic of cloud providers, why not count the inter-AZ traffic of AWS regions? Why not count the inter-rack traffic and so on. The difference between the internet and private networks is not about distance travelled it's about ownership. For instance if traffic goes between Google and AWS at a peering location in CA, then that is internet traffic even if the data doesn't go very far.

Re: The Australian Square Kilometre Array Pathfinder hits the big-data highway

#49

Post by discodave »

autocorr wrote: post #37
ChuckMcM wrote: post #29

Earlier quoted context omitted.

In parallel :-) Divide that number by the number of telescopes in operation (in this case 12) and you're fetching 433 GB/sec on a telescope. Imagine that is I/Q data coming from the antenna. Now you're only looking for neutral hydrogen which has a much narrower spectrum, so you process that data with a band pass decimating filter, that takes you down to a few GB/sec. Then take the redshift into account and process ev…

The example of being able to reduce the data rate by only configuring the system to look in a narrow frequency range in the spectra is perceptive! Although unfortunately you cannot divide the data rate down by the number of antennas by processing each antenna in parallel. Interferometers such as this must correlate the signals between each unique pair of antennas and in approximate real time (1 sec to 1 min range). T…

If it's per unique pair of telescopes then will the data rate scale O(n^2) to the number of dishes? If that is the case then holy balls the final SKA will produce a lot of data.

Re: The Australian Square Kilometre Array Pathfinder hits the big-data highway

#50

Post by autocorr »

discodave wrote: post #49
autocorr wrote: post #37

Earlier quoted context omitted.

The example of being able to reduce the data rate by only configuring the system to look in a narrow frequency range in the spectra is perceptive! Although unfortunately you cannot divide the data rate down by the number of antennas by processing each antenna in parallel. Interferometers such as this must correlate the signals between each unique pair of antennas and in approximate real time (1 sec to 1 min range). T…

If it's per unique pair of telescopes then will the data rate scale O(n^2) to the number of dishes? If that is the case then holy balls the final SKA will produce a lot of data.

It's pretty crazy! And yes, the number of measurements scales as N(N-1)/2, so O(N^2). The funny part about this was, the SKA has been in design for >15 or so years at this point, and they had to predict that computational hardware would be good enough to make it work when they actually start building it! The hardware in 2000 wouldn't have been anywhere close to capable of powering the SKA's correlator. This is actually also a game that they had to play with the Large Synoptic Survey Telescope (LSST)[1], breaking ground now, with the relatively pedestrian 20 TB a night for archiving.

[1] https://lsst.org

Post reply on HN