Live data from Hacker News

The Australian Square Kilometre Array Pathfinder hits the big-data highway

blog.csiro.au

21–30 of 52 posts

Re: The Australian Square Kilometre Array Pathfinder hits the big-data highway

#21

Stilling waiting for when we put a radio telescope at L3 and L2 giving mankind a 14Tm baseline interferometry. The current 12Mm interferometry is nice, but earth is so small.

We have something along those lines, albeit not quite as long-baseline. Spektr-R is a radio satellite on a elliptical orbit that gives baselines up to 350,000 km (0.35 Gm), linked to Earth-based facilities.

https://en.wikipedia.org/wiki/Spektr-R

Re: The Australian Square Kilometre Array Pathfinder hits the big-data highway

#22

The csiro logo they have looks very familiar: https://regmedia.co.uk/2012/10/08/cisco.jpg https://www.csiro.au/~/media/Web-team/Images/CSIRO_Logo/logo...

Cisco went to court... and lost. Here's an article where I 've learned that there's a stylized version of the Golden Gate in Cisco's logo and that CSIRO's logo is the shape of Australia.

http://www.itnews.com.au/news/csiro-beats-cisco-in-fight-ove...

Re: The Australian Square Kilometre Array Pathfinder hits the big-data highway

#23

The csiro logo they have looks very familiar: https://regmedia.co.uk/2012/10/08/cisco.jpg https://www.csiro.au/~/media/Web-team/Images/CSIRO_Logo/logo...

FWIW, The CSIRO logo is in the shape of Australia (the dot representing Tasmania), as it's an Australian organisation. It also sort of looks like WiFi signal bars to me, and I guess the CSIRO has some legitimacy for that seeing as they were involved in inventing WiFi: https://www.csiro.au/en/Research/D61/Areas/Wireless-and-netw...

Re: The Australian Square Kilometre Array Pathfinder hits the big-data highway

#24
post #19

How do you even begin to deal with that much data per second? What do they do with it? Storing more than a few minutes of data isn't practical is it? Do they process it in realtime?

From the article:

> Once out of the telescope, the data is going through a new, almost automatic data-processing system we’ve developed.

Of course, they don't go into detail (instead making a very odd analogy to a bread machine), but I think the implication is that the data is processed immediately before being recorded. A similar approach is used at the LHC.

A better analogy might be; when you take a picture on your phone, it comes out of the camera chip in a form equivalent to a bitmap file. But those are big and a pain to deal with, so its converted to jpeg or something else reasonable before being stored, perhaps losing a bit of irrelevant noise in the process.

Re: The Australian Square Kilometre Array Pathfinder hits the big-data highway

#25
post #9
post #4

Actually, it generates 5.2T B /sec. "Its antennas are now churning out 5.2 terabytes of data per second"

Either case, I doubt it is "15% of the internets current data rate" as they claim. E.g. the newish submarine cable between the US and Japan has 60 Tb capacity. And that's just one cable.

To clarify definitions, data rate will always be less than or equal to capacity. In practice, average rate will be much less than peak capacity.

Re: The Australian Square Kilometre Array Pathfinder hits the big-data highway

#26

Stilling waiting for when we put a radio telescope at L3 and L2 giving mankind a 14Tm baseline interferometry. The current 12Mm interferometry is nice, but earth is so small.

Radio scintillation of the Interstellar Medium becomes an issue, I daresay the radio equivalent of "active optics" could help.

I did mention this at an outreach talk with Astroblack morphologies[0] and Tim O'Brien[1] who pointed out that ISM scintillation was on his slides

I did think up 100s of radio telescopes in a bird cage orbit at the distance of Jupiter, impractical of course but Space VLBI does reward thinking big.

Isn't it worth having telescopes well out the ecliptic plane, I seem to recollect VLBI is about filling in the UV plane. It was a long time ago, and it was just a second year physics project.

[0] - http://www.artscatalyst.org/astro-black-morphologies-flow-mo...

[1] - http://www.jb.man.ac.uk/~tob/

Re: The Australian Square Kilometre Array Pathfinder hits the big-data highway

#27
post #22

The csiro logo they have looks very familiar: https://regmedia.co.uk/2012/10/08/cisco.jpg https://www.csiro.au/~/media/Web-team/Images/CSIRO_Logo/logo...

Cisco went to court... and lost. Here's an article where I 've learned that there's a stylized version of the Golden Gate in Cisco's logo and that CSIRO's logo is the shape of Australia. http://www.itnews.com.au/news/csiro-beats-cisco-in-fight-ove...

Honestly, i never linked the cisco logo to the goldengate. It looked like meters of an audio output or something else electronic, an o-scope. The proportions seem off, probably because i never see the bridge from the side.

Im betting they want to play down the link to the city name. "Frisco" is a week trademark because of its common use. Cisco is akin to the "Syfy" channel. You might win on the letters, but not the spoken sounds ... As oppossed to the famous canadian case of "MikeRoeSoft.com" where sound trumped letters.

Re: The Australian Square Kilometre Array Pathfinder hits the big-data highway

#28
post #22

The csiro logo they have looks very familiar: https://regmedia.co.uk/2012/10/08/cisco.jpg https://www.csiro.au/~/media/Web-team/Images/CSIRO_Logo/logo...

Cisco went to court... and lost. Here's an article where I 've learned that there's a stylized version of the Golden Gate in Cisco's logo and that CSIRO's logo is the shape of Australia. http://www.itnews.com.au/news/csiro-beats-cisco-in-fight-ove...

I'm embarrassed to admit that I never realized that "Cisco" was, in addition to the company's name, short/slang for San Francisco, nor that their logo was meant to evoke the Golden Gate Bridge.

Re: The Australian Square Kilometre Array Pathfinder hits the big-data highway

#29
post #19

How do you even begin to deal with that much data per second? What do they do with it? Storing more than a few minutes of data isn't practical is it? Do they process it in realtime?

In parallel :-) Divide that number by the number of telescopes in operation (in this case 12) and you're fetching 433 GB/sec on a telescope. Imagine that is I/Q data coming from the antenna. Now you're only looking for neutral hydrogen which has a much narrower spectrum, so you process that data with a band pass decimating filter, that takes you down to a few GB/sec. Then take the redshift into account and process everything that is in your 'target range' (a red shift of .26) plus or minus a bit and now your down to perhaps megabytes per second per telescope.

Re: The Australian Square Kilometre Array Pathfinder hits the big-data highway

#30
post #19

How do you even begin to deal with that much data per second? What do they do with it? Storing more than a few minutes of data isn't practical is it? Do they process it in realtime?

This from CSIRO has a lot of detail answering your question – CSIRO ASKAP Science Data Archive: Overview, Requirements and Use Cases

http://www.atnf.csiro.au/projects/askap/ASKAP_SW_0017_v1.0_d...

Post reply on HN