Live data from Hacker News

Ask HN: Are there any projects to do historical archiving of RF spectrum?

news.ycombinator.com

11–20 of 27 posts

Re: Ask HN: Are there any projects to do historical archiving of RF spectrum?

#11
>all the radio information as 9/11 transpired

I know it's not everything, but if you are interested in this the New York Times has an excellent feature using the ATC tapes and some recorded phone lines in the military/air defense system.

http://www.nytimes.com/interactive/2011/09/08/nyregion/911-t...

Re: Ask HN: Are there any projects to do historical archiving of RF spectrum?

#12
As vitovito and pigeons mentioned storage and collection locations are the main issues. Things like this exist, but they don't archive data and are designed for military use- Google: Wideband Recording System . They typically are mobile units that focus on fast capture rate and decently large storage (a few terabytes).

sigidwiki is worth looking at, but it's more for classification: http://www.sigidwiki.com/wiki/Signal_Identification_Guide

Re: Ask HN: Are there any projects to do historical archiving of RF spectrum?

#13
I find this fascinating because the context of transmissions occurring simultaneously is indeed ephemeral.

This data gets lost with traditional archiving, which is squarely considers context (including temporal simultaneity across multiple objects) to be metadata.

These sorts of efforts can be partially retrofitted/approximated with timecodes on existing archived material, which may also be a separate, worthwhile endeavor to pursue.

Re: Ask HN: Are there any projects to do historical archiving of RF spectrum?

#14
post #10
post #6

From my notes on a radio archiving project I'm doing: > Could you record the entire radio spectrum and extract stations and broadcasts later? > In the USA, the AM radio band is 540-1710 kHz, a spread of 1170 kHZ, > http://rtl-sdr.better-than.tv/?page_id=237 states that spectrum recordings are also a function of how many samples per second you choose to record: > 2.8msps - 44.8mbps = 5.6 MB/sec. > 2msps - 32mbps = 4 M…

Assuming most of this is dead air, don't you think this would be a highly compressible data set (even lossless)?

Highly is perhaps an overstatement, but compressible like most clear signal data, behaving similar to PCM audio.

I compressed '20090922_950_GLFS_outdoor.wav' (150 kHz - 1750 kHz) from the Perseus site [1] with some lossless compressors. In each case, I independently round-tripped the de/compression and the results were confirmed to be lossless.

The compression ratios are in line or slightly (but not overwhelmingly) better than what you'd expect [2] from audio:

  original:		 1.000
  7zip-lzma2-xz-ultra:	 0.820 (not a signal compressor)
  wavpack-normal:	 0.724
  wavpack-high:		 0.691
  wavpack-hhx6:		 0.593
I would've liked to test additional compressors. In addition to WavPack, FLAC, TAK, ALAC, OptimFROG, TTA, WMA Lossless all purport to have some level of high-resolution support, but I'm not sure how many kHz they go up to. FLAC wouldn't take the file, and the rest I don't have access to right this moment.

[1] http://www.myradiobase.de/perseus/

[2] http://wiki.hydrogenaud.io/index.php?title=Lossless_comparis...

Re: Ask HN: Are there any projects to do historical archiving of RF spectrum?

#15
As vitovito mentions, wideband data in high resolution is incredibly expensive from a storage perspective. There are entities that do it, but this also has the same privacy concerns as saving every packet that comes over a wire - in fact if you record 2.4 MHz and 5 MHz you will be picking up someone's (hopefully encrypted) private data. I don't think it is illegal, but the technical aspects of this are pretty daunting.

You would be better off decoding and storing that, but that gets into v& territory pretty quickly, depending on who and what you decode (definitely don't decrypt/crack).

Re: Ask HN: Are there any projects to do historical archiving of RF spectrum?

#16
post #10
post #6

From my notes on a radio archiving project I'm doing: > Could you record the entire radio spectrum and extract stations and broadcasts later? > In the USA, the AM radio band is 540-1710 kHz, a spread of 1170 kHZ, > http://rtl-sdr.better-than.tv/?page_id=237 states that spectrum recordings are also a function of how many samples per second you choose to record: > 2.8msps - 44.8mbps = 5.6 MB/sec. > 2msps - 32mbps = 4 M…

Assuming most of this is dead air, don't you think this would be a highly compressible data set (even lossless)?

At least in HF, this "dead air" is anything but dead (lightning, channels with lots of fading, things you only catch due to ionospheric effects, etc). Noise is basically any signal you don't care about in lower frequency bands. Also many transmissions look like wideband noise at higher frequency due to direct sequence spreadspectrum type techniques.

Re: Ask HN: Are there any projects to do historical archiving of RF spectrum?

#18
Archiving the entire spectrum can range anywhere from infeasible to impossible. First you'd need to specify the frequency ranges you want to listen to.

Then you've got a data storage problem which has been covered.

After that you've got the propagation problem. Not everything is seen from everywhere.

In any event you might want to check out http://www.reversebeacon.net/

It's a project that's got a different "simpler" archiving goal. It is just to track spots of people calling CQ in the CW bands.

Re: Ask HN: Are there any projects to do historical archiving of RF spectrum?

#19

This is super interesting! And if anyone else is curious about creating a project that monitors the range of RF amplitude and frequency at many data points around the world, then send me your contact info! I want to see a record of this spectrum (human and non-human made) in frequency and amplitude to see how much it is (and hopefully "has") changed in physical space. I think interesting things would come out of this…

Any idea if its legal? I assume it is but I have no idea.

Listening is 100% fine in the USA. Transmission without a ham license, and in certain bands, will net you some fines and maybe jail time.

Re: Ask HN: Are there any projects to do historical archiving of RF spectrum?

#20

Earlier quoted context omitted.

Any idea if its legal? I assume it is but I have no idea.

Listening is 100% fine in the USA. Transmission without a ham license, and in certain bands, will net you some fines and maybe jail time.

Any spectrum recordings would almost be guaranteed to include copyrighted content, and in the US distributing those recordings would fall afoul of the DMCA.
Post reply on HN