Live data from Hacker News

Archivists are uploading hundreds of random VHS tapes to the internet

vice.com

31–40 of 129 posts

Re: Archivists are uploading hundreds of random VHS tapes to the internet

#31

How does one archive data for long term storage in 2020? From what little I've read, all of the media accessable to the layman has an archive lifetime of less than 30 years before physical degradation- NAND, tape, disc, whatever. That makes for a brittle civilization when the vast majority of our knowledge is stored on media and would be unrecoverable just 3 decades after a global calamity.

If you want data to survive you've always needed to copy it. Digital storage just makes that easier to do in bulk. Copying Bibles was a full time task for huge teams of monks but you can (and should) make backups routinely on a daily or weekly basis with barely a thought.

The LOCKSS project is an approach used in university networks around the world to preserve research data for the long term.

https://www.lockss.org/

The big issue isn't the technology, it's the vast amounts of data that are being created at this point. Storage is cheap, but the labor that goes into managing the longevity of datasets isn't: it's essentially continually keeping your infrastructure up-to-date whilst also ensuring the integrity and readability of the datasets as was intended when they were first created. It implies regular checks of bit integrity, readability of your data, checking that you can restore your data, ensuring that you can access the data, making sure that you can find the data and everything is catalogued, ensuring that you have the rights and license to use the data,...

When it comes to physical archives of the past, you have to be aware of your own survivorship bias. We only have an idea of what is preserved to the extent that documents are archived, recorded and thus discoverable.

What we do not know is how much knowledge and information was lost to the past. When you look at documents, you're always limited to what's there. And when you hit the boundaries of what's there, then you may have indications that there was far more in the past, but you have to conclude: sadly that's lost. Either because it is physically lost, or because it might be somewhere in the archive but it's not registered yet in a catalogue and therefor not accessible.

That's why I think that making backups with "barely a thought" is only as effective as to the extent to which you have organized your data, used accessible / readable data formats and filesystems.

For instance, most people these days generate endless streams of photos with their digital devices, which then get automagically uploaded to cloud services. And that's great. The downside of that is that your ability to find a specific picture from 5 years ago is entirely restricted to the extent that you were able to organize and add specific metadata to that picture. Let alone, if you did take the opportunity to do so.

That's why I advise people to sit down, and take time to go through their digital albums to pick the nicest or most important pictures they have, print them out on quality photo paper in several copies and store them with labels in albums at different physical locations.

When it comes to longevity, your physical albums will still be accessible to your descendants some 70 or 100 years down the line. Something that isn't remotely guaranteed by cloud solutions.

And that's just photos. Consider e-mail or the countless of closed messaging apps you have been using these past years. And then scale the problem beyond the personal but to entirety of large organizations, many of which are required by law to keep an archive of their documents, correspondence and so on, not just for decades but sometimes also for perpetuity.

Re: Archivists are uploading hundreds of random VHS tapes to the internet

#32
post #18

I'm almost afraid to write this as I fear it will be removed soon and am surprised it hasn't been already but this was the first thing I looked for... https://archive.org/details/star-wars-ONTV-Early-80s/Star+Wa...

To hell with that ugliness, go all-out and look for a torrent of Star Wars Despecialized Editions. As good as you can get the originals, these days.

Team Negative 1's scan & restorations of 35mm prints are my favorites. Empire's not done yet, but the other two are.

Re: Archivists are uploading hundreds of random VHS tapes to the internet

#33
post #14

How does one archive data for long term storage in 2020? From what little I've read, all of the media accessable to the layman has an archive lifetime of less than 30 years before physical degradation- NAND, tape, disc, whatever. That makes for a brittle civilization when the vast majority of our knowledge is stored on media and would be unrecoverable just 3 decades after a global calamity.

After going down this rabbit hole, I concluded that M-Disc is the right trade off at the moment. They’re not too expensive, the writers are available, they’ll last, and they could be reverse engineered if discovered in the future. The second choice is using hard drives (easily available) and every so often power them up and copy data to new drives. If you have a small quantity of data, then encode and laser print ont…

I also came across M-Disc but iirc there is no real verification of the longevity claims outside of marketing from the manufacturer.

>The second choice is using hard drives (easily available) and every so often power them up and copy data to new drives.

I suppose the question is whether doing so could remain under the error correction threshold indefinitely, since there will be errors accumulating both during copying and over time in cold storage. If manufacture of new drives stops, it also isn't clear to me if only the data stored on them has a 30 year life or if the medium itself decays regardless of whether it is in use or not.

In theory I imaging keeping an unused NAND or even magnetic drive in cool dry storage should preserve it's physical integrity indefinitely...

Re: Archivists are uploading hundreds of random VHS tapes to the internet

#34

How does one archive data for long term storage in 2020? From what little I've read, all of the media accessable to the layman has an archive lifetime of less than 30 years before physical degradation- NAND, tape, disc, whatever. That makes for a brittle civilization when the vast majority of our knowledge is stored on media and would be unrecoverable just 3 decades after a global calamity.

Do what archivists do - copy. The challenge is keeping things accessible. Only copying does that for electronic media. That includes getting things like photos printed.

How many times can you copy digital media before it degrades beyond error correction? I don't know the error rate per copied bit.

I think this is an excellent place for neural networks. They can preserve vast amounts of data compactly for many data types because they statistically compress high level abstract data which can then be used to fill in regions with high error rate, although if you did that at a large scale you'd probably end up with some constant error rate fluctuating around the average true value.

All indications point to the fact that we seem to be working against the unstoppable Force of entropy - indefinite error free preservation of data is ultimately impossible.

Re: Archivists are uploading hundreds of random VHS tapes to the internet

#35
post #21

Earlier quoted context omitted.

> They’re not too expensive, the writers are available, they’ll last, and they could be reverse engineered if discovered in the future. IIRC, the writers for M-DISCs are special, but the reader can be any DVD or Blu-ray drive. Honestly, I think the biggest consideration for digital archival media isn't so much the longevity of the media, but the future availability of equipment to read it. That's one of the biggest b…

> the decoding software is very common. [Citation needed]

You want a citation for the claim that many people are literate and/or can visually process images?

Re: Archivists are uploading hundreds of random VHS tapes to the internet

#36
post #29

Earlier quoted context omitted.

I have 30 year old CDs that play as good as the first time, as well as older vinyl discs. I'm sure I own 20 year old DVDs in same condition.

> I have 30 year old CDs that play as good as the first time, as well as older vinyl discs. Mass produced CDs are produced using a different process than the one consumer CDR writers use, and they are thus a much more stable storage medium. The data-containing layer is literally formed out of metal using a kind of mold: https://en.wikipedia.org/wiki/Compact_Disc_manufacturing#Ele... CDRs are written by altering a lay…

Yep...not to mention all discs are ultimately made of plastic which is subject to environmental degradation - primarily oxidation from atmosphere and UV damage, both which can lead to yellowing/clouding of the transparent medium, and eventually brittleness and fracture.

Re: Archivists are uploading hundreds of random VHS tapes to the internet

#37

How does one archive data for long term storage in 2020? From what little I've read, all of the media accessable to the layman has an archive lifetime of less than 30 years before physical degradation- NAND, tape, disc, whatever. That makes for a brittle civilization when the vast majority of our knowledge is stored on media and would be unrecoverable just 3 decades after a global calamity.

Honestly, paper lasts thousands of years. Maybe the solution is microprint and OCR? How long does microfiche last?

Re: Archivists are uploading hundreds of random VHS tapes to the internet

#38
post #29

Earlier quoted context omitted.

I have 30 year old CDs that play as good as the first time, as well as older vinyl discs. I'm sure I own 20 year old DVDs in same condition.

> I have 30 year old CDs that play as good as the first time, as well as older vinyl discs. Mass produced CDs are produced using a different process than the one consumer CDR writers use, and they are thus a much more stable storage medium. The data-containing layer is literally formed out of metal using a kind of mold: https://en.wikipedia.org/wiki/Compact_Disc_manufacturing#Ele... CDRs are written by altering a lay…

You can extend the lives of writable and rewritable optical media by proper storage. The key is to avoid light and heat as much as possible, to keep the dyes stable. I'm not sure about 30 years, but you can probably get 10 out of properly stored optical media. See the following advice:

>A disc should always be handled by grasping its outer edges, center hole or center hub clamping area. Avoid flexing the disc, exposing it to direct sunlight, excessive heat and/or humidity, handle it only when being used and do not eat, drink and smoke near it. Discs should be stored in jewel cases rather than sleeves as cases do not contact the discs’ surfaces and generally provide better protection again scratches, dust, light and rapid humidity changes. Once placed in their cases discs can be further protected by keeping them in a closed box, drawer or cabinet. For long-term storage and archival situations it is advisable to follow manufacturer instructions. For further information consult the international standards for preserving optical media (ISO 18925:2002, Imaging materials — optical disc media — storage practices). [0]

---

[0] : http://www.osta.org/technology/cdqa12.htm

Re: Archivists are uploading hundreds of random VHS tapes to the internet

#39

Earlier quoted context omitted.

Audio CDs can survive a bit of degradation to the odd data bits here or there; the music will just skip the millisecond of missing data and your ears won’t notice. Likewise with vinyl. Data discs on the other hand have the issue that if a single data bit is lost, a whole file could be corrupted, especially if it’s a zipped file. In addition, printed audio CDs are of a different build than CD-Rs which have been found…

It takes more than a single bit--there are Huffman codes built into the spec, but it is certainly possible for the bits to degrade enough to render a sector or even the whole disc unreadable.

If you're serious about using optical media as archival storage, you can mitigate this by incorporating your own error correcting codes into the data storage format.

Re: Archivists are uploading hundreds of random VHS tapes to the internet

#40

Earlier quoted context omitted.

Do what archivists do - copy. The challenge is keeping things accessible. Only copying does that for electronic media. That includes getting things like photos printed.

How many times can you copy digital media before it degrades beyond error correction? I don't know the error rate per copied bit. I think this is an excellent place for neural networks. They can preserve vast amounts of data compactly for many data types because they statistically compress high level abstract data which can then be used to fill in regions with high error rate, although if you did that at a large scal…

Considering we can read parchments thousands of years old, and stone tablets going back even further, perhaps we should focus efforts on copying what we can to the time-tested preserved data formats.
Post reply on HN