Live data from Hacker News

Xz format inadequate for long-term archiving (2016)

lzip.nongnu.org

51–60 of 60 posts

Re: Xz format inadequate for long-term archiving (2016)

#51
A bit of speculation here, but perhaps xz won over lzip because it has a real manpage?

lzip has the usual infuriating short summary of options with a "run info lzip for the complete manual". Also the source code repository doesn't even seem linked directly from the lzip homepage - technical considerations aren't the only thing that determines if software is "better", it also has to be well presented.

Re: Xz format inadequate for long-term archiving (2016)

#52
post #47

Earlier quoted context omitted.

Yes, programming languages come and go, but I don't see how that matters. Some future historian will either have access to a working copy of xz or they will not. If they don't, and they want to implement it, having a copy of the source code is far better than anything else you could give them. Sure, future programming languages will be quite different, but humans will certainly be able to read and understand C code.…

> If humanity has forgotten how to read C code (and lost all knowledge of it), how are they going to read this documentation you seem to prefer? Sure but are they going to remember something like, weird precedence rules (See: &), undefined behaviour, etc. Just because they want to reimplement a specific, small, program does not mean they want to relearn several languages. What you're saying could easily blow up from…

Are you saying it would be easier to implement xz from mathematical documentation than from computer program? I don't think so. I tried (multiple times) to implement algorithms from "mathematical documentation" in academic papers, and it is usually very bad, there are always missing parts. If I had a choice, I'd choose ALGOL-58 over human-language description anytime.

Re: Xz format inadequate for long-term archiving (2016)

#53

Earlier quoted context omitted.

Why would they need to recreate it solely based on 'documentation'? It is open source, the source code is the documentation. It seems just as likely that the source would survive as it is likely that some complete technical documentation would survive. Maybe they wouldn't be able to compile it (probably they would be able to compile it, I don't see why they wouldn't have some kind of computer emulator available), but…

You're exhibiting shortsightedness when it comes to "source". If I give you some RPG [1] or maybe some ALGO 58 [2] source code are you going to just compile and run it no problem? How about some FLOW-MATIC [3]? Point being that computer languages come and go. [1] https://en.wikipedia.org/wiki/IBM_RPG [2] https://en.wikipedia.org/wiki/ALGOL_58 [3] https://en.wikipedia.org/wiki/FLOW-MATIC

There are languages which achieve critical mass and stay, and languages which don't, and disappear.

RPG is still around, and IBM still sells it on their cloud. But the language is highly proprietary, so don't expect a cheap access to it.

ALGOL-58 is one of the languages which died; but ALGOL-68 is in the current debian repos, and would take under 30 seconds to install.

FLOW-MATIC has died, but COBOL is around and again, easily installable.

I think you are underestimating how much legacy software there is. For example, Fortran 77 is still actively used, and there are programs written in it every day. There is immense amount of programs written in C89. The support for those languages is likely to stay forever.

In general, I think this topic is very interesting. Imagine 1000 years have passed, and all the computers are running YEAR3000 architecture which is incompatible with all the software we have today. Archeologists discover a treasure trove of texts and binary files from 21th century internet. They know ASCII and English, but nothing else. What can they do?

The answer is surprisingly simple:

(1) Write an emulator for an simple CPU, like an ARMv5. Here is a good one: https://dmitry.gr/?r=05.Projects&proj=07.%20Linux%20on%208bi...

You'd need to manually port this code to whatever language you are using now. But this should be doable -- the software has 6000 lines of very straightforward C89 code. It does not use any OS services, nor does it rely on UB or complex language features.

(2) Use it to boot Linux (the image is included in that webpage). This allows you to run Ubuntu from 2009 on your YEAR3000 architecture.

(3) If your archive contains repository snapshot from 2009 to your machine. You can now install and run all the 20th century software on your YEAR3000 computers. Congrats!

(4) The only thing missing is graphics support. Just run x11vnc (included in the Jaunty repo) over serial port (included in dmitry.gr's emulator). VNC protocol is simple and well specified.

... and that's how I'd bootstrap 20th century computing on 30th century infrastructure. Sure, it will take some effort, -- but this only needs to be done once, and running programs will be easy from there on.

Re: Xz format inadequate for long-term archiving (2016)

#54
post #47

Earlier quoted context omitted.

Yes, programming languages come and go, but I don't see how that matters. Some future historian will either have access to a working copy of xz or they will not. If they don't, and they want to implement it, having a copy of the source code is far better than anything else you could give them. Sure, future programming languages will be quite different, but humans will certainly be able to read and understand C code.…

> If humanity has forgotten how to read C code (and lost all knowledge of it), how are they going to read this documentation you seem to prefer? Sure but are they going to remember something like, weird precedence rules (See: &), undefined behaviour, etc. Just because they want to reimplement a specific, small, program does not mean they want to relearn several languages. What you're saying could easily blow up from…

>> What you're saying could easily blow up from 'how to code C' to 'reading the GCC / Clang compiler source code to figure out how a specific UB was implemented, which the program in this specific case falls into', which I'm sure nobody wants to spend their weekend doing

There will be many, many people that will gladly dig into the minutia and technical details of arcane hardware, especially when it means making progress towards filling in the historical record. This is already the case today, there is a working https://en.wikipedia.org/wiki/Colossus_computer reconstructed just because it was historically significant.

Re: Xz format inadequate for long-term archiving (2016)

#55
post #48

Earlier quoted context omitted.

Are you saying that a failure happens which isn’t detected and when the 2nd failure occurs we notice because the data is inaccessible? Which raid s/w does this ?

Correct. All that depend on the SCSI block layer, which includes libata and thus common consumer SATA drives. A NAS or better drive will come out of the box with short error time outs, typically 70 deciseconds, and quickly issue a read error with the LBA of the offending bad sector, and the RAID can then know to obtain a copy or reconstruct from parity, write the good data to the bad sector thus fixing it. Either the…

The link you posted talks about the raid software kicking a whole disk out of the raid array when the disk takes too long to respond (basically but not exactly) due to 2 timeout variables mismatch

The post I was responding to implied a raid array could be degraded and you wouldn’t know till it completely failed

Interesting nevertheless

Re: Xz format inadequate for long-term archiving (2016)

#56
post #40

Earlier quoted context omitted.

Imagine files being moved from tape to disk to optical disc to NAS over the years. What now? https://news.ycombinator.com/item?id=16886607

You move them, then you verify the integrity of the new copy, then you can get rid of the old one. You don't need to build integrity checks and extra FEC at every level of the system redundantly. Just like in the end-to-end principle when applied to networking: you have a single strong integrity check at the very furthest endpoint possible, and then you don't build in integrity & ECC at every level of the stack, you…

Appealing to "more manual work" is not very compelling to be honest.

Re: Xz format inadequate for long-term archiving (2016)

#57
post #47

Earlier quoted context omitted.

> If humanity has forgotten how to read C code (and lost all knowledge of it), how are they going to read this documentation you seem to prefer? Sure but are they going to remember something like, weird precedence rules (See: &), undefined behaviour, etc. Just because they want to reimplement a specific, small, program does not mean they want to relearn several languages. What you're saying could easily blow up from…

>> What you're saying could easily blow up from 'how to code C' to 'reading the GCC / Clang compiler source code to figure out how a specific UB was implemented, which the program in this specific case falls into', which I'm sure nobody wants to spend their weekend doing There will be many, many people that will gladly dig into the minutia and technical details of arcane hardware, especially when it means making prog…

I think you missed the implication that I didn't state explicitly, but figured was pretty clear:

> which I'm sure nobody wants to spend their weekend doing [if their original goal was to simply reconstruct xz].

Re: Xz format inadequate for long-term archiving (2016)

#58
post #40

Earlier quoted context omitted.

You move them, then you verify the integrity of the new copy, then you can get rid of the old one. You don't need to build integrity checks and extra FEC at every level of the system redundantly. Just like in the end-to-end principle when applied to networking: you have a single strong integrity check at the very furthest endpoint possible, and then you don't build in integrity & ECC at every level of the stack, you…

Appealing to "more manual work" is not very compelling to be honest.

-_- I never appealed to 'more manual work', nor did I say that. If you refuse to understand my point and want to make up things I did not say, so be it.

Re: Xz format inadequate for long-term archiving (2016)

#59
post #58

Earlier quoted context omitted.

Appealing to "more manual work" is not very compelling to be honest.

-_- I never appealed to 'more manual work', nor did I say that. If you refuse to understand my point and want to make up things I did not say, so be it.

> you verify the integrity

Re: Xz format inadequate for long-term archiving (2016)

#60
post #58

Earlier quoted context omitted.

-_- I never appealed to 'more manual work', nor did I say that. If you refuse to understand my point and want to make up things I did not say, so be it.

> you verify the integrity

Don't be a moron. You knew perfectly well what I meant. (Did I also mean that 'you', a human, should be checking hashsums and FEC by hand for every network packet...?)
Post reply on HN