Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tar Files Created on macOS Display Errors When Extracting on Linux (2024)

aruljohn.com

Post reply on HN
Print view View original

71–80 of 109 posts

Re: Tar Files Created on macOS Display Errors When Extracting on Linux (2024)

#71

Post by fingerlocks »

Joker_vD wrote: post #16
gerdesj wrote: post #15

Earlier quoted context omitted.

thumbs.db and those weird MS alternative stream files for recording origination. filesystem attributes are for decorating files with meaning. Anything else that attempts to use filesystems in "interesting" ways is silly. Apple and MS really ought to consider why they do this sort of fragile, idiosyncratic nonsense.

But... thumbs.db is precisely not an "attempt to use filesystems in "interesting" ways" — it's literally a just hidden file with previews stored in it. Storing the preview in the alternative stream of the file with the picture itself would be "an interesting way".

And .DS_Store is just your folder level preferences in Finder. If you don’t use Finder they won’t be created

Re: Tar Files Created on macOS Display Errors When Extracting on Linux (2024)

#72

Post by mxmlnkn »

The title seems misleading.

These are not errors. They are simply warnings about extended attributes being ignored when extracting files, which seems completely fine to me, and creating the tar without those extended attributes has exactly the same outcome, but throws away the metadata at archive time instead of extraction time.

Furthermore, this is not an Apple/macOS issue. The tool used is bsdtar, so it would also affect all BSD-variants that default to bsdtar/libarchive, and those systems also have extended attributes, e.g., for SELinux, which would get added to the TAR.

Re: Tar Files Created on macOS Display Errors When Extracting on Linux (2024)

#73

Post by adrian_b »

LatencyKills wrote: post #2

Ex-Apple engineer here. This is, for better or worse, just the way Apple approaches this type of problem. From Apple's perspective, this is the way to preserve Finder / Gatekeeper / metadata semantics. It avoids silent data loss when round-tripping archives between Macs. This behavior also maintains consistency with copyfile(3) (as well as the Archive Utility behavior). Apple treats tar less like “portable Unix inter…

Yes, I completely disagree with TFA.

The problem described in TFA is not specific to Apple, but the same problem appears when archiving any decent filesystem that has been designed during the last 3 decades and not a half of century ago, including all Linux file systems.

The problem described in TFA is not caused by Apple, but by the author using an obsolete tar program and not being aware of this.

The traditional tar file format cannot store a lot of the metadata that is contained in modern file systems (e.g. high resolution timestamps, access control lists, extended file attributes), so it is useless for such file systems.

Most modern "tar" implementations have added extensions to the tar file format, to make it usable with modern file systems, such as Linux XFS or Linux EXT4. But many of these extensions are incompatible between themselves, so certain tar files can be fully extracted only with the same tar program that has created them.

I strongly recommend against using the old tar or cpio file formats. Even with various extensions it is not guaranteed that they always work correctly.

I always use only the pax file format, which has also required extensions in order to work with the modern file systems, but the pax extensions are cleaner than those for tar, because the file format is better designed.

Libarchive, which was mentioned in TFA, is available in most Linux distributions or it can be built from source on any Linux computer. It provides an executable that is preferable to tar (better invoked as "bsdtar --format=pax") for the backup or transfer of any Linux files.

I have not checked recently GNU tar or other tar programs available on Linux, and I hope that meanwhile they have been upgraded to be able to archive losslessly the Linux file systems, but some years ago that was not true, so using tar or cpio on Linux could easily corrupt the archived files.

Re: Tar Files Created on macOS Display Errors When Extracting on Linux (2024)

#74

Post by adrian_b »

Terretta wrote: post #3
LatencyKills wrote: post #2

Ex-Apple engineer here. This is, for better or worse, just the way Apple approaches this type of problem. From Apple's perspective, this is the way to preserve Finder / Gatekeeper / metadata semantics. It avoids silent data loss when round-tripping archives between Macs. This behavior also maintains consistency with copyfile(3) (as well as the Archive Utility behavior). Apple treats tar less like “portable Unix inter…

Arguably, principle of least surprise is very Apple. If I point "tape archive" at a file system, I want that file system archived to tape. And so, tar does. If I don't, well, that's a fine option, and there's a fine option for that. So it's less of a "workaround" or something that "gets worse", than, "No, I don't really want a tape archive of this filesystem, only of some of it." And that's supported. That said, neve…

> I want that file system archived to tape. And so, tar does.

The traditional UNIX tar and cpio utilities cannot archive the modern Linux file systems without loss of metadata.

Most modern tar programs implement various file format extensions as a workaround for this, but the extensions may be incompatible between distinct tar programs and frequently they are very poorly documented.

Some years in the past, libarchive was the only archiver available on Linux that guaranteed lossless backups for the Linux file systems, e.g. xfs or ext4 (and also lossless file transfers between Linux file systems and FreeBSD file systems). Therefore that is what I have been using on Linux since then.

Presumably since then GNU tar and other tar programs should have caught up with it, but I have not verified this.

Whichever tar program was used in TFA, it was an obsolete tar program, and that was the real problem, not that the archives had been created on an Apple computer.

Re: Tar Files Created on macOS Display Errors When Extracting on Linux (2024)

#75

Post by adrian_b »

JoshTriplett wrote: post #7
Terretta wrote: post #3

Earlier quoted context omitted.

Arguably, principle of least surprise is very Apple. If I point "tape archive" at a file system, I want that file system archived to tape. And so, tar does. If I don't, well, that's a fine option, and there's a fine option for that. So it's less of a "workaround" or something that "gets worse", than, "No, I don't really want a tape archive of this filesystem, only of some of it." And that's supported. That said, neve…

> Arguably, principle of least surprise is very Apple. Principle of least surprise is good engineering practice. The question is always whose surprise. Someone who expects tar to behave like other UNIX systems is going to be surprised by this. Someone who expects tar on Apple to have perfect fidelity would be surprised by not-this. I increasingly feel like build systems should never be relying on any "native" utiliti…

What should be really surprising for the users of UNIX-like operating systems is when they lose data because traditional UNIX utilities like cp, tar or cpio do not make complete copies of files, as one would expect from their description.

What is worse is that these utilities do not give any warnings when they do not make complete copies. For cp, the root cause is that it has bad default options, while for tar and cpio the standard file formats cannot store the metadata of modern file systems.

The various tar programs have their own different file format extensions to deal with modern file systems, which are guaranteed to work only when using the same tar program for both creation and extraction. The better tar programs implement both their own file format extensions and the file format extensions used by other popular tar programs.

The author of the TFA has used some obsolete tar program, which is the cause for the surprising behavior that was seen.

To avoid loss of data on Linux, I always use the PAX file format instead of tar or cpio, with the extensions implemented by "bsdtar --create --format=pax" from libarchive, and I always alias cp to '/bin/cp --no-dereference --recursive --one-file-system --preserve=all --strip-trailing-slashes --verbose --interactive', where cp has been built with extended attributes support.

Re: Tar Files Created on macOS Display Errors When Extracting on Linux (2024)

#76

Post by mg794613 »

Oh Arul John, just because you don't understand, means it's a error.

What horrible advice also to download different tar versions, for something that should just be explained properly.

If it weren't for the "2024" in the title, I would have thought this to be a result from AI.

But it's not artificial intelligence. It's real stupidity.

Re: Tar Files Created on macOS Display Errors When Extracting on Linux (2024)

#78

Post by nottorp »

angry_octet wrote: post #9

We might also ask, why doesn't Linux also track such meta-data? Are Linux users not also subject to drive-by downloads impersonating valid files? Should we be one chmod a+x away from compromise?

> Are Linux users not also subject to drive-by downloads impersonating valid files?

Linux users generally install software with apt or rpm. Or steam.

The existence of any executable file outside the system dirs it a red flag in itself.

Re: Tar Files Created on macOS Display Errors When Extracting on Linux (2024)

#79

Post by dwattttt »

fingerlocks wrote: post #70
dwattttt wrote: post #66

Earlier quoted context omitted.

They could control their own Apple TVs to allow that dialogue to be dismissed via the TV controls.

Agreed, but why not just finishing setting it up? Or do people own Apple TVs without iPhones? That never occurred to me since a large part of the value prop is phone integration

Yes, I believe it's possible to buy an Apple TV without owning an iPhone.

Re: Tar Files Created on macOS Display Errors When Extracting on Linux (2024)

#80

Post by noisem4ker »

gerdesj wrote: post #15
ryandrake wrote: post #14

Earlier quoted context omitted.

.DS_Store, .fseventsd, .Spotlight-V100, .Trashes, and ._this and ._that These can all die in a fire too, as far as I am concerned. macOS loves to treat the user's filesystem as its own personal garbage dump.

thumbs.db and those weird MS alternative stream files for recording origination. filesystem attributes are for decorating files with meaning. Anything else that attempts to use filesystems in "interesting" ways is silly. Apple and MS really ought to consider why they do this sort of fragile, idiosyncratic nonsense.

> Thumbs.db

Windows has been storing thumbnail cache in the user profile folder since Vista (2006).

It's been 20 years. Time to let it go.

Post reply on HN