Don't Use RAR • Hacker News Board

Re: Don't Use RAR

#201

Post by vetinari » Fri, Feb 25, 2022, 7:15 PM UTC

ClumsyPilot wrote: post #197

vetinari wrote: post #64

Earlier quoted context omitted.

And what's the difference compared to forcing docx/xlsx or psd? The network effects affect every exchange, not just archives.

If a government forces it's people to use particular software, it has a duty to provide that software to the entire population. If a parliament decides to pay $X billion to Microsoft, for a lisence for all of UK, well, that sucks, but they can fight over it like they do with all other random spending they do. If it's not willing to pay the cost, it is basically creating a monopoly for that firm, and I have to pay for…

I see where you come from, and in principle agree with you, but it is not how the real world works.

For an extreme example, see South Korea and their ActiveX problem.

Re: Don't Use RAR

#202

Post by ASalazarMX » Fri, Feb 25, 2022, 7:25 PM UTC

btdmaster wrote: post #195

ronjouch wrote: post #153

Earlier quoted context omitted.

Confirming this, and confirming that the recovery record (that I had set to a hefty 10%, I think, because the data was important to me) did let me recover archives that suffered physical media damage, which would have been irrecoverable had I used something else. I wish 7zip had this feature. This was one of the most satisfying progress bars: from "your data is dead" to a few minutes later "Successfully recovered ful…

Par2 is always an option, and RAR implements Reed-Solomon error correcting codes and there are many recent implementations of those: https://github.com/klauspost/reedsolomon .

> Reed-Solomon error correcting codes

Had to double check that. I would have sworn those codes were called Solomon-Reed, not Reed-Solomon. Guess I have my little Mandela Effect now!

Re: Don't Use RAR

#203

Post by kazinator » Fri, Feb 25, 2022, 7:27 PM UTC

bo1024 wrote: post #185

kazinator wrote: post #140

Earlier quoted context omitted.

Yes. The GPL is not a license which dictates how you can use software, such as that you must not read it, or use ideas from it. EULA restrictions like "no reverse engineering" are not rooted in copyright law, and borrowing ideas is in the realm of patents. EULA's try to leverage copyright law in order to impose non-copyright restrictions, using the concept that if you violate any of the arbitrary non-copyright restri…

I always found this aspect of 'copyright' very confusing. I still don't understand the legal basis here.

I don't fully understand it myself, because if you have a copy of the work (e.g. purchased copy of some proprietary software) the entity which distributed that to you was the copyright-holding purveyor. They did the copying, using their reserved right to do so.

No matter what yo udo with the copyrighted work, if you're not copying it, you're not infringing on copyright. The idea that your right to have a copy can lapse due to violating the EULA has holes in it, because copyright isn't about the right to have a copy, but about the right to produce and redistribute copies.

For instance, on a bit of a tangent here, if you steal a book out of someone's backpack, that is not copyright infringement, and cannot be. You didn't copy anything.

The proprietary EULA wants the law to believe that you're stealing if you continue to use the software after violating the agreement. But without connecting that to the concept of copyright infringement, the accusation has no basis, since you're just using what you paid for it.

Disassembling a binary executable to understand it is exactly the same as reading a book to understand it. Copyright is made for books and such.

Re: Don't Use RAR

#204

Post by Sohcahtoa82 » Fri, Feb 25, 2022, 7:43 PM UTC

ryandrake wrote: post #199

dole wrote: post #178

Earlier quoted context omitted.

RAR's initial release was 1993, 7z's 1999. IIRC, I remembered seeing RAR's in the scene around 1995, 1996 or so. My take: RAR handled splitting to multiple arbitrary-sized volumes more gracefully than ARJ/ARC which were CLI first and never had a Windows GUI that was as nicely polished as WinRAR was. SFVs made error checking and redownloading corrupted volumes easy without relying on the compression format to handle i…

> I remembered seeing RAR's in the scene around 1995, 1996 or so. My take: RAR handled splitting to multiple arbitrary-sized volumes more gracefully than ARJ/ARC which were CLI first and never had a Windows GUI that was as nicely polished as WinRAR was. Of course there were still people who screwed that up. I remember downloading some software once that came in the form of a multi-volume RAR, maybe 20 floppy disk-siz…

What really bothers me now are movies being distributed over torrents as a multi-volume RAR.

Just...why?

The BitTorrent protocol will handle corrupt data and fix it at the segment level. You won't have to redownload an entire file.

Splitting a file into pieces made sense 20+ years ago when connections (Both your physical ISP connection and the logical TCP connection) were unstable, web servers didn't always support download resuming, and software didn't handle graceful unexpected disconnections, but those days are long behind us. We transfer data over encrypted channels that include checksums at the packet level. We use software that can handle disconnections, automatically reconnect, and resume where it left off, not to mention detect when data went bad and re-download just the bad part.

There's just no damn reason to split a 5 GB .mkv into 100 50 MB .rar files which will then take my poor RPi 15 minutes to decompress.

Re: Don't Use RAR

#205

Post by vetinari » Fri, Feb 25, 2022, 7:47 PM UTC

Hello71 wrote: post #173

vetinari wrote: post #102

Earlier quoted context omitted.

Yes, I mean command line; it is a long time that I used the gui, though I don't think it is updated often. With command line, I arrived eventually to a set of switches to use, but the -mt option seems to allocate core per file. That means, that if the filesizes of the archived files are not somewhat distributed, it won't work very well. The extreme is, if you have one huge file and few small ones, the small ones will…

> the -mt option seems to allocate core per file this can't be true in solid archive mode, which should almost always be used (otherwise you might as well use zip). also: $ tee {1..10} 11 $ time 7z a x.7z {1..11}; rm x.7z 7-Zip (z) 21.07 (x64) : Copyright (c) 1999-2021 Igor Pavlov : 2021-12-26 64-bit locale=en_US.UTF-8 Threads:12, ASM Scanning the drive: 11 files, 78888917 bytes (76 MiB) Creating archive: x.7z Add ne…

Seems that I will have to check what parameters to really use:

    $ time 7z a x.7z ubuntu-20.04.4-desktop-amd64.iso 
    
    7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
    p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,24 CPUs AMD Ryzen Threadripper 2920X 12-Core Processor  (800F82),ASM,AES-NI)
    
    Scanning the drive:
    1 file, 3379068928 bytes (3223 MiB)
    
    Creating archive: x.7z
    
    Items to compress: 1
    
                                         
    Files read from disk: 1
    Archive size: 3277419065 bytes (3126 MiB)
    Everything is Ok
    
    real 1m23,295s
    user 25m24,447s
    sys 0m27,991s

is quite a difference compared to:

    $ time 7z a -t7z -m0=lzma x.7z ubuntu-20.04.4-desktop-amd64.iso 
    
    7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
    p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,24 CPUs AMD Ryzen Threadripper 2920X 12-Core Processor  (800F82),ASM,AES-NI)
    
    Scanning the drive:
    1 file, 3379068928 bytes (3223 MiB)
    
    Creating archive: x.7z
    
    Items to compress: 1
    
                                         
    Files read from disk: 1
    Archive size: 3305891811 bytes (3153 MiB)
    Everything is Ok
    
    real 7m5,647s
    user 13m9,589s
    sys 0m6,757s

Re: Don't Use RAR

#206

Post by ryandrake » Fri, Feb 25, 2022, 8:43 PM UTC

Sohcahtoa82 wrote: post #204

ryandrake wrote: post #199

Earlier quoted context omitted.

> I remembered seeing RAR's in the scene around 1995, 1996 or so. My take: RAR handled splitting to multiple arbitrary-sized volumes more gracefully than ARJ/ARC which were CLI first and never had a Windows GUI that was as nicely polished as WinRAR was. Of course there were still people who screwed that up. I remember downloading some software once that came in the form of a multi-volume RAR, maybe 20 floppy disk-siz…

What really bothers me now are movies being distributed over torrents as a multi-volume RAR. Just...why? The BitTorrent protocol will handle corrupt data and fix it at the segment level. You won't have to redownload an entire file. Splitting a file into pieces made sense 20+ years ago when connections (Both your physical ISP connection and the logical TCP connection) were unstable, web servers didn't always support d…

I think what we're seeing maybe can be chalked up to distributors adhering to their group's internal rules that just haven't changed since the 90s. Another likely "Scene rule" thing that lasted a long while were movies that were encoded aiming for a particular size. The quality was tuned up down such that every release was exactly 700MB. Why 700MB? Well, turns out there is an ancient form of data storage called CD-ROM which is limited to 700MB, and the rule (or habit) to aim for that size just never changed.

Re: Don't Use RAR

#207

Post by kiririn » Fri, Feb 25, 2022, 11:06 PM UTC

noAnswer wrote: post #183

jillesvangurp wrote: post #157

Earlier quoted context omitted.

I think for the common use case of distributing giga bytes of data in a reasonably efficient way, 7zip is fine (e.g. simheaven.com uses that for huge scenery downloads). Where things get tricky is splitting the resulting archive in multiple files. This used to be common with rar and I think 7zip also supports this. That is convenient when downloading over a slow/unstable connection (or if you don't know how to resume…

The reason I'm a RAR fan is because TAR failed me multiple times! (The first one was ~21 years ago. I lost most of my home.tar.gz-Backup. Because tar has no index, if something bad comes along, everything after that can't be read.) 17 years ago, I took a deep dive into rar and replaced my tar.gz backup process with it. I had a cronjob that checked the tar.gz-Backups from the past week and there was a 50% change of "U…

It’s gzip that caused your issue, not tar. The lack of index in tar is actually a strength for surviving corruption, as there are no critical areas, ie you can lose any parts of a tar archive and recover useful data from the rest

Tar.gz and tar.xz should never have been made a thing or become so popular, the only safe options are tar, tar.lz and to an extent tar.bz2

Re: Don't Use RAR

#208

Post by pabs3 » Fri, Feb 25, 2022, 11:27 PM UTC

cesarb wrote: post #91

pabs3 wrote: post #80

There is open source support for RAR in unar: https://theunarchiver.com/command-line

I've encountered RAR files unar cannot open. It seems there are several versions of the RAR format, and unar only understands the older ones.

Hmm, I thought unar supported the latest versions, while the open source version of unrar was the one that didn't support the latest versions.

Re: Don't Use RAR

#209

Post by darkhorn » Fri, Feb 25, 2022, 11:38 PM UTC

rvz wrote: post #18

darkhorn wrote: post #7

Quoted post unavailable.

and? Everyone is using nginx and that was made by a Russian, does that mean we should stop using it? This is why 'guilt by association' doesn't work with your 'point' .

I forgot about Nginx. Thank you.

Re: Don't Use RAR

#210

Post by mmis1000 » Sat, Feb 26, 2022, 3:35 PM UTC

duskwuff wrote: post #200

mmis1000 wrote: post #191

Earlier quoted context omitted.

No, the major problem of tar is 'it is not indexable'. In any case, browse a tar means you need to scan the whole achieve to locate all files and generate the entry list. And of course it get slower when the files are a lot or archive is big. This is a format even worse than zip.(Zip do have a central record section) And it also suffer the same problem from zip: the charset is undefined. Because they are older then t…

Technically speaking, TAR is indexable -- the index is distributed through the archive, at the head of each file. So long as the TAR is uncompressed, you can skip from one header to the next by reading the size of each file. It ends up looking like a bunch of random I/O, but it's manageable. (The headers are even sector-aligned!) The problem arises when the TAR is compressed with a stream compressor (as most are). Si…

> Technically speaking, TAR is indexable -- the index is distributed through the archive, at the head of each file.

Actually, zip works in the same way as tar. So the central dict is locate at the end of file because it can only be decided after all works finished.

And zip actually struct the first part of file with file info before every block in layout similar to tar, so stream decoding is possible. (Although it IS a violation of specification to decode file based on per file info instead of central record in zip standard)

> On the flip side, this means that those archive formats can't archive the contents of a filesystem which contains inconsistently encoded (or flat-out nonsense) filenames.

It is a good thing IMO because you can't contribute to the problem more now (at the price not able to backup already screwed up disks. Well..., at least old format(tar) still works on old disk)