81–90 of 111 posts
Earlier quoted context omitted.
That seems counter to GP's suggestion of doing compression at a separate layer
Not really. There's no "dar compression" format. It calls different compression tools just like tar.
Earlier quoted context omitted.
I meant that they should be separate tools that can be piped together. For example: you have 1 directory of many files (1Gb in total) `zip out.zip dir/` This results in a single out.zip file that is, let's say 500Mb (1:2 compression) If you want to shard it, you have a separate tool, let's call it `shard` that works on any type of byte streams: `shard -I out.zip -O out_shards/ --shard_size 100Mb` This results in `out…
The key thing that you get by integrating the two tools is the ability to more easily extract a single file from a multipart archive— Instead of having to reconstruct the entire file, you can look in the part/diskette with the index to find out which other part/diskette you need to use to get at the file you want.
AFAIK you can create a ZIP archive saved to floppy disks even if your source hard disk has low/almost no free space.
Phil Katz (creator of the ZIP file format) had a different set of design constraints.
Remember, secure encryption, good compression, and truely random data are indistinguishable. It's best to paste that encrypted payload into a JPG with some bullshit magic headers and upload that to a trusted Exfil pivot instead. Or, to get SuperMarioKart.rom to work with your chromeApp-XEMU emulator to play during down-time at work, just rename it to SMB.png, and email it you yourself.
Sure the various measures of entropy should be high, but I always just assumed that compression wouldn't pass almost any cryptographic randomness test.
Why do scanners need to look inside compressed archives at all? If the malicious file is extracted, it can be detected then. If it’s not extracted and instead used directly from the archive, then the offending code that executes malicious payloads from inside archives should be detectable? Is that the role of AutoIt in this scenario?
Common situation is detecting malware sent through a phishing mail. You want to intercept those before a user can unpack the file.
Earlier quoted context omitted.
I don't think you understand the reason for the ZIP archive file design. Back in the late 1980s, backup media for consumers was limited to mostly floppy disks, some users had tape/another hard disk. Say you had a variable number of files to compress and write out to a ZIP archive. IF you write out the central directory first, followed by all the individually possibly compressed and/or encrypted files, you'd have to c…
> coders discovered that scanning the ZIP local entries to be a faster way to build up the ZIP archive entries, otherwise you're forced to seek all the way to the end of a ZIP archive and work backwards to locate the central directory Would this have worked? Reserve a few bytes at the beginning of the archive at a fixed location offset from the start, and say "this is where we will write the offset to where the centr…
The central directory at the end of the archive fits that constraint. Any design where a placeholder needs to be updated later won't.
Why do scanners need to look inside compressed archives at all? If the malicious file is extracted, it can be detected then. If it’s not extracted and instead used directly from the archive, then the offending code that executes malicious payloads from inside archives should be detectable? Is that the role of AutoIt in this scenario?
Because they want to check on the wire, before it hits an endpoint. Common situation is detecting malware sent through a phishing mail. You want to intercept those before a user can unpack the file.
see https://www.fortinet.com/resources/cyberglossary/defense-in-...
Earlier quoted context omitted.
That's a very bad way of solving that issue. If transmission is a problem, either use a proper retry-friendly protocol (such as bittorrent) or split the file. Using hacks on the data format just leads to additional pain
couldn't agree more! We need to separate and design modules as unitary as possible: - zip should ARCHIVE/COMPRESS, i.e. reduce the file size and create a single file from the file system point of view. The complexity should go in the compression algorithm. - Sharding/sending multiple coherent pieces of the same file (zip or not) is a different module and should be handled by specialized and agnostic protocols that do…
Also, I enjoyed your Freudian slip:
single respectability tools
->
single responsibility tools
Last I checked virustotal doesn't test nested archives for viruses even though it's an issue old as modern computing
I have no idea why those virus scanners don't check nested archives. Maybe time/cpu constraints?