Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hackers use ZIP file concatenation to evade detection

bleepingcomputer.com

Post reply on HN
Print view View original

71–80 of 111 posts

Re: Hackers use ZIP file concatenation to evade detection

#71

Post by shagie »

meehai wrote: post #48
TeMPOraL wrote: post #47

Earlier quoted context omitted.

Well, 1) is zip with compression into single file, 2) is zip without compression into multiple files. You can also combine the two. And in all cases, you need a container format. The tasks are related enough that I don't really see the problem here.

I meant that they should be separate tools that can be piped together. For example: you have 1 directory of many files (1Gb in total) `zip out.zip dir/` This results in a single out.zip file that is, let's say 500Mb (1:2 compression) If you want to shard it, you have a separate tool, let's call it `shard` that works on any type of byte streams: `shard -I out.zip -O out_shards/ --shard_size 100Mb` This results in `out…

In unix, that is split https://en.wikipedia.org/wiki/Split_(Unix) (and its companion cat).

The problem is that on DOS (and Windows), it didn't have the unix philosophy of a tool that did one thing well and you couldn't depend on the necessary small tools being available. Thus, each compression tool also included its own file spanning system.

https://en.wikipedia.org/wiki/File_spanning

Re: Hackers use ZIP file concatenation to evade detection

#72

Post by notimetorelax »

MereInterest wrote: post #62
thrdbndndn wrote: post #58

Earlier quoted context omitted.

> or split the file Wait, I'm confused. Isn't this what OP was talking about?

Splitting the file doesn’t need to be part of the file format itself. I could split a file into N parts, then concatenate the parts together at a later time, regardless of what is actually in the file. The OP was saying that zip files can specify their own special type of splitting, done within the format itself, rather than operating on the raw bytes of a saved file.

Why would you use manual tools to achieve what ZIP archive can give you out of the box? E.g. if you do this manually you’d need to worry about file checksum to ensure you put it together correctly.

Re: Hackers use ZIP file concatenation to evade detection

#73

Post by anthk »

thrdbndndn wrote: post #33
kevin_thibedeau wrote: post #23

Earlier quoted context omitted.

The central directory allows zip archives to be split across multiple files on separate media without needing to read them all in for selective extraction. Not particularly useful today but invaluable in the sneakernet era with floppies.

Still useful today. Try to transmit a 100G file through any service is usually a pain especially if one end has non-stable Internet.

nncp, bittorrent...

Re: Hackers use ZIP file concatenation to evade detection

#74

Post by leni536 »

nh2 wrote: post #54
leni536 wrote: post #53

Earlier quoted context omitted.

This contradicts the specification, which explicitly supports stream-processing zip files, which necessarily can't happen if your source of truth is the central directory record. Unless if you can wrap your stream processing in some kind of transaction that you can drop once you discover foul play.

Source what you're referring to / explanation? In ZIP, later info wins. I don't see how that isn't always streamable.

Hmm, it appears that you are right. I vaguely remembered that zip was streamable, but it appears that it only means that it's stream writable, as in you can write zip file contents from a datastream of unknown size, and append the checksum and file size later in the zip file stream.

However such a zip file is definitely not stream readable, as the local file header no longer contains the size of the following file data, so you can't know where to end a given file. So for reading you definitely have to locate the central directory record.

In my defense the spec says[1].

> 4.3.2 Each file placed into a ZIP file MUST be preceded by a "local file header" record for that file. Each "local file header" MUST be accompanied by a corresponding "central directory header" record within the central directory section of the ZIP file.

Then in 4.3.6 it describes the file format, which seems to be fundamentally incompatible to altering zip files by appending data, as the resulting file would not conform to this format.

So basically some implementations (maybe opportunistically, relying on compressed sizes being available in the local file headers) stream read from the zip file, assuming that it's a valid zip file, but not validating. Some other implementations only use the central directory record at the end, but don't validate the file format either.

A validating zip file parser should be possible, by locating the central directory record and checking that the referred files with their metadata fully cover the file contents of the zip files, without gaps and overlaps. But this probably won't win any benchmarks.

[1] https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT

Re: Hackers use ZIP file concatenation to evade detection

#75

Post by olddustytrail »

Retr0id wrote: post #39
acka wrote: post #36

Earlier quoted context omitted.

DAR (Disk ARchiver)[1] looks to be a good alternative. It supports random access, encryption, and individual file compression within the archive. [1] http://dar.linux.free.fr/

That seems counter to GP's suggestion of doing compression at a separate layer

Not really. There's no "dar compression" format. It calls different compression tools just like tar.

Re: Hackers use ZIP file concatenation to evade detection

#76

Post by immibis »

poincaredisk wrote: post #65
wahern wrote: post #10

From a security perspective, and as a programmer, I've never liked ZIP files precisely because there are two mechanisms to identify the contents, the per-file header and the central directory. When you're defining a format, protocol, or w'ever, ideally there should be a single source of truth, a single valid & useable parse, etc; basically, the structure of the data or process should be intrinsically constraining. Th…

>there are two mechanisms to identify the contents, the per-file header and the central directory There is only one right, standard mandated, way to identify the contents (central directory). For one or another reason many implementations ignore it, but I don't think it's fair to say that the zip format in ambiguous.

Sometimes you want to read the file front-to-back in a streaming fashion.

Re: Hackers use ZIP file concatenation to evade detection

#77

Post by unoti »

canucker2016 wrote: post #26
wahern wrote: post #10

From a security perspective, and as a programmer, I've never liked ZIP files precisely because there are two mechanisms to identify the contents, the per-file header and the central directory. When you're defining a format, protocol, or w'ever, ideally there should be a single source of truth, a single valid & useable parse, etc; basically, the structure of the data or process should be intrinsically constraining. Th…

I don't think you understand the reason for the ZIP archive file design. Back in the late 1980s, backup media for consumers was limited to mostly floppy disks, some users had tape/another hard disk. Say you had a variable number of files to compress and write out to a ZIP archive. IF you write out the central directory first, followed by all the individually possibly compressed and/or encrypted files, you'd have to c…

> coders discovered that scanning the ZIP local entries to be a faster way to build up the ZIP archive entries, otherwise you're forced to seek all the way to the end of a ZIP archive and work backwards to locate the central directory

Would this have worked? Reserve a few bytes at the beginning of the archive at a fixed location offset from the start, and say "this is where we will write the offset to where the central directory will start." Then build the whole archive, writing the central directory at the end. Then seek back to that known offset at the start of the file and write the offset to the central directory. When creating the archive, we can write the central directory to a temp file, and then append that in to the end of the file we're building at the end, and fix up the offset.

Seems like this strategy would enable us to both have a number of files in the archive that are known at the beginning, and also allow us to do a high-speed seek to the central directory when reading the archive.

I imagine people thought about this idea and didn't do it for one reason or another. I can imagine why we didn't do that for Unix TAR-- most tape devices are a one-way write stream and don't have random access. But ZIP was designed for disks; I'm curious why this idea wouldn't have solved both problems.

Re: Hackers use ZIP file concatenation to evade detection

#78

Post by hombre_fatal »

MereInterest wrote: post #62
thrdbndndn wrote: post #58

Earlier quoted context omitted.

> or split the file Wait, I'm confused. Isn't this what OP was talking about?

Splitting the file doesn’t need to be part of the file format itself. I could split a file into N parts, then concatenate the parts together at a later time, regardless of what is actually in the file. The OP was saying that zip files can specify their own special type of splitting, done within the format itself, rather than operating on the raw bytes of a saved file.

Without knowing the specifics of what's being talked about, I guess it makes sense that zip did that because the OS doesn't make it easy for the average user to concatenate files, and it would be hard to concatenate 10+ files in the right order. If you have to use a cli then it's not really a solution for most people, nor is it something I want to have to do anyways.

The OS level solution might be a naming convention like "{filename}.{ext}.{n}" like "videos.zip.1" where you right-click it and choose "concatenate {n} files" and turns them into "{filename}.{ext}".

Re: Hackers use ZIP file concatenation to evade detection

#79

Post by jonjonsonjr »

edflsafoiewq wrote: post #68
murderfs wrote: post #64

Earlier quoted context omitted.

Why do you believe that archiving and compressing belong in the same layer more than sharding does? The unixy tool isn't zip, it's tar | gzip.

tar|gzip does not allow random access to files. You have to decompress the whole tarball (up to the file you want).

Even worse, in the general case, you should really decompress the whole tarball up to the end because the traditional mechanism for efficiently overwriting a file in a tarball is to append another copy of it to the end. (This is similar to why you should only trust the central directory for zip files.)
Post reply on HN