Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Magika: AI powered fast and efficient file type identification

opensource.googleblog.com

Post reply on HN
Print view View original

161–170 of 262 posts

Re: Magika: AI powered fast and efficient file type identification

#161

Post by chmod775 »

ebursztein wrote: post #65
stevepike wrote: post #10

Oh man, this brings me back! Almost 10 years ago I was working on a rails app trying to detect the file type of uploaded spreadsheets (xlsx files were being detected as application/zip, which is technically true but useless). I found "magic" that could detect these and submitted a patch at https://bugs.freedesktop.org/show_bug.cgi?id=78797 . My patch got rejected for needing to look at the first 3KB bytes of the file…

Co-author of Magika here (Elie) so we didn't include the measurements in the blog post to avoid making it too long but we did those measurements. Overall file takes about 6ms (single file) 2.26ms per files when scanning multiples. Magika is at 65ms single file and 5.3ms when scanning multiples. So Magika is for the worst case scenario about 10x slower due to the time it takes to load the model and 2x slower on repeat…

Is that single-threaded libmagic vs Magika using every core on the system? What are the numbers like if you run multiple libmagic instances in parallel for multiple files, or limit both libmagic and magika to a single core?

Testing it on my own system, magika seems to use a lot more CPU-time:

    file /usr/lib/*  0,34s user 0,54s system 43% cpu 2,010 total
    ./file-parallel.sh  0,85s user 1,91s system 580% cpu 0,477 total
    bin/magika /usr/lib/*  92,73s user 1,11s system 393% cpu 23,869 total
Looks about 50x slower to me. There's 5k files in my lib folder. It's definitely still impressively fast given how the identification is done, but the difference is far from negligible.

Re: Magika: AI powered fast and efficient file type identification

#162

Post by jdiff »

michaelmior wrote: post #160
TomNomNom wrote: post #147

Earlier quoted context omitted.

It provided the wrong file-types for some files, so I cannot rely on its output to be correct. If you wanted to, for example, use this tool to route different files to different format-specific handlers it would sometimes send files to the wrong handlers.

Except a 100% correct implementation doesn't exist AFAIK. So if I want to do anything that makes a decision based on the type of a file, I have to pick some algorithm to do that. If I can do that correctly 99% of the time, that's better than not being able to make that decision at all, which is where I'm left if a perfect implementation doesn't exist.

Nobody's asking for perfection. But the AI is offering inexplicable and obvious nondeterministic mistakes that the traditional algorithms don't suffer from.

Magika goes wrong and your fonts become audio files and nobody knows why. Magic goes wrong and your ZIP-based documents get mistaken for generic ZIP files. If you work with that edge case a lot, you can anticipate it with traditional algorithms. You can't anticipate nondeterministic hallucination.

Re: Magika: AI powered fast and efficient file type identification

#163

Post by jeffbee »

brabel wrote: post #119
stevepike wrote: post #10

Oh man, this brings me back! Almost 10 years ago I was working on a rails app trying to detect the file type of uploaded spreadsheets (xlsx files were being detected as application/zip, which is technically true but useless). I found "magic" that could detect these and submitted a patch at https://bugs.freedesktop.org/show_bug.cgi?id=78797 . My patch got rejected for needing to look at the first 3KB bytes of the file…

> They had a hard limit that they wouldn't see past the first 256 bytes. Then they could never detect zip files with certainty, given that to do that you need to read up to 65KB (+ 22) at the END of the file. The reason is that the zip archive format allows "gargabe" bytes both in the beginning of the file and in between local file headers.... and it's actually not uncommon to prepend a program that self-extracts the…

That's why the whole question is ill formed. A file does not have exactly one type. It may be a valid input in various contexts. A zip archive may also very well be something else.

Re: Magika: AI powered fast and efficient file type identification

#164

Post by Eiim »

I ran a quick test on 100 semi-random files I had laying around. Of those, 81 were detected correctly, 6 were detected as the wrong file type, and 12 were detected with an unspecific file type (unknown binary/generic text) when a more specific type existed. In 4 of the unspecific cases, a low-confidence guess was provided, which was wrong in each case. However, almost all of the files which were detected wrong/unspecific are of types not supported by Magika, with one exception of a JSON file containing a lot of JS code as text, which was detected as JS code. For comparison, file 5.45 (the version I happened to have installed) got 83 correct, 6 wrong, and 10 not specific. It detected the weird JSON correctly, but also had its own strange issues, such as detecting a CSV as just "data". The "wrong" here was somewhat skewed by the 4 GLSL shader code files that were in the dataset for some reason, all of which it detected as C code (Magika called them unknown). The other two "wrong" detections were also code formats that it seems it doesn't support. It was also able to output a lot more information about the media files. Not sure what to make of these tests but perhaps they're useful to somebody.

Re: Magika: AI powered fast and efficient file type identification

#165

Post by nicklecompte »

NiloCK wrote: post #2

A somewhat surprising and genuinely useful application of the family of techniques. I wonder how susceptible it is to adversarial binaries or, hah, prompt-injected binaries.

Elsewhere in the thread kevincox[1] points out that it's extremely susceptible to adversarial binaries:

> Worse it seems that for unknown formats it confidently claims that it is one of the known formats. Rather than saying "unknown" or "binary data".

Seems like this is genuinely useless for anybody but AI researchers.

[1] https://news.ycombinator.com/item?id=39395677

Re: Magika: AI powered fast and efficient file type identification

#166

Post by nicklecompte »

Andugal wrote: post #114

I have a question: Is something like Magika enough to check if a file is malicious or not? Example: users can upload PNG file (and only PNG is accepted). If Malika detects that the file is a PNG, does this mean the file is clean?

This comment from kevincox[1] says the answer is a hard "no":

> Worse it seems that for unknown formats it confidently claims that it is one of the known formats. Rather than saying "unknown" or "binary data".

There are other comments in this thread that make me think Google contaminated their test data with training data and the 99% results should not be taken at face value. OTOH I am not particularly surprised that Magika would be better than the other tools at distinguishing semi-unstructured plain text e.g. Java source vs. C++ source or YAMLs versus INIs. But that's a very different use case than many security applications. The comments here suggest Magika is especially susceptible to binary obfuscation.

[1] https://news.ycombinator.com/item?id=39395677

Re: Magika: AI powered fast and efficient file type identification

#167

Post by Solvency »

ebursztein wrote: post #139
TomNomNom wrote: post #132

This looks cool. I ran this on some web crawl data I have locally, so: all files you'd find on regular websites; HTML, CSS, JavaScript, fonts etc. It identified some simple HTML files (html, head, title, body, p tags and not much else) as "MS Visual Basic source (VBA)", "ASP source (code)", and "Generic text document" where the `file` utility correctly identified all such examples as "HTML document text". Some woff a…

Thanks for the feedback -- we will look into it. If you can share with us the list of URL that would be very helpful so we can reproduce - send us an email at magika-dev@google.com if that is possible. For crawling we have planned a head only model to avoid fetching the whole file but it is not ready yet -- we weren't sure what use-cases would emerge so that is good to know that such model might be useful. We mostly…

[flagged]

Re: Magika: AI powered fast and efficient file type identification

#168

Post by [deleted] »

barrkel wrote: post #128
m0shen wrote: post #37

Earlier quoted context omitted.

Made a small test to try it out: https://gist.github.com/moshen/784ee4a38439f00b17855233617e9... hyperfine ./magika.bash ./file.bash Benchmark 1: ./magika.bash Time (mean ± σ): 706.2 ms ± 21.1 ms [User: 10520.3 ms, System: 1604.6 ms] Range (min … max): 684.0 ms … 738.9 ms 10 runs Benchmark 2: ./file.bash Time (mean ± σ): 23.6 ms ± 1.1 ms [User: 15.7 ms, System: 7.9 ms] Range (min … max): 22.4 ms … 29.0 ms 111 runs Su…

Realistically, either you're identifying one file interactively and you don't care about latency differences in the 10s of ms, or you're identifying in bulk (batch command line or online in response to requests), in which case you should measure the marginal cost and exclude Python startup and model loading times.

[deleted]

Re: Magika: AI powered fast and efficient file type identification

#169

Post by jsnell »

jdiff wrote: post #162
michaelmior wrote: post #160

Earlier quoted context omitted.

Except a 100% correct implementation doesn't exist AFAIK. So if I want to do anything that makes a decision based on the type of a file, I have to pick some algorithm to do that. If I can do that correctly 99% of the time, that's better than not being able to make that decision at all, which is where I'm left if a perfect implementation doesn't exist.

Nobody's asking for perfection. But the AI is offering inexplicable and obvious nondeterministic mistakes that the traditional algorithms don't suffer from. Magika goes wrong and your fonts become audio files and nobody knows why. Magic goes wrong and your ZIP-based documents get mistaken for generic ZIP files. If you work with that edge case a lot, you can anticipate it with traditional algorithms. You can't anticip…

Where are you getting the non-determinism part from? It would seem surprising for there to be anything non-deterministic about an ML model like this, and nothing in the original reports seems to suggest that either.

Re: Magika: AI powered fast and efficient file type identification

#170

Post by beeboobaa »

TomNomNom wrote: post #142
ebursztein wrote: post #139

Earlier quoted context omitted.

Thanks for the feedback -- we will look into it. If you can share with us the list of URL that would be very helpful so we can reproduce - send us an email at magika-dev@google.com if that is possible. For crawling we have planned a head only model to avoid fetching the whole file but it is not ready yet -- we weren't sure what use-cases would emerge so that is good to know that such model might be useful. We mostly…

Sure thing :) Here's[0] a .tgz file with 3 files in it that are misidentified by magika but correctly identified by the `file` utility: asp.html, vba.html, unknown.woff These are files that were in one of my crawl datasets. [0]: https://poc.lol/files/magika-test.tgz

Do you have permission to redistribute these files?
Post reply on HN