Magika: AI powered fast and efficient file type identification

Post by eapriv » Sat, Feb 17, 2024, 7:03 AM UTC

michaelt wrote: post #97

m0shen wrote: post #33

As someone that has worked in a space that has to deal with uploaded files for the last few years, and someone who maintains a WASM libmagic Node package ( https://github.com/moshen/wasmagic ) , I have to say I really love seeing new entries into the file type detection space. Though I have to say when looking at the Node module, I don't understand why they released it. Their docs say it's slow: https://github.com/go…

> The model appears to only detect 116 file types [...] Where libmagic detects... a lot. Over 1600 last time I checked As I'm sure you know, in a lot of applications, you're preparing things for a downstream process which supports far fewer than 1600 file types. For example, a printer driver might call on file to check if an input is postscript or PDF, to choose the appropriate converter - and for any other format, j…

Post by delijati » Sat, Feb 17, 2024, 10:16 AM UTC

Post by jdiff » Sat, Feb 17, 2024, 2:06 PM UTC

EnigmaFlare wrote: post #229

epcoa wrote: post #227

Earlier quoted context omitted.

> or how "soft" they were. From the comment: It identified some simple HTML files (html, head, title, body, p tags and not much else ) as "MS Visual Basic source (VBA)", "ASP source (code)", and "Generic text document" where the `file` utility correctly identified all such examples as "HTML document text". That's pretty soft. Nothing "adversarial" claimed either. > Being strictly superior to a competent human is a pr…

Those are only soft to a human. I looked at a couple and I picked them correctly but I don't know what details the classifier was seeing which I was blind to. Not to say it was correct, just that we can't call them soft just because they're short and easy for a human. > The bar is the file utility. It has higher accuracy than that. You would reject it just because the failures are different even though they're less?

Post by dieortin » Sat, Feb 17, 2024, 4:01 PM UTC

westurner wrote: post #181

ebursztein wrote: post #139

Earlier quoted context omitted.

Thanks for the feedback -- we will look into it. If you can share with us the list of URL that would be very helpful so we can reproduce - send us an email at magika-dev@google.com if that is possible. For crawling we have planned a head only model to avoid fetching the whole file but it is not ready yet -- we weren't sure what use-cases would emerge so that is good to know that such model might be useful. We mostly…

What is the MIME type of a .tar file; and what are the MIME types of the constituent concatenated files within an archive format like e.g. tar? hachoir/subfile/main.py: https://github.com/vstinner/hachoir/blob/main/hachoir/subfil... File signature: https://en.wikipedia.org/wiki/File_signature PhotoRec: https://en.wikipedia.org/wiki/PhotoRec "File Format Gallery for Kaitai Struct"; 185+ binary file format specificatio…

Post by _a_a_a_ » Sat, Feb 17, 2024, 6:10 PM UTC

timschmidt wrote: post #238

_a_a_a_ wrote: post #232

Earlier quoted context omitted.

I share your distaste for people whose only contribution is subtraction but suggest you lay off the sarcasm though. Trolls; don't feed. (Well done on your project BTW)

I don't see any sarcasm from me in the thread. I had serious questions. Perhaps you could point out what you see? Thanks for the supportive words about the project.