Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Magika: AI powered fast and efficient file type identification

opensource.googleblog.com

Post reply on HN
Print view View original

131–140 of 262 posts

Re: Magika: AI powered fast and efficient file type identification

#131

Post by theon144 »

michaelt wrote: post #97
m0shen wrote: post #33

As someone that has worked in a space that has to deal with uploaded files for the last few years, and someone who maintains a WASM libmagic Node package ( https://github.com/moshen/wasmagic ) , I have to say I really love seeing new entries into the file type detection space. Though I have to say when looking at the Node module, I don't understand why they released it. Their docs say it's slow: https://github.com/go…

> The model appears to only detect 116 file types [...] Where libmagic detects... a lot. Over 1600 last time I checked As I'm sure you know, in a lot of applications, you're preparing things for a downstream process which supports far fewer than 1600 file types. For example, a printer driver might call on file to check if an input is postscript or PDF, to choose the appropriate converter - and for any other format, j…

Okay, but your one file type is more likely to be included in the 1600 that libmagic supports rather than Magika's 116?

For that matter, the file types I care about are unfortunately misdetected by Magika (which is also an important point - the `file` command at least gives up and says "data" when it doesn't know, whereas the Magika demo gives a confidently wrong answer).

I don't want to criticize the release because it's not meant to be a production-ready piece of software, and I'm sure the current 116 types isn't a hard limit, but I do understand the parent comment's contention.

Re: Magika: AI powered fast and efficient file type identification

#132

Post by TomNomNom »

This looks cool. I ran this on some web crawl data I have locally, so: all files you'd find on regular websites; HTML, CSS, JavaScript, fonts etc.

It identified some simple HTML files (html, head, title, body, p tags and not much else) as "MS Visual Basic source (VBA)", "ASP source (code)", and "Generic text document" where the `file` utility correctly identified all such examples as "HTML document text".

Some woff and woff2 files it identified as "TrueType Font Data", others are "Unknown binary data (unknown)" with low confidence guesses ranging from FLAC audio to ISO 9660. Again, the `file` utility correctly identifies these files as "Web Open Font Format".

I like the idea, but the current implementation can't be relied on IMO; especially not for automation.

A minor pet peeve also: it doesn't seem to detect when its output is a pipe and strip the shell colour escapes resulting in `^[[1;37` and `^[[0;39m` wrapping every line if you pipe the output into a vim buffer or similar.

Re: Magika: AI powered fast and efficient file type identification

#133

Post by alkonaut »

lopkeny12ko wrote: post #51

I don't understand why this needs to exist. Isn't file type detection inherently deterministic by nature? A valid tar archive will always have the same first few magic bytes. An ELF binary has a universal ELF magic and header. If the magic is bad, then the file is corrupted and not a valid XYZ file. What's the value in throwing in "heuristics" and probabilistic inference into a process that is black and white by desi…

But even with determinism, it's not always right. It's not too rare to find a text file with a byte order mark indicating UTF16 (0xFE 0xFF) but then actually containing utf-8. But what "format" does it have then? Is it UTF-8 or UTF-16? Same with e.g. a jar file missing a manifest. That's just a zip, even though I'm sure some runtime might eat it.

But the question is when you have the issue of having to guess the format of a file? Is it when reverse engineering? Last time I did something like this was in the 90's when trying to pick apart some texture from a directory of files called asset0001.k and it turns out it was a bitmap or whatever. Fun times.

Re: Magika: AI powered fast and efficient file type identification

#134

Post by michaelmior »

TomNomNom wrote: post #132

This looks cool. I ran this on some web crawl data I have locally, so: all files you'd find on regular websites; HTML, CSS, JavaScript, fonts etc. It identified some simple HTML files (html, head, title, body, p tags and not much else) as "MS Visual Basic source (VBA)", "ASP source (code)", and "Generic text document" where the `file` utility correctly identified all such examples as "HTML document text". Some woff a…

> the current implementation can't be relied on IMO

What's your reasoning for not relying on this? (It seems to me that this would be application-dependent at the very least.)

Re: Magika: AI powered fast and efficient file type identification

#135

Post by stevepike »

metafunctor wrote: post #70
stevepike wrote: post #10

Oh man, this brings me back! Almost 10 years ago I was working on a rails app trying to detect the file type of uploaded spreadsheets (xlsx files were being detected as application/zip, which is technically true but useless). I found "magic" that could detect these and submitted a patch at https://bugs.freedesktop.org/show_bug.cgi?id=78797 . My patch got rejected for needing to look at the first 3KB bytes of the file…

I've ended up implementing a layer on top of "magic" which, if magic detects application/zip, reads the zip file manifest and checks for telltale file names to reliably detect Office files. The "magic" library does not seem to be equipped with the capabilities needed to be robust against the zip manifest being ordered in a different way than expected. But this deep learning approach... I don't know. It might be hard…

If you're curious, here's how I solved it for ruby back in the day. Still used magic bytes, but added an overlay on top of the freedesktop.org DB: https://github.com/mimemagicrb/mimemagic/pull/20

Re: Magika: AI powered fast and efficient file type identification

#136

Post by michaelmior »

wongarsu wrote: post #115
true_religion wrote: post #89

Earlier quoted context omitted.

What end users are working with arbitrary files that they don’t know the identification of? This entire use case seems to be one suited for servers handling user media.

File managers that render preview images. Even detecting which software to open the file with when you click it. Of course on Windows the convention is to use the file extension, but on other platforms the convention is to look at the file contents

> on other platforms the convention is to look at the file contents

MacOS (that is, Finder) also looks at the extension. That has also been the case with any file manager I've used on Linux distros that I can recall.

Re: Magika: AI powered fast and efficient file type identification

#137

Post by michaelmior »

michaelt wrote: post #95
true_religion wrote: post #89

Earlier quoted context omitted.

What end users are working with arbitrary files that they don’t know the identification of? This entire use case seems to be one suited for servers handling user media.

Theoretically? Anyone running a virus scanner. Of course, it's arguably unlikely a virus scanner would opt for an ML-based approach, as they specifically need to be robust against adversarial inputs.

> it's arguably unlikely a virus scanner would opt for an ML-based approach

Several major players such as Norton, McAfee, and Symantec all at least claim to use AI/ML in their antivirus products.

Re: Magika: AI powered fast and efficient file type identification

#138

Post by Majestic121 »

Vogtinator wrote: post #88
samtheprogram wrote: post #13

Earlier quoted context omitted.

This is beyond what file is capable of. It’s also mentioned in the third paragraph. RTFA.

FWICT file is more capable, predictable and also faster while being more energy-efficient at the same time.

That's not what the performance table in the article is implying, with a precision and recall higher on Magika hovering around 99%, while magic is at 92% prec and 72% recall.

One can doubt the representativity of their dataset, but if what is in the article is correct, Magika is clearly way more capable and predictable

Re: Magika: AI powered fast and efficient file type identification

#139

Post by ebursztein »

TomNomNom wrote: post #132

This looks cool. I ran this on some web crawl data I have locally, so: all files you'd find on regular websites; HTML, CSS, JavaScript, fonts etc. It identified some simple HTML files (html, head, title, body, p tags and not much else) as "MS Visual Basic source (VBA)", "ASP source (code)", and "Generic text document" where the `file` utility correctly identified all such examples as "HTML document text". Some woff a…

Thanks for the feedback -- we will look into it. If you can share with us the list of URL that would be very helpful so we can reproduce - send us an email at magika-dev@google.com if that is possible.

For crawling we have planned a head only model to avoid fetching the whole file but it is not ready yet -- we weren't sure what use-cases would emerge so that is good to know that such model might be useful.

We mostly use Magika internally to route files for AV scanning as we wrote in the blog post, so it is possible that despite our best effort to test Magika extensively on various file types it is not as good on fonts format as it should be. We will look into.

Thanks again for sharing your experience with Magika this is very useful.

Re: Magika: AI powered fast and efficient file type identification

#140

Post by jdiff »

michaelmior wrote: post #134
TomNomNom wrote: post #132

This looks cool. I ran this on some web crawl data I have locally, so: all files you'd find on regular websites; HTML, CSS, JavaScript, fonts etc. It identified some simple HTML files (html, head, title, body, p tags and not much else) as "MS Visual Basic source (VBA)", "ASP source (code)", and "Generic text document" where the `file` utility correctly identified all such examples as "HTML document text". Some woff a…

> the current implementation can't be relied on IMO What's your reasoning for not relying on this? (It seems to me that this would be application-dependent at the very least.)

I'm not the person you asked, but I'm not sure I understand your question and I'd like to. It whiffed multiple common softballs, to the point it brings into question the claims made about its performance. What reasoning is there to trust it?
Post reply on HN