Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Magika: AI powered fast and efficient file type identification

opensource.googleblog.com

Post reply on HN
Print view View original

111–120 of 262 posts

Re: Magika: AI powered fast and efficient file type identification

#111

Post by comboy »

metafunctor wrote: post #70
stevepike wrote: post #10

Oh man, this brings me back! Almost 10 years ago I was working on a rails app trying to detect the file type of uploaded spreadsheets (xlsx files were being detected as application/zip, which is technically true but useless). I found "magic" that could detect these and submitted a patch at https://bugs.freedesktop.org/show_bug.cgi?id=78797 . My patch got rejected for needing to look at the first 3KB bytes of the file…

I've ended up implementing a layer on top of "magic" which, if magic detects application/zip, reads the zip file manifest and checks for telltale file names to reliably detect Office files. The "magic" library does not seem to be equipped with the capabilities needed to be robust against the zip manifest being ordered in a different way than expected. But this deep learning approach... I don't know. It might be hard…

Many commenters seem to be using magic instead of file, any reasons?

Re: Magika: AI powered fast and efficient file type identification

#112

Post by amelius »

potatoman22 wrote: post #52
lopkeny12ko wrote: post #51

I don't understand why this needs to exist. Isn't file type detection inherently deterministic by nature? A valid tar archive will always have the same first few magic bytes. An ELF binary has a universal ELF magic and header. If the magic is bad, then the file is corrupted and not a valid XYZ file. What's the value in throwing in "heuristics" and probabilistic inference into a process that is black and white by desi…

This also works for formats like Python, HTML, and JSON.

Yes, but shouldn't the file type be part of the file, or (better) of the metadata of the file?

Knowing is better than guessing.

Re: Magika: AI powered fast and efficient file type identification

#113

Post by Someone »

lifthrasiir wrote: post #12

I'm extremely confused about the claim that other tools have a worse precision or recall for APK or JAR files which are very much regular. Like, they should be a valid ZIP file with `META-INF/MANIFEST.MF` present (at least), and APK would need `classes.dex` as well, but at this point there is no other format that can be confused with APK or JAR I believe. I'd like to see which file was causing unexpected drop on prec…

People do create JAR files without a META-INF/MANIFEST.MF entry.

The tooling even supports it. https://docs.oracle.com/en/java/javase/21/docs/specs/man/jar...:

  -M or --no-manifest
     Doesn't create a manifest file for the entries

Re: Magika: AI powered fast and efficient file type identification

#115

Post by wongarsu »

true_religion wrote: post #89
djxfade wrote: post #81

Earlier quoted context omitted.

Totally disagree. Most end users are on laptops and mobile devices these days, not desktop towers. Thus power efficiency is important for battery life. Performance per watt would be an interesting comparison.

What end users are working with arbitrary files that they don’t know the identification of? This entire use case seems to be one suited for servers handling user media.

File managers that render preview images. Even detecting which software to open the file with when you click it.

Of course on Windows the convention is to use the file extension, but on other platforms the convention is to look at the file contents

Re: Magika: AI powered fast and efficient file type identification

#116

Post by kevincox »

rvnx wrote: post #94
awaythrow999 wrote: post #92

Wonder how this would handle a polyglot[0][1], that is valid as a PDF document, a ZIP archive, and a Bash script that runs a Python webserver, which hosts Kaitai Struct’s WebIDE which, allowing you to view the file’s own annotated bytes. [0]: https://www.alchemistowl.org/pocorgtfo/ [1]: https://www.alchemistowl.org/pocorgtfo/pocorgtfo16.pdf Edit: just tested, and it does only identify the zip layer

You can try it here: https://google.github.io/magika/ It's relatively limited compared to `file` (~10% coverage), it's more like a specialized classificator for basic file formats, so such cases are really out-of-scope. I guess it's more for detecting common file formats then with high recall. However, where is the actual source of the model ? Let's say I want to add a new file format myself. Apparently only the sour…

Apparenty the Super Mario Bros. 3 ROM is 100% a SWF file.

Cool that you can use it online though. Might end up using it like that. Although it seems like it may focus on common formats.

Re: Magika: AI powered fast and efficient file type identification

#117

Post by kevincox »

s1mon wrote: post #20
thorum wrote: post #5

Supported file types: https://github.com/google/magika/blob/main/docs/supported-co...

It's surprising that there are so many file types that seem relatively common which are missing from this list. There are no raw image file formats. There's nothing for CAD - either source files or neutral files. There's no MIDI files, or any other music creation types. There's no APL, Pascal, COBOL, assembly source file formats etc.

Worse it seems that for unknown formats it confidently claims that it is one of the known formats. Rather than saying "unknown" or "binary data".

Re: Magika: AI powered fast and efficient file type identification

#118

Post by e1g »

comboy wrote: post #111
metafunctor wrote: post #70

Earlier quoted context omitted.

I've ended up implementing a layer on top of "magic" which, if magic detects application/zip, reads the zip file manifest and checks for telltale file names to reliably detect Office files. The "magic" library does not seem to be equipped with the capabilities needed to be robust against the zip manifest being ordered in a different way than expected. But this deep learning approach... I don't know. It might be hard…

Many commenters seem to be using magic instead of file , any reasons?

magic is the core detection logic of file that was extracted out to be available as a library. So these days file is just a higher level wrapper around magic

Re: Magika: AI powered fast and efficient file type identification

#119

Post by brabel »

stevepike wrote: post #10

Oh man, this brings me back! Almost 10 years ago I was working on a rails app trying to detect the file type of uploaded spreadsheets (xlsx files were being detected as application/zip, which is technically true but useless). I found "magic" that could detect these and submitted a patch at https://bugs.freedesktop.org/show_bug.cgi?id=78797 . My patch got rejected for needing to look at the first 3KB bytes of the file…

> They had a hard limit that they wouldn't see past the first 256 bytes.

Then they could never detect zip files with certainty, given that to do that you need to read up to 65KB (+ 22) at the END of the file. The reason is that the zip archive format allows "gargabe" bytes both in the beginning of the file and in between local file headers.... and it's actually not uncommon to prepend a program that self-extracts the archive, for example. The only way to know if a file is a valid zip archive is to look for the End of Central Directory Entry, which is always at the end of the file AND allows for a comment of unknown length at the end (and as the comment length field takes 2 bytes, the comment can be up to 65K long).

Re: Magika: AI powered fast and efficient file type identification

#120

Post by vertis »

_puk wrote: post #105
vertis wrote: post #102

Earlier quoted context omitted.

I mean if you care about that you shouldn't be running anything that isn't highly optimized. Don't open webpages that might be CPU or GPU intensive. Don't run Electron apps, or really anything that isn't built in a compiled language. Certainly you should do an audit of all the Android and iOS apps as well, to make sure they've been made in a efficient manner. Block ads as well, they waste power. This file identificat…

Whilst energy usage is indeed a small aspect this early on when using bespoke models, we do have to consider that this is a model for simply identifying a file type. What happens when we introduce more bespoke models for manipulating the data in that file? This feels like it could slowly boil to the point of programs using magnitudes higher power, at which point it'll be hard to claw it back.

That's a slippery slope argument, which is a common logical fallacy[0]. This model being inefficient compared to the best possible implementation does not mean that future additions will also be inefficient.

It's the equivalent to saying many people programming in Ruby is causing all future programs to be less efficient. Which is not true. In fact, many people programming in Ruby has caused Ruby to become more efficient because it gets optimised as it gets used more (or Python for that matter).

It's not as energy efficient as C, but it hasn't caused it to get worse and worse, and spiral out of control.

Likewise smart contracts are incredibly inefficient mechanisms of computation. The result is mostly that people don't use them for any meaningful amounts of computation, that all gets done "Off Chain".

Generative AI is definitely less efficient, but it's likely to improve over time, and indeed things like quantization has allowed models that would normally to require much more substantial hardware resources (and therefore, more energy intensive) to be run on smaller systems.

[0]: https://en.wikipedia.org/wiki/Slippery_slope

Post reply on HN