Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Magika: AI powered fast and efficient file type identification

opensource.googleblog.com

Post reply on HN
Print view View original

21–30 of 262 posts

Re: Magika: AI powered fast and efficient file type identification

#21

Post by stevepike »

renonce wrote: post #19
stevepike wrote: post #10

Oh man, this brings me back! Almost 10 years ago I was working on a rails app trying to detect the file type of uploaded spreadsheets (xlsx files were being detected as application/zip, which is technically true but useless). I found "magic" that could detect these and submitted a patch at https://bugs.freedesktop.org/show_bug.cgi?id=78797 . My patch got rejected for needing to look at the first 3KB bytes of the file…

From the first paragraph: > enabling precise file identification within milliseconds, even when running on a CPU. Maybe your old-fashioned implementations were detecting in microseconds?

Yeah I saw that, but that could cover a pretty wide range and it's not clear to me whether that relies on preloading a model.

Re: Magika: AI powered fast and efficient file type identification

#24

Post by SnowflakeOnIce »

semitones wrote: post #7

Is it really common enough for files not to be annotated with a useful/correct file type extension (e.g. .mp3, .txt) that a library like this is needed?

Yes!

Sometimes a file has no extension. Other times the extension is a lie. Still other times, you may be dealing with an unnamed bytestring and wish to know what kind of content it is.

This last case happens quite a lot in Nosey Parker [1], a detector of secrets in textual data. There, it is possible to come across unnamed files in Git history, and it would be useful to the user to still indicate what type of file it seems to be.

I added file type detection based on libmagic to Nosey Parker a while back, but it's not compiled in by default because libmagic is slow and complicates the build process. Also, libmagic is implemented as a large C library whose primary job is parsing, which makes the security side of me jittery.

I will likely add enabled-by-default filetype detection to Nosey Parker using Magika's ONNX model.

[1] https://github.com/praetorian-inc/noseyparker

Re: Magika: AI powered fast and efficient file type identification

#25

Post by _3u10 »

s1mon wrote: post #20
thorum wrote: post #5

Supported file types: https://github.com/google/magika/blob/main/docs/supported-co...

It's surprising that there are so many file types that seem relatively common which are missing from this list. There are no raw image file formats. There's nothing for CAD - either source files or neutral files. There's no MIDI files, or any other music creation types. There's no APL, Pascal, COBOL, assembly source file formats etc.

No tracker / .mod files either, just use file.

Re: Magika: AI powered fast and efficient file type identification

#26

Post by rafram »

userbinator wrote: post #8

Today web browsers, code editors, and countless other software rely on file-type detection to decide how to properly render a file. "web browsers"? Odd to see this coming from Google itself. https://en.wikipedia.org/wiki/Content_sniffing was widely criticised for being problematic for security.

Content sniffing can be disabled by the server (X-Content-Type-Options: nosniff), but it’s still used by default. Web browsers have to assume that servers are stupid, and that for relatively harmless cases, it’s fine to e.g. render a PNG loaded by an even if it’s served as text/plain.

Re: Magika: AI powered fast and efficient file type identification

#28

Post by jamesdwilson »

NiloCK wrote: post #2

A somewhat surprising and genuinely useful application of the family of techniques. I wonder how susceptible it is to adversarial binaries or, hah, prompt-injected binaries.

For the extremely limited number of file types supported, I question the utility of this compared to `magic`

Re: Magika: AI powered fast and efficient file type identification

#29

Post by charcircuit »

lifthrasiir wrote: post #12

I'm extremely confused about the claim that other tools have a worse precision or recall for APK or JAR files which are very much regular. Like, they should be a valid ZIP file with `META-INF/MANIFEST.MF` present (at least), and APK would need `classes.dex` as well, but at this point there is no other format that can be confused with APK or JAR I believe. I'd like to see which file was causing unexpected drop on prec…

apks are also zipaligned so it's not like random users are going to be making them either

Re: Magika: AI powered fast and efficient file type identification

#30

Post by wruza »

samtheprogram wrote: post #13
rfl890 wrote: post #11

We have had file(1) for years

This is beyond what file is capable of. It’s also mentioned in the third paragraph. RTFA.

Some HN readers may not know about file(1) even. It's fine to mention that $subj enhances that, but the rtfa part seems pretty unnecessary.
Post reply on HN