Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Magika: AI powered fast and efficient file type identification

opensource.googleblog.com

Post reply on HN
Print view View original

71–80 of 262 posts

Re: Magika: AI powered fast and efficient file type identification

#71

Post by m0shen »

supriyo-biswas wrote: post #61
m0shen wrote: post #54

Earlier quoted context omitted.

This is false in every sense for https://www.darwinsys.com/file/ (probably the most used file version). It depends on the magic for a specific file, but it can check any part of your file. Many Linux distros are years out of date, you might be using a very old version. FILE_45: ./src/file -m magic/magic.mgc ../../OpenCalc.v2.3.1.apk ../../OpenCalc.v2.3.1.apk: Android package (APK), with zipflinger virtual entry, with…

Interesting! I checked with file 5.44 from Ubuntu 23.10 and 5.45 on macOS using homebrew, and in both cases, I got “Zip archive data, at least v2.0 to extract” for the file here[1]. I don’t have an Android phone to check and I’m also not familiar with Android tooling, so is this a corrupt APK? [1] https://download.apkpure.net/custom/com.apkpure.aegon-319781...

That doesn't appear to be a valid link. Try building `file` from source and using the provided default magic database.

Re: Magika: AI powered fast and efficient file type identification

#72

Post by photoGrant »

s1mon wrote: post #20
thorum wrote: post #5

Supported file types: https://github.com/google/magika/blob/main/docs/supported-co...

It's surprising that there are so many file types that seem relatively common which are missing from this list. There are no raw image file formats. There's nothing for CAD - either source files or neutral files. There's no MIDI files, or any other music creation types. There's no APL, Pascal, COBOL, assembly source file formats etc.

Yeah this quickly went from 'additional helpful tool in the kit' to 'probably should use something else first'

Re: Magika: AI powered fast and efficient file type identification

#73

Post by jpk »

ebursztein wrote: post #65
stevepike wrote: post #10

Oh man, this brings me back! Almost 10 years ago I was working on a rails app trying to detect the file type of uploaded spreadsheets (xlsx files were being detected as application/zip, which is technically true but useless). I found "magic" that could detect these and submitted a patch at https://bugs.freedesktop.org/show_bug.cgi?id=78797 . My patch got rejected for needing to look at the first 3KB bytes of the file…

Co-author of Magika here (Elie) so we didn't include the measurements in the blog post to avoid making it too long but we did those measurements. Overall file takes about 6ms (single file) 2.26ms per files when scanning multiples. Magika is at 65ms single file and 5.3ms when scanning multiples. So Magika is for the worst case scenario about 10x slower due to the time it takes to load the model and 2x slower on repeat…

Do you have a sense of performance in terms of energy use? 2x slower is fine, but is that at the same wattage, or more?

Re: Magika: AI powered fast and efficient file type identification

#74

Post by supriyo-biswas »

m0shen wrote: post #71
supriyo-biswas wrote: post #61

Earlier quoted context omitted.

Interesting! I checked with file 5.44 from Ubuntu 23.10 and 5.45 on macOS using homebrew, and in both cases, I got “Zip archive data, at least v2.0 to extract” for the file here[1]. I don’t have an Android phone to check and I’m also not familiar with Android tooling, so is this a corrupt APK? [1] https://download.apkpure.net/custom/com.apkpure.aegon-319781...

That doesn't appear to be a valid link. Try building `file` from source and using the provided default magic database.

I also tried this with the sources of file from the homepage you linked above, and I still get the same results.

You could try this for yourself using the same APKPure file which I uploaded at the following alternative link[1]. Further, while this could be a corrupt APK, I can’t see any signs of that from a cursory inspection as both the `classes.dex` and `META-INF` directory are present, and this is APKPure’s own APK, instead of an APK contributed for an app contributed by a third-party.

[1] https://wormhole.app/Mebmy#CDv86juV9H4aRCL2DSJeDw

Re: Magika: AI powered fast and efficient file type identification

#75

Post by schleck8 »

LiamPowell wrote: post #66
ebursztein wrote: post #63

Earlier quoted context omitted.

Indeed but as pointed out in the blog post -- file is significantly less accurate that Magika. There are also some file type that we support and file doesn't as reported in the table.

I can't immediately find the dataset used for benchmarking. Is file actually failing on common files or just particularly nasty examples? If it's the latter then how does it compare to Magika on files that an average person is likely to see?

> Is file actually failing on common files or just particularly nasty examples? If it's the latter then how does it compare to Magika on files that an average person is likely to see?

That's not the point in file type guessing is it? Google employs it as an additional security measure for user submitted content which absolutely makes sense given what malware devs do with file types.

Re: Magika: AI powered fast and efficient file type identification

#77

Post by JacobThreeThree »

account-5 wrote: post #76

Assuming that I've not misunderstood, how does this compare to things like: TrID [0]?? Apart from being open source. [0] https://mark0.net/soft-trid-e.html

The bulk of the short article is a set of performance benchmarks comparing Magika to TrID and others.

Re: Magika: AI powered fast and efficient file type identification

#78

Post by alephnan »

jpk wrote: post #73
ebursztein wrote: post #65

Earlier quoted context omitted.

Co-author of Magika here (Elie) so we didn't include the measurements in the blog post to avoid making it too long but we did those measurements. Overall file takes about 6ms (single file) 2.26ms per files when scanning multiples. Magika is at 65ms single file and 5.3ms when scanning multiples. So Magika is for the worst case scenario about 10x slower due to the time it takes to load the model and 2x slower on repeat…

Do you have a sense of performance in terms of energy use? 2x slower is fine, but is that at the same wattage, or more?

That sounds like a nit / premature optimization.

Electricity is cheap. If this is sufficiently or actually important for your org, you should measure it yourself. There are too many variables and factors subject to your org’s hardware.

Re: Magika: AI powered fast and efficient file type identification

#79

Post by account-5 »

JacobThreeThree wrote: post #77
account-5 wrote: post #76

Assuming that I've not misunderstood, how does this compare to things like: TrID [0]?? Apart from being open source. [0] https://mark0.net/soft-trid-e.html

The bulk of the short article is a set of performance benchmarks comparing Magika to TrID and others.

Argh, the risks of browsing the web without JavaScript and/or third party scripts enabled, you miss content, because rendering text and images on the modern web can't be done without them, apparently. (Sarcasm).

You are of course correct. I can see the images showing the comparison. Apologies.

Post reply on HN