We have had file(1) for years
Magika: AI powered fast and efficient file type identification
11–20 of 262 posts
Re: Magika: AI powered fast and efficient file type identification
#12I'm extremely confused about the claim that other tools have a worse precision or recall for APK or JAR files which are very much regular. Like, they should be a valid ZIP file with `META-INF/MANIFEST.MF` present (at least), and APK would need `classes.dex` as well, but at this point there is no other format that can be confused with APK or JAR I believe. I'd like to see which file was causing unexpected drop on precision or recall.
Re: Magika: AI powered fast and efficient file type identification
#13We have had file(1) for years
This is beyond what file is capable of. It’s also mentioned in the third paragraph.
RTFA.
Re: Magika: AI powered fast and efficient file type identification
#14We have had file(1) for years
[flagged]
Re: Magika: AI powered fast and efficient file type identification
#15Re: Magika: AI powered fast and efficient file type identification
#16Is it really common enough for files not to be annotated with a useful/correct file type extension (e.g. .mp3, .txt) that a library like this is needed?
Nothing is ever simple. Even for the most basic .txt files it’s still useful to know what the character encoding is (utf? 8/16? Latin-whatever? etc.) and what the line format is (\n,\cr\lf,\n\lf) as well as determining if some maniac removed all the indentation characters and replaced them with a mystery number of spaces.
Then there are all the container formats that have different kinds of formats embedded in them (mov,mkv,pdf etc.)
Re: Magika: AI powered fast and efficient file type identification
#17This feels like old school google. I like that it's just a static webpage that basically can't be shut down or sunsetted. It reminds of when Google just made useful stuff and gave them away for free on a webpage like translate and google books. Obviously less life changing than the above but still a great option to have when I need this.
Re: Magika: AI powered fast and efficient file type identification
#18At $job we have been using Apache Tika for years.
Works but occasionally having bugs and weird collisions when working with billions of files.
Happy to see new contributions in the space.
Re: Magika: AI powered fast and efficient file type identification
#19Oh man, this brings me back! Almost 10 years ago I was working on a rails app trying to detect the file type of uploaded spreadsheets (xlsx files were being detected as application/zip, which is technically true but useless). I found "magic" that could detect these and submitted a patch at https://bugs.freedesktop.org/show_bug.cgi?id=78797 . My patch got rejected for needing to look at the first 3KB bytes of the file…
From the first paragraph:
> enabling precise file identification within milliseconds, even when running on a CPU.
Maybe your old-fashioned implementations were detecting in microseconds?
Re: Magika: AI powered fast and efficient file type identification
#20Supported file types: https://github.com/google/magika/blob/main/docs/supported-co...
It's surprising that there are so many file types that seem relatively common which are missing from this list. There are no raw image file formats. There's nothing for CAD - either source files or neutral files. There's no MIDI files, or any other music creation types. There's no APL, Pascal, COBOL, assembly source file formats etc.