I don't understand why this needs to exist. Isn't file type detection inherently deterministic by nature? A valid tar archive will always have the same first few magic bytes. An ELF binary has a universal ELF magic and header. If the magic is bad, then the file is corrupted and not a valid XYZ file. What's the value in throwing in "heuristics" and probabilistic inference into a process that is black and white by design.
Magika: AI powered fast and efficient file type identification
51–60 of 262 posts
Re: Magika: AI powered fast and efficient file type identification
#52I don't understand why this needs to exist. Isn't file type detection inherently deterministic by nature? A valid tar archive will always have the same first few magic bytes. An ELF binary has a universal ELF magic and header. If the magic is bad, then the file is corrupted and not a valid XYZ file. What's the value in throwing in "heuristics" and probabilistic inference into a process that is black and white by desi…
This also works for formats like Python, HTML, and JSON.
Re: Magika: AI powered fast and efficient file type identification
#53Earlier quoted context omitted.
From the first paragraph: > enabling precise file identification within milliseconds, even when running on a CPU. Maybe your old-fashioned implementations were detecting in microseconds?
Yeah I saw that, but that could cover a pretty wide range and it's not clear to me whether that relies on preloading a model.
> At inference time Magika uses Onnx as an inference engine to ensure files are identified in a matter of milliseconds, almost as fast as a non-AI tool even on CPU.
Re: Magika: AI powered fast and efficient file type identification
#54I'm extremely confused about the claim that other tools have a worse precision or recall for APK or JAR files which are very much regular. Like, they should be a valid ZIP file with `META-INF/MANIFEST.MF` present (at least), and APK would need `classes.dex` as well, but at this point there is no other format that can be confused with APK or JAR I believe. I'd like to see which file was causing unexpected drop on prec…
The `file` command checks only the first few bytes, and doesn’t parse the structure of the file. APK files are indeed reported as Zip archives by the latest version of `file`.
This is false in every sense for https://www.darwinsys.com/file/ (probably the most used file version). It depends on the magic for a specific file, but it can check any part of your file. Many Linux distros are years out of date, you might be using a very old version.
FILE_45:
./src/file -m magic/magic.mgc ../../OpenCalc.v2.3.1.apk
../../OpenCalc.v2.3.1.apk: Android package (APK), with zipflinger virtual entry, with APK Signing BlockRe: Magika: AI powered fast and efficient file type identification
#55[dead]
Re: Magika: AI powered fast and efficient file type identification
#56We have had file(1) for years
This group of Linux users used to brag Linux will identify files even if you change the extension, Windoze needs to police you about changing extension, nearly 20 years back.
Re: Magika: AI powered fast and efficient file type identification
#57Is it really common enough for files not to be annotated with a useful/correct file type extension (e.g. .mp3, .txt) that a library like this is needed?
At multiple points in my career I've been responsible for apis that accept PDFs. Many non-tech savvy people seeing this, will just change the extension of the file they're uploading to `.pdf`.
To make matters worse, there is some business software out there that will actually bastardize the PDF format and put garbage before the PDF file header. So for some things you end up writing custom validation and cleanup logic anyway.
Re: Magika: AI powered fast and efficient file type identification
#58Oh man, this brings me back! Almost 10 years ago I was working on a rails app trying to detect the file type of uploaded spreadsheets (xlsx files were being detected as application/zip, which is technically true but useless). I found "magic" that could detect these and submitted a patch at https://bugs.freedesktop.org/show_bug.cgi?id=78797 . My patch got rejected for needing to look at the first 3KB bytes of the file…
From the first paragraph: > enabling precise file identification within milliseconds, even when running on a CPU. Maybe your old-fashioned implementations were detecting in microseconds?
[deleted]
Re: Magika: AI powered fast and efficient file type identification
#59As someone that has worked in a space that has to deal with uploaded files for the last few years, and someone who maintains a WASM libmagic Node package ( https://github.com/moshen/wasmagic ) , I have to say I really love seeing new entries into the file type detection space. Though I have to say when looking at the Node module, I don't understand why they released it. Their docs say it's slow: https://github.com/go…
We did release the npm package because indeed we create a web demo and thought people might want to also use it. We know it is not as fast as the python version or a C++ version -- which why we did mark it as experimental.
The release include the python package and the cli which are quite fast and is the main way we did expect people to use -- sorry if that hasn't be clear in the post.
The goal of the release is to offer a tool that is far more accurate that other tools and works on the major file types as we hope it to be useful to the community.
Glad to hear it worked on your files
Re: Magika: AI powered fast and efficient file type identification
#60I don't understand why this needs to exist. Isn't file type detection inherently deterministic by nature? A valid tar archive will always have the same first few magic bytes. An ELF binary has a universal ELF magic and header. If the magic is bad, then the file is corrupted and not a valid XYZ file. What's the value in throwing in "heuristics" and probabilistic inference into a process that is black and white by desi…
This also works for formats like Python, HTML, and JSON.
file (https://www.darwinsys.com/file/) already detects all these formats.