After reading thru all the comments, honestly I still don't get the point of this system. What is potential practical value or applications of this model?
Magika: AI powered fast and efficient file type identification
191–200 of 262 posts
Re: Magika: AI powered fast and efficient file type identification
#192Earlier quoted context omitted.
Do you have permission to redistribute these files?
You are asking what if this guy has "web crawl data" that google does not have? And what if he says no, he does not have permission.
> You are asking what if this guy has "web crawl data" that google does not have?
No, I'm asking if he has permission to redistribute these files.
Re: Magika: AI powered fast and efficient file type identification
#193I guess I'm kind of a dummy on this, but why is it impressive to identify that a .js file is Javascript, a .md file is Markdown, etc?
Re: Magika: AI powered fast and efficient file type identification
#194I ran a quick test on 100 semi-random files I had laying around. Of those, 81 were detected correctly, 6 were detected as the wrong file type, and 12 were detected with an unspecific file type (unknown binary/generic text) when a more specific type existed. In 4 of the unspecific cases, a low-confidence guess was provided, which was wrong in each case. However, almost all of the files which were detected wrong/unspec…
> The "wrong" here was somewhat skewed by the 4 GLSL shader code files that were in the dataset for some reason, all of which it detected as C code To be fair though, a snippet of GLSL shader code can be perfectly valid C.
Indeed, which is why I felt the need to call it out here. I'm not certain if the files on question actually happened to be valid C but whether that's a meaningful mistake regardless is left to the reader to decide.
Re: Magika: AI powered fast and efficient file type identification
#195Reminds me when someone asked (at StackOverflow) on how to recognize binaries for different architetures, like x86 or ARM-something or Apple M1 and so on.
I gave the idea to use the technique of NCD (Normalized compression distance), based on Kolmogorov complexity. Celibrasi, R. was one great researcher in this area, and I think he worked at Google at some point.
Using AI seems to follow the same path: "learn" what represents some specific file and then compare the unknown file to those references (AI:all the parameters, NCD:compression against a known type).
Re: Magika: AI powered fast and efficient file type identification
#196> So far, libmagic and most other file-type-identification software have been relying on a handcrafted collection of heuristics and custom rules to detect each file format. This manual approach is both time consuming and error prone as it is hard for humans to create generalized rules by hand. Pure nonsense. The rules are accurate, based on the actual formats, and not "heuristics".
1. Not all file formats are well specified
2. Not all files are precisely following the specification
3. Not all file formats are mutually exclusive
Those facts are clearly reflected in the table.
Re: Magika: AI powered fast and efficient file type identification
#197But will it let you print on Tuesday[1]?
1: https://bugs.launchpad.net/ubuntu/+source/cupsys/+bug/255161...
Re: Magika: AI powered fast and efficient file type identification
#198I guess I'm kind of a dummy on this, but why is it impressive to identify that a .js file is Javascript, a .md file is Markdown, etc?
Because it's done by inspecting the content, not the name of the file.
Re: Magika: AI powered fast and efficient file type identification
#199Oh man, this brings me back! Almost 10 years ago I was working on a rails app trying to detect the file type of uploaded spreadsheets (xlsx files were being detected as application/zip, which is technically true but useless). I found "magic" that could detect these and submitted a patch at https://bugs.freedesktop.org/show_bug.cgi?id=78797 . My patch got rejected for needing to look at the first 3KB bytes of the file…
FWIW, file can now distinguish many types of zip containers, including Oxml files.
Re: Magika: AI powered fast and efficient file type identification
#200Earlier quoted context omitted.
You are asking what if this guy has "web crawl data" that google does not have? And what if he says no, he does not have permission.
> You are asking what if this guy has "web crawl data" that google does not have? No, I'm asking if he has permission to redistribute these files.
Are you attempting to assert that use of these files solely for the purpose of improving a software system meant to classify file types does not fall under fair use?