Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Magika: AI powered fast and efficient file type identification

opensource.googleblog.com

Post reply on HN
Print view View original

31–40 of 262 posts

Re: Magika: AI powered fast and efficient file type identification

#31

Post by HtmlProgrammer »

lifthrasiir wrote: post #12

I'm extremely confused about the claim that other tools have a worse precision or recall for APK or JAR files which are very much regular. Like, they should be a valid ZIP file with `META-INF/MANIFEST.MF` present (at least), and APK would need `classes.dex` as well, but at this point there is no other format that can be confused with APK or JAR I believe. I'd like to see which file was causing unexpected drop on prec…

Minecraft mods 14 years ago used to tell you to open the JAR and delete the META-INF when installing them so can’t rely on that one…

Re: Magika: AI powered fast and efficient file type identification

#32

Post by floobertoober »

samtheprogram wrote: post #15
Am4TIfIsER0ppos wrote: post #14

Earlier quoted context omitted.

[flagged]

This a 1mb Keras ML model that’s open source. I passionately dislike Surveillance Capitalism but bringing it up when it’s completely irrelevant only weakens the argument. RTFA.

> Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".

From the HN guidelines

Re: Magika: AI powered fast and efficient file type identification

#33

Post by m0shen »

As someone that has worked in a space that has to deal with uploaded files for the last few years, and someone who maintains a WASM libmagic Node package ( https://github.com/moshen/wasmagic ) , I have to say I really love seeing new entries into the file type detection space.

Though I have to say when looking at the Node module, I don't understand why they released it.

Their docs say it's slow:

https://github.com/google/magika/blob/120205323e260dad4e5877...

It loads the model an runtime:

https://github.com/google/magika/blob/120205323e260dad4e5877...

They mark it as Experimental in the documentation, but it seems like it was just made for the web demo.

Also as others have mentioned. The model appears to only detect 116 file types:

https://github.com/google/magika/blob/120205323e260dad4e5877...

Where libmagic detects... a lot. Over 1600 last time I checked:

https://github.com/file/file/tree/4cbd5c8f0851201d203755b76c...

I guess I'm confused by this release. Sure it detected most of my list of sample files, but in a sample set of 4 zip files, it misidentified one.

Re: Magika: AI powered fast and efficient file type identification

#35

Post by kazinator »

> So far, libmagic and most other file-type-identification software have been relying on a handcrafted collection of heuristics and custom rules to detect each file format.

This manual approach is both time consuming and error prone as it is hard for humans to create generalized rules by hand.

Pure nonsense. The rules are accurate, based on the actual formats, and not "heuristics".

Re: Magika: AI powered fast and efficient file type identification

#36

Post by lebean »

m0shen wrote: post #33

As someone that has worked in a space that has to deal with uploaded files for the last few years, and someone who maintains a WASM libmagic Node package ( https://github.com/moshen/wasmagic ) , I have to say I really love seeing new entries into the file type detection space. Though I have to say when looking at the Node module, I don't understand why they released it. Their docs say it's slow: https://github.com/go…

It's for researchers, probably.

Re: Magika: AI powered fast and efficient file type identification

#37

Post by m0shen »

m0shen wrote: post #33

As someone that has worked in a space that has to deal with uploaded files for the last few years, and someone who maintains a WASM libmagic Node package ( https://github.com/moshen/wasmagic ) , I have to say I really love seeing new entries into the file type detection space. Though I have to say when looking at the Node module, I don't understand why they released it. Their docs say it's slow: https://github.com/go…

Made a small test to try it out: https://gist.github.com/moshen/784ee4a38439f00b17855233617e9...

    hyperfine ./magika.bash ./file.bash
    Benchmark 1: ./magika.bash
      Time (mean ± σ):     706.2 ms ±  21.1 ms    [User: 10520.3 ms, System: 1604.6 ms]
      Range (min … max):   684.0 ms … 738.9 ms    10 runs
    
    Benchmark 2: ./file.bash
      Time (mean ± σ):      23.6 ms ±   1.1 ms    [User: 15.7 ms, System: 7.9 ms]
      Range (min … max):    22.4 ms …  29.0 ms    111 runs
    
    Summary
      './file.bash' ran
       29.88 ± 1.65 times faster than './magika.bash'

Re: Magika: AI powered fast and efficient file type identification

#39

Post by m0shen »

lebean wrote: post #36
m0shen wrote: post #33

As someone that has worked in a space that has to deal with uploaded files for the last few years, and someone who maintains a WASM libmagic Node package ( https://github.com/moshen/wasmagic ) , I have to say I really love seeing new entries into the file type detection space. Though I have to say when looking at the Node module, I don't understand why they released it. Their docs say it's slow: https://github.com/go…

It's for researchers, probably.

Yeah, there is this line:

    By open-sourcing Magika, we aim to help other software improve their file identification accuracy and offer researchers a reliable method for identifying file types at scale.
Which implies a production-ready release for general usage, as well as usage by security researchers.

Re: Magika: AI powered fast and efficient file type identification

#40

Post by [deleted] »

kazinator wrote: post #35

> So far, libmagic and most other file-type-identification software have been relying on a handcrafted collection of heuristics and custom rules to detect each file format. This manual approach is both time consuming and error prone as it is hard for humans to create generalized rules by hand. Pure nonsense. The rules are accurate, based on the actual formats, and not "heuristics".

[deleted]
Post reply on HN