Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Magika: AI powered fast and efficient file type identification

opensource.googleblog.com

Post reply on HN
Print view View original

171–180 of 262 posts

Re: Magika: AI powered fast and efficient file type identification

#171

Post by kevincox »

Andugal wrote: post #114

I have a question: Is something like Magika enough to check if a file is malicious or not? Example: users can upload PNG file (and only PNG is accepted). If Malika detects that the file is a PNG, does this mean the file is clean?

The only way to do this reliably is to render the PNG to pixels then render it back to an PNG with a trusted encoder. Of course now you are taking the risk of vulnerabilities in the "render to pixels" step. But the result will be clean.

AKA parse, don't validate.

Re: Magika: AI powered fast and efficient file type identification

#173

Post by m0shen »

tudorw wrote: post #145
m0shen wrote: post #33

As someone that has worked in a space that has to deal with uploaded files for the last few years, and someone who maintains a WASM libmagic Node package ( https://github.com/moshen/wasmagic ) , I have to say I really love seeing new entries into the file type detection space. Though I have to say when looking at the Node module, I don't understand why they released it. Their docs say it's slow: https://github.com/go…

Can we do the 1600 if known, if not, let the AI take a guess?

Absolutely, and honestly in a non-interactive ingestion workflow you're probably doing multiple checks anyway. I've worked with systems that call multiple libraries and hand-coded validation for each incoming file.

Maybe it's my general malaise, or disillusionment with the software industry, but when I wrote that I was really just expecting more.

Re: Magika: AI powered fast and efficient file type identification

#176

Post by NoGravitas »

samtheprogram wrote: post #13
rfl890 wrote: post #11

We have had file(1) for years

This is beyond what file is capable of. It’s also mentioned in the third paragraph. RTFA.

Yes, it's slower than file(1), uses more energy, recognizes fewer file types, and is less accurate.

Re: Magika: AI powered fast and efficient file type identification

#177

Post by YoshiRulz »

So instead of spending some of their human resources to improve libmagic, they used some of their computing power to create an "open source" neural net, which is technically more accurate than the "error-prone" hand-written rules (ignoring that it supports far fewer filetypes), and which is much less effective in an adversarial context, and they want it to "help other software improve their file identification accuracy," which of course it can't since neural nets aren't introspectable. Thanks guys.

Re: Magika: AI powered fast and efficient file type identification

#178

Post by m0shen »

m0shen wrote: post #37
m0shen wrote: post #33

As someone that has worked in a space that has to deal with uploaded files for the last few years, and someone who maintains a WASM libmagic Node package ( https://github.com/moshen/wasmagic ) , I have to say I really love seeing new entries into the file type detection space. Though I have to say when looking at the Node module, I don't understand why they released it. Their docs say it's slow: https://github.com/go…

Made a small test to try it out: https://gist.github.com/moshen/784ee4a38439f00b17855233617e9... hyperfine ./magika.bash ./file.bash Benchmark 1: ./magika.bash Time (mean ± σ): 706.2 ms ± 21.1 ms [User: 10520.3 ms, System: 1604.6 ms] Range (min … max): 684.0 ms … 738.9 ms 10 runs Benchmark 2: ./file.bash Time (mean ± σ): 23.6 ms ± 1.1 ms [User: 15.7 ms, System: 7.9 ms] Range (min … max): 22.4 ms … 29.0 ms 111 runs Su…

I've updated this script with some single-file cli numbers, which are (as expected) not good. Mostly just comparing python startup time for that.

    make
    sqlite3

Re: Magika: AI powered fast and efficient file type identification

#180

Post by 12_throw_away »

YoshiRulz wrote: post #177

So instead of spending some of their human resources to improve libmagic, they used some of their computing power to create an "open source" neural net, which is technically more accurate than the "error-prone" hand-written rules (ignoring that it supports far fewer filetypes), and which is much less effective in an adversarial context, and they want it to "help other software improve their file identification accura…

Come on, can't you help but be impressed by this amazing AI tech? That gives us sci-fi tools like ... a less-accurate, incomplete, stochastic, un-debuggable, slower, electricity-guzzling version of `file`.
Post reply on HN