Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Magika: AI powered fast and efficient file type identification

opensource.googleblog.com

Post reply on HN
Print view View original

121–130 of 262 posts

Re: Magika: AI powered fast and efficient file type identification

#121

Post by fuzztester »

nindalf wrote: post #46
krick wrote: post #44

What are use-cases for this? I mean, obviously detecting the filetype is useful, but we kinda already have plenty of tools to do that, and I cannot imagine, why we need some "smart" way of doing this. If you are not a human, and you are not sure what is it (like, an unknown file being uploaded to a server) you would be better off just rejecting it completely, right? After all, there's absolutely no way an "AI powered…

> no way an "AI powered" tool can be more reliable The article provides accuracy benchmarks. > you would be better off just rejecting it completely They mention using it in gmail and Drive, neither of which have the luxury of rejecting files willy-nilly.

I have not tried it recently, but IIRC, Gmail does reject attachments which are zip files, for security reasons.

Re: Magika: AI powered fast and efficient file type identification

#122

Post by TacticalCoder »

To me the obvious use case is to first use the file command but then, when file returns "DATA" (meaning it couldn't guess the file type), call magika.

I guess I'll be writing a wrapper (only for when using my shell in interactive mode) around file doing just that when I come back from vacation. I hate it when file cannot do its thing.

Put it this way: I use file a lot and I know at times it cannot detect a filetype. But is file often wrong when it does have a match? I don't think so...

So in most of the cases I'd have file correctly give me the filetype, very quickly but then in those rare cases where file cannot find anything, I'd then use the slower but apparently more capable magika.

Re: Magika: AI powered fast and efficient file type identification

#123

Post by cjg »

Andugal wrote: post #114

I have a question: Is something like Magika enough to check if a file is malicious or not? Example: users can upload PNG file (and only PNG is accepted). If Malika detects that the file is a PNG, does this mean the file is clean?

> does this mean the file is clean?

No.

Re: Magika: AI powered fast and efficient file type identification

#125

Post by TacticalCoder »

lopkeny12ko wrote: post #51

I don't understand why this needs to exist. Isn't file type detection inherently deterministic by nature? A valid tar archive will always have the same first few magic bytes. An ELF binary has a universal ELF magic and header. If the magic is bad, then the file is corrupted and not a valid XYZ file. What's the value in throwing in "heuristics" and probabilistic inference into a process that is black and white by desi…

> What's the value in throwing in "heuristics" and probabilistic inference into a process that is black and white by design.

I use the file command all the time. The value is when you get this:

    ... $  file somefile.xyz
    somefile.xyz: data
AIUI from reading TFA, magika can determine more filetypes than what the file command can detect.

It'd actually be very easy to determine if there's any value in magika: run file on every file on your filesystem and then for every file where the file command returns "data", run magika and see if magika is right.

If it's right, there's your value.

P.S: it may also be easier to run on Windows than the file command? But then I can't do much to help people who are on Windows.

Re: Magika: AI powered fast and efficient file type identification

#126

Post by cornholio »

alephnan wrote: post #78
jpk wrote: post #73

Earlier quoted context omitted.

Do you have a sense of performance in terms of energy use? 2x slower is fine, but is that at the same wattage, or more?

That sounds like a nit / premature optimization. Electricity is cheap. If this is sufficiently or actually important for your org, you should measure it yourself. There are too many variables and factors subject to your org’s hardware.

The hardware requirements of a massively parallel algorithm can't possibly be "a nit" in any universe inhabited by rational beings.

Re: Magika: AI powered fast and efficient file type identification

#127

Post by TacticalCoder »

Andugal wrote: post #114

I have a question: Is something like Magika enough to check if a file is malicious or not? Example: users can upload PNG file (and only PNG is accepted). If Malika detects that the file is a PNG, does this mean the file is clean?

If that PNG of yours is not just an example note that you can detect easily if the PNG as any extra data (which may or may not indicate an attempt as mischief) and reject the (rare) PNGs with extra data. I ran a script checking the thousands of PNGs on my system and found three with extra data, all three probably due to the "PNG acropalypse" bug (but mischief cannot be ruled out).

P.S: btw I'm not implying using extra data that shouldn't be there in a PNG is the only way to have a malicious PNG.

Re: Magika: AI powered fast and efficient file type identification

#128

Post by barrkel »

m0shen wrote: post #37
m0shen wrote: post #33

As someone that has worked in a space that has to deal with uploaded files for the last few years, and someone who maintains a WASM libmagic Node package ( https://github.com/moshen/wasmagic ) , I have to say I really love seeing new entries into the file type detection space. Though I have to say when looking at the Node module, I don't understand why they released it. Their docs say it's slow: https://github.com/go…

Made a small test to try it out: https://gist.github.com/moshen/784ee4a38439f00b17855233617e9... hyperfine ./magika.bash ./file.bash Benchmark 1: ./magika.bash Time (mean ± σ): 706.2 ms ± 21.1 ms [User: 10520.3 ms, System: 1604.6 ms] Range (min … max): 684.0 ms … 738.9 ms 10 runs Benchmark 2: ./file.bash Time (mean ± σ): 23.6 ms ± 1.1 ms [User: 15.7 ms, System: 7.9 ms] Range (min … max): 22.4 ms … 29.0 ms 111 runs Su…

Realistically, either you're identifying one file interactively and you don't care about latency differences in the 10s of ms, or you're identifying in bulk (batch command line or online in response to requests), in which case you should measure the marginal cost and exclude Python startup and model loading times.

Re: Magika: AI powered fast and efficient file type identification

#129

Post by cle »

lopkeny12ko wrote: post #51

I don't understand why this needs to exist. Isn't file type detection inherently deterministic by nature? A valid tar archive will always have the same first few magic bytes. An ELF binary has a universal ELF magic and header. If the magic is bad, then the file is corrupted and not a valid XYZ file. What's the value in throwing in "heuristics" and probabilistic inference into a process that is black and white by desi…

It's not always deterministic, sometimes it's fuzzy depending on the file type. Example of this is a one-line CSV file. I tested one case of that, libmagic detects it as a text file while magika correctly detects it as a CSV (and gives a confidence score, which is killer).

Re: Magika: AI powered fast and efficient file type identification

#130

Post by cle »

kazinator wrote: post #35

> So far, libmagic and most other file-type-identification software have been relying on a handcrafted collection of heuristics and custom rules to detect each file format. This manual approach is both time consuming and error prone as it is hard for humans to create generalized rules by hand. Pure nonsense. The rules are accurate, based on the actual formats, and not "heuristics".

Besides compound file types, not all formats are well-specified either. Example is CSV.
Post reply on HN