Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Magika: AI powered fast and efficient file type identification

opensource.googleblog.com

Post reply on HN
Print view View original

221–230 of 262 posts

Re: Magika: AI powered fast and efficient file type identification

#221

Post by duskwuff »

jdiff wrote: post #162
michaelmior wrote: post #160

Earlier quoted context omitted.

Except a 100% correct implementation doesn't exist AFAIK. So if I want to do anything that makes a decision based on the type of a file, I have to pick some algorithm to do that. If I can do that correctly 99% of the time, that's better than not being able to make that decision at all, which is where I'm left if a perfect implementation doesn't exist.

Nobody's asking for perfection. But the AI is offering inexplicable and obvious nondeterministic mistakes that the traditional algorithms don't suffer from. Magika goes wrong and your fonts become audio files and nobody knows why. Magic goes wrong and your ZIP-based documents get mistaken for generic ZIP files. If you work with that edge case a lot, you can anticipate it with traditional algorithms. You can't anticip…

Seconding this.

Something like Magika is potentially useful as a second pass if conventional methods of detecting a file type fail or yield a low-confidence result. But, for the majority of binary files, those conventional methods are perfectly adequate. If the first few bytes of a file are "GIF89a", you don't need an AI to tell you that it's probably a GIF image.

Re: Magika: AI powered fast and efficient file type identification

#223

Post by renewiltord »

beeboobaa wrote: post #170
TomNomNom wrote: post #142

Earlier quoted context omitted.

Sure thing :) Here's[0] a .tgz file with 3 files in it that are misidentified by magika but correctly identified by the `file` utility: asp.html, vba.html, unknown.woff These are files that were in one of my crawl datasets. [0]: https://poc.lol/files/magika-test.tgz

Do you have permission to redistribute these files?

LOL nice b8 m8. For the rest of you who are curious, the files look like this:

    
    Access Denied
    
    Access Denied
     
    You don't have permission to access "http://placement.api.test4.example.com/" on this server.

    Reference #18.9cb0f748.1695037739.283e2e00
Legend. "Do you have permission" hahaha.

Re: Magika: AI powered fast and efficient file type identification

#224

Post by samatman »

diffeomorphism wrote: post #154
vertis wrote: post #120

Earlier quoted context omitted.

That's a slippery slope argument, which is a common logical fallacy[0]. This model being inefficient compared to the best possible implementation does not mean that future additions will also be inefficient. It's the equivalent to saying many people programming in Ruby is causing all future programs to be less efficient. Which is not true. In fact, many people programming in Ruby has caused Ruby to become more effici…

That is a fallacy fallacy. Just because some slopes are not slippery that does not mean none of them are.

The slippery slope fallacy is: "this is a slope. you will slip down it." and is always fallacious. Always. The valid form of such an argument is: "this is a slope, and it is a slippery one, therefore, you will slip down it."

Re: Magika: AI powered fast and efficient file type identification

#225

Post by runald »

This is useful for detecting file types of unknown blobs with custom file extension, when the file command just returns data. Though it doesn't correctly identify lua code for some reason, it guesses with low probability that it's either ruby or javascript, or anything but lua.

Re: Magika: AI powered fast and efficient file type identification

#226

Post by EnigmaFlare »

jdiff wrote: post #140
michaelmior wrote: post #134

Earlier quoted context omitted.

> the current implementation can't be relied on IMO What's your reasoning for not relying on this? (It seems to me that this would be application-dependent at the very least.)

I'm not the person you asked, but I'm not sure I understand your question and I'd like to. It whiffed multiple common softballs, to the point it brings into question the claims made about its performance. What reasoning is there to trust it?

It had 3 failures. How is that a sign it's untrustworthy? I'm sure all alternatives have more than 3 failures. You might be making assumptions about the distribution of successes and failures (GP didn't say how many files they tested to find those 3) or how "soft" they were. In an extreme case, they might even have been crafted adversarial examples. But even if not, they might have features that really do look more like some other file type from the point of view of the classifier even if it's not easily apparent to a human. Being strictly superior to a competent human is a pretty high bar to set.

Re: Magika: AI powered fast and efficient file type identification

#227

Post by epcoa »

EnigmaFlare wrote: post #226
jdiff wrote: post #140

Earlier quoted context omitted.

I'm not the person you asked, but I'm not sure I understand your question and I'd like to. It whiffed multiple common softballs, to the point it brings into question the claims made about its performance. What reasoning is there to trust it?

It had 3 failures. How is that a sign it's untrustworthy? I'm sure all alternatives have more than 3 failures. You might be making assumptions about the distribution of successes and failures (GP didn't say how many files they tested to find those 3) or how "soft" they were. In an extreme case, they might even have been crafted adversarial examples. But even if not, they might have features that really do look more l…

> or how "soft" they were.

From the comment: It identified some simple HTML files (html, head, title, body, p tags and not much else) as "MS Visual Basic source (VBA)", "ASP source (code)", and "Generic text document" where the `file` utility correctly identified all such examples as "HTML document text".

That's pretty soft. Nothing "adversarial" claimed either.

> Being strictly superior to a competent human is a pretty high bar to set.

The bar is the file utility.

Re: Magika: AI powered fast and efficient file type identification

#228

Post by EnigmaFlare »

jdiff wrote: post #162
michaelmior wrote: post #160

Earlier quoted context omitted.

Except a 100% correct implementation doesn't exist AFAIK. So if I want to do anything that makes a decision based on the type of a file, I have to pick some algorithm to do that. If I can do that correctly 99% of the time, that's better than not being able to make that decision at all, which is where I'm left if a perfect implementation doesn't exist.

Nobody's asking for perfection. But the AI is offering inexplicable and obvious nondeterministic mistakes that the traditional algorithms don't suffer from. Magika goes wrong and your fonts become audio files and nobody knows why. Magic goes wrong and your ZIP-based documents get mistaken for generic ZIP files. If you work with that edge case a lot, you can anticipate it with traditional algorithms. You can't anticip…

Doesn't seem all that non-deterministic. I tested the vba.html example multiple times and it always said it was VBA. I added a space between and and it correctly picked HTML as most likely but with a low confidence.

So I think we can say it's sensitive to mysterious features, not that it's non-deterministic. Still leads to your same conclusion that you can't anticipate the failures. But I don't think you can with traditional tools either. Some magic numbers are just plain text (like MZ) which could legitimately accidentally appear at the beginning of a plain text file, for example.

Re: Magika: AI powered fast and efficient file type identification

#229

Post by EnigmaFlare »

epcoa wrote: post #227
EnigmaFlare wrote: post #226

Earlier quoted context omitted.

It had 3 failures. How is that a sign it's untrustworthy? I'm sure all alternatives have more than 3 failures. You might be making assumptions about the distribution of successes and failures (GP didn't say how many files they tested to find those 3) or how "soft" they were. In an extreme case, they might even have been crafted adversarial examples. But even if not, they might have features that really do look more l…

> or how "soft" they were. From the comment: It identified some simple HTML files (html, head, title, body, p tags and not much else ) as "MS Visual Basic source (VBA)", "ASP source (code)", and "Generic text document" where the `file` utility correctly identified all such examples as "HTML document text". That's pretty soft. Nothing "adversarial" claimed either. > Being strictly superior to a competent human is a pr…

Those are only soft to a human. I looked at a couple and I picked them correctly but I don't know what details the classifier was seeing which I was blind to. Not to say it was correct, just that we can't call them soft just because they're short and easy for a human.

> The bar is the file utility.

It has higher accuracy than that. You would reject it just because the failures are different even though they're less?

Re: Magika: AI powered fast and efficient file type identification

#230

Post by eapriv »

TeMPOraL wrote: post #182
jsnell wrote: post #169

Earlier quoted context omitted.

Where are you getting the non-determinism part from? It would seem surprising for there to be anything non-deterministic about an ML model like this, and nothing in the original reports seems to suggest that either.

Large ML models tend to be uncorrectably non-deterministic simply from doing lots of floating point math in parallel. Addition and multiplication of floats is neither commutative nor associative - you may get different results depending on the order in which you add/multiply numbers.

Addition and multiplication of floats are commutative.
Post reply on HN