Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Magika: AI powered fast and efficient file type identification

opensource.googleblog.com

Post reply on HN
Print view View original

141–150 of 262 posts

Re: Magika: AI powered fast and efficient file type identification

#141

Post by jdiff »

michaelmior wrote: post #136
wongarsu wrote: post #115

Earlier quoted context omitted.

File managers that render preview images. Even detecting which software to open the file with when you click it. Of course on Windows the convention is to use the file extension, but on other platforms the convention is to look at the file contents

> on other platforms the convention is to look at the file contents MacOS (that is, Finder) also looks at the extension. That has also been the case with any file manager I've used on Linux distros that I can recall.

You might be surprised. Rename your Photo.JPG as Photo.PNG and you'll still get a perfectly fine thumbnail. The extension is a hint, but it isn't definitive, especially when you start downloading from the web.

Re: Magika: AI powered fast and efficient file type identification

#142

Post by TomNomNom »

ebursztein wrote: post #139
TomNomNom wrote: post #132

This looks cool. I ran this on some web crawl data I have locally, so: all files you'd find on regular websites; HTML, CSS, JavaScript, fonts etc. It identified some simple HTML files (html, head, title, body, p tags and not much else) as "MS Visual Basic source (VBA)", "ASP source (code)", and "Generic text document" where the `file` utility correctly identified all such examples as "HTML document text". Some woff a…

Thanks for the feedback -- we will look into it. If you can share with us the list of URL that would be very helpful so we can reproduce - send us an email at magika-dev@google.com if that is possible. For crawling we have planned a head only model to avoid fetching the whole file but it is not ready yet -- we weren't sure what use-cases would emerge so that is good to know that such model might be useful. We mostly…

Sure thing :)

Here's[0] a .tgz file with 3 files in it that are misidentified by magika but correctly identified by the `file` utility: asp.html, vba.html, unknown.woff

These are files that were in one of my crawl datasets.

[0]: https://poc.lol/files/magika-test.tgz

Re: Magika: AI powered fast and efficient file type identification

#143

Post by DontSignAnytng »

prmph wrote: post #93
underdeserver wrote: post #90

Earlier quoted context omitted.

In general you're right, but I can't think of a single local use for identifying file types by a human on a laptop - at least, one with scale where this matters. It's all going to be SaaS services where people upload stuff.

We are building a data analysis tool with great UX, where users select data files, which are then parsed and uploaded to S3 directly, on their client machines. The server only takes over after this step. Since the data files can be large, this approach bypasses having to trnasfer the file twice, first to the server, and then to S3 after parsing.

This dont sound like very common scenario.

Re: Magika: AI powered fast and efficient file type identification

#144

Post by Eiim »

TacticalCoder wrote: post #125
lopkeny12ko wrote: post #51

I don't understand why this needs to exist. Isn't file type detection inherently deterministic by nature? A valid tar archive will always have the same first few magic bytes. An ELF binary has a universal ELF magic and header. If the magic is bad, then the file is corrupted and not a valid XYZ file. What's the value in throwing in "heuristics" and probabilistic inference into a process that is black and white by desi…

> What's the value in throwing in "heuristics" and probabilistic inference into a process that is black and white by design. I use the file command all the time. The value is when you get this: ... $ file somefile.xyz somefile.xyz: data AIUI from reading TFA, magika can determine more filetypes than what the file command can detect. It'd actually be very easy to determine if there's any value in magika : run file on…

From elsewhere in this thread, it appears that Magika detects far fewer file types than file (116 vs ~1600), which makes sense. For file, you just need to drop in a few rules to add a new, somewhat obscure type. An AI approach like Magika will need lots of training and test data for each new file type. Where Magika might have a leg up is with distinguishing different textual data files (i.e., source code), but I don't see that as a particularly big use case honestly.

Re: Magika: AI powered fast and efficient file type identification

#145

Post by tudorw »

m0shen wrote: post #33

As someone that has worked in a space that has to deal with uploaded files for the last few years, and someone who maintains a WASM libmagic Node package ( https://github.com/moshen/wasmagic ) , I have to say I really love seeing new entries into the file type detection space. Though I have to say when looking at the Node module, I don't understand why they released it. Their docs say it's slow: https://github.com/go…

Can we do the 1600 if known, if not, let the AI take a guess?

Re: Magika: AI powered fast and efficient file type identification

#146

Post by ebursztein »

TomNomNom wrote: post #142
ebursztein wrote: post #139

Earlier quoted context omitted.

Thanks for the feedback -- we will look into it. If you can share with us the list of URL that would be very helpful so we can reproduce - send us an email at magika-dev@google.com if that is possible. For crawling we have planned a head only model to avoid fetching the whole file but it is not ready yet -- we weren't sure what use-cases would emerge so that is good to know that such model might be useful. We mostly…

Sure thing :) Here's[0] a .tgz file with 3 files in it that are misidentified by magika but correctly identified by the `file` utility: asp.html, vba.html, unknown.woff These are files that were in one of my crawl datasets. [0]: https://poc.lol/files/magika-test.tgz

Thank you - we are adding them to our test suit for the next version.

Re: Magika: AI powered fast and efficient file type identification

#147

Post by TomNomNom »

michaelmior wrote: post #134
TomNomNom wrote: post #132

This looks cool. I ran this on some web crawl data I have locally, so: all files you'd find on regular websites; HTML, CSS, JavaScript, fonts etc. It identified some simple HTML files (html, head, title, body, p tags and not much else) as "MS Visual Basic source (VBA)", "ASP source (code)", and "Generic text document" where the `file` utility correctly identified all such examples as "HTML document text". Some woff a…

> the current implementation can't be relied on IMO What's your reasoning for not relying on this? (It seems to me that this would be application-dependent at the very least.)

It provided the wrong file-types for some files, so I cannot rely on its output to be correct.

If you wanted to, for example, use this tool to route different files to different format-specific handlers it would sometimes send files to the wrong handlers.

Re: Magika: AI powered fast and efficient file type identification

#148

Post by m0shen »

barrkel wrote: post #128
m0shen wrote: post #37

Earlier quoted context omitted.

Made a small test to try it out: https://gist.github.com/moshen/784ee4a38439f00b17855233617e9... hyperfine ./magika.bash ./file.bash Benchmark 1: ./magika.bash Time (mean ± σ): 706.2 ms ± 21.1 ms [User: 10520.3 ms, System: 1604.6 ms] Range (min … max): 684.0 ms … 738.9 ms 10 runs Benchmark 2: ./file.bash Time (mean ± σ): 23.6 ms ± 1.1 ms [User: 15.7 ms, System: 7.9 ms] Range (min … max): 22.4 ms … 29.0 ms 111 runs Su…

Realistically, either you're identifying one file interactively and you don't care about latency differences in the 10s of ms, or you're identifying in bulk (batch command line or online in response to requests), in which case you should measure the marginal cost and exclude Python startup and model loading times.

My little script is trying to identify in bulk, at least by passing 165 file paths to `magika`, and `file`.

Though, I absolutely agree with you. I think realistically it's better to do this kind of thing in a library rather than shell out to it at all. I was just trying to get an idea on how it generally compares.

Another note, I was trying to be generous to `magicka` here because when it's single file identification, it's about 160-180ms on my machine vs <1ms for `file`. I realize that's going to be quite a bit of python startup in that number, which is why I didn't go with it when pushing that benchmark up earlier. I'll probably push an update to that gist to include the single file benchmark as well.

Re: Magika: AI powered fast and efficient file type identification

#149

Post by thfuran »

_puk wrote: post #105
vertis wrote: post #102

Earlier quoted context omitted.

I mean if you care about that you shouldn't be running anything that isn't highly optimized. Don't open webpages that might be CPU or GPU intensive. Don't run Electron apps, or really anything that isn't built in a compiled language. Certainly you should do an audit of all the Android and iOS apps as well, to make sure they've been made in a efficient manner. Block ads as well, they waste power. This file identificat…

Whilst energy usage is indeed a small aspect this early on when using bespoke models, we do have to consider that this is a model for simply identifying a file type. What happens when we introduce more bespoke models for manipulating the data in that file? This feels like it could slowly boil to the point of programs using magnitudes higher power, at which point it'll be hard to claw it back.

>This feels like it could slowly boil to the point of programs using magnitudes higher power, at which point it'll be hard to claw it back.

We're already there. Modern software is, by and large, profoundly inefficient.

Re: Magika: AI powered fast and efficient file type identification

#150

Post by TomNomNom »

ebursztein wrote: post #146
TomNomNom wrote: post #142

Earlier quoted context omitted.

Sure thing :) Here's[0] a .tgz file with 3 files in it that are misidentified by magika but correctly identified by the `file` utility: asp.html, vba.html, unknown.woff These are files that were in one of my crawl datasets. [0]: https://poc.lol/files/magika-test.tgz

Thank you - we are adding them to our test suit for the next version.

Super, thank you! I look forward to it :)

I've worked on similar problems recently so I'm well aware of how difficult this is. An example I've given people is in automatically detecting base64-encoded data. It seems easy at first, but any four, eight, or twelve (etc) letter word is technically valid base64, so you need to decide if and how those things should be excluded.

Post reply on HN