I wonder what the output will be on polyglot files like run-anywhere binaries produced by cosmopolitan [1]
Magika: AI powered fast and efficient file type identification
91–100 of 262 posts
Re: Magika: AI powered fast and efficient file type identification
#92Wonder how this would handle a polyglot[0][1], that is valid as a PDF document, a ZIP archive, and a Bash script that runs a Python webserver, which hosts
Kaitai Struct’s WebIDE which, allowing you to view the file’s own annotated bytes.
[0]: https://www.alchemistowl.org/pocorgtfo/
[1]: https://www.alchemistowl.org/pocorgtfo/pocorgtfo16.pdf
Edit: just tested, and it does only identify the zip layer
Re: Magika: AI powered fast and efficient file type identification
#93Earlier quoted context omitted.
Totally disagree. Most end users are on laptops and mobile devices these days, not desktop towers. Thus power efficiency is important for battery life. Performance per watt would be an interesting comparison.
In general you're right, but I can't think of a single local use for identifying file types by a human on a laptop - at least, one with scale where this matters. It's all going to be SaaS services where people upload stuff.
We are building a data analysis tool with great UX, where users select data files, which are then parsed and uploaded to S3 directly, on their client machines. The server only takes over after this step.
Since the data files can be large, this approach bypasses having to trnasfer the file twice, first to the server, and then to S3 after parsing.
Re: Magika: AI powered fast and efficient file type identification
#94Wonder how this would handle a polyglot[0][1], that is valid as a PDF document, a ZIP archive, and a Bash script that runs a Python webserver, which hosts Kaitai Struct’s WebIDE which, allowing you to view the file’s own annotated bytes. [0]: https://www.alchemistowl.org/pocorgtfo/ [1]: https://www.alchemistowl.org/pocorgtfo/pocorgtfo16.pdf Edit: just tested, and it does only identify the zip layer
You can try it here:
https://google.github.io/magika/
It's relatively limited compared to `file` (~10% coverage), it's more like a specialized classificator for basic file formats, so such cases are really out-of-scope.
I guess it's more for detecting common file formats then with high recall.
However, where is the actual source of the model ? Let's say I want to add a new file format myself.
Apparently only the source of the interpreter is here, not the source of the model nor the training set, which is the most important thing.
Re: Magika: AI powered fast and efficient file type identification
#95Earlier quoted context omitted.
Totally disagree. Most end users are on laptops and mobile devices these days, not desktop towers. Thus power efficiency is important for battery life. Performance per watt would be an interesting comparison.
What end users are working with arbitrary files that they don’t know the identification of? This entire use case seems to be one suited for servers handling user media.
Theoretically? Anyone running a virus scanner.
Of course, it's arguably unlikely a virus scanner would opt for an ML-based approach, as they specifically need to be robust against adversarial inputs.
Re: Magika: AI powered fast and efficient file type identification
#96Wonder how this would handle a polyglot[0][1], that is valid as a PDF document, a ZIP archive, and a Bash script that runs a Python webserver, which hosts Kaitai Struct’s WebIDE which, allowing you to view the file’s own annotated bytes. [0]: https://www.alchemistowl.org/pocorgtfo/ [1]: https://www.alchemistowl.org/pocorgtfo/pocorgtfo16.pdf Edit: just tested, and it does only identify the zip layer
You can try it here: https://google.github.io/magika/ It's relatively limited compared to `file` (~10% coverage), it's more like a specialized classificator for basic file formats, so such cases are really out-of-scope. I guess it's more for detecting common file formats then with high recall. However, where is the actual source of the model ? Let's say I want to add a new file format myself. Apparently only the sour…
Yes, I totally agree; it's not what I would qualify as open source.
Do you plan to release the training code along the research paper? What about the dataset?
In any case, it's very neat to have ML-based technique and lightweight model for such tasks!
Re: Magika: AI powered fast and efficient file type identification
#97As someone that has worked in a space that has to deal with uploaded files for the last few years, and someone who maintains a WASM libmagic Node package ( https://github.com/moshen/wasmagic ) , I have to say I really love seeing new entries into the file type detection space. Though I have to say when looking at the Node module, I don't understand why they released it. Their docs say it's slow: https://github.com/go…
> The model appears to only detect 116 file types [...] Where libmagic detects... a lot. Over 1600 last time I checked
As I'm sure you know, in a lot of applications, you're preparing things for a downstream process which supports far fewer than 1600 file types.
For example, a printer driver might call on file to check if an input is postscript or PDF, to choose the appropriate converter - and for any other format, just reject the input.
Or someone training an ML model to generate Python code might have a load of files they've scraped from the web, but might want to discard anything that isn't Python.
Re: Magika: AI powered fast and efficient file type identification
#98> Magika: AI powered fast and efficient file type identification
of 116 file types with proprietary puny model with no training code and no dataset.
> We are releasing a paper later this year detailing how the Magika model was trained and its performance on large datasets.
And ? How do you advance industry by this googleblog post and source code that is useless without closed source model ? All I see here is loud marketing name, loud promises, but actually barely anything useful. Hooly rooftop characters sideproject?
Re: Magika: AI powered fast and efficient file type identification
#99Earlier quoted context omitted.
Totally disagree. Most end users are on laptops and mobile devices these days, not desktop towers. Thus power efficiency is important for battery life. Performance per watt would be an interesting comparison.
What end users are working with arbitrary files that they don’t know the identification of? This entire use case seems to be one suited for servers handling user media.
Browsers often need to guess a file type
Re: Magika: AI powered fast and efficient file type identification
#100Why is this piece of code being sold as open source, when in reality it just calls into proprietary ML blob that is tiny and useless, and actual source code of model is closed while properly useful large model is non existing ?