Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Magika: AI powered fast and efficient file type identification

opensource.googleblog.com

Post reply on HN
Print view View original

81–90 of 262 posts

Re: Magika: AI powered fast and efficient file type identification

#81

Post by djxfade »

alephnan wrote: post #78
jpk wrote: post #73

Earlier quoted context omitted.

Do you have a sense of performance in terms of energy use? 2x slower is fine, but is that at the same wattage, or more?

That sounds like a nit / premature optimization. Electricity is cheap. If this is sufficiently or actually important for your org, you should measure it yourself. There are too many variables and factors subject to your org’s hardware.

Totally disagree. Most end users are on laptops and mobile devices these days, not desktop towers. Thus power efficiency is important for battery life. Performance per watt would be an interesting comparison.

Re: Magika: AI powered fast and efficient file type identification

#82

Post by cole-k »

callalex wrote: post #16
semitones wrote: post #7

Is it really common enough for files not to be annotated with a useful/correct file type extension (e.g. .mp3, .txt) that a library like this is needed?

Nothing is ever simple. Even for the most basic .txt files it’s still useful to know what the character encoding is (utf? 8/16? Latin-whatever? etc.) and what the line format is (\n,\cr\lf,\n\lf) as well as determining if some maniac removed all the indentation characters and replaced them with a mystery number of spaces. Then there are all the container formats that have different kinds of formats embedded in them (…

A fun read in service of your first point: https://en.wikipedia.org/wiki/Bush_hid_the_facts

Re: Magika: AI powered fast and efficient file type identification

#83

Post by _vufv »

johnea wrote: post #22

The results of which you'll never be 100% sure are correct...

Other methods use heuristics to guess many filetypes and in the benchmark they show worse performance (in terms of precision). Assuming benchmarks are not biased, the fact that this approach uses AI heuristics instead of hard-coded heuristics shouldn't make it strictly worse.

Re: Magika: AI powered fast and efficient file type identification

#84

Post by thangalin »

My FOSS desktop text editor performs a subset of file type identification using the first 12 bytes, detecting the type quite quickly:

* https://gitlab.com/DaveJarvis/KeenWrite/-/blob/main/src/main...

There's a much larger list of file signatures at:

* https://github.com/veniware/Space-Maker/blob/master/FileSign...

Re: Magika: AI powered fast and efficient file type identification

#85

Post by invernizzi »

m0shen wrote: post #33

As someone that has worked in a space that has to deal with uploaded files for the last few years, and someone who maintains a WASM libmagic Node package ( https://github.com/moshen/wasmagic ) , I have to say I really love seeing new entries into the file type detection space. Though I have to say when looking at the Node module, I don't understand why they released it. Their docs say it's slow: https://github.com/go…

Hello! We wrote the Node library as a first functional version. Its API is already stable, but it's a bit slower than the Python library for two reasons: it loads the model at runtime, and it doesn't do batch lookups, meaning it calls the model for each file. Other than that, it's just as fast for single file lookups, which is the most common usecase.

Re: Magika: AI powered fast and efficient file type identification

#86

Post by vintermann »

s1mon wrote: post #20
thorum wrote: post #5

Supported file types: https://github.com/google/magika/blob/main/docs/supported-co...

It's surprising that there are so many file types that seem relatively common which are missing from this list. There are no raw image file formats. There's nothing for CAD - either source files or neutral files. There's no MIDI files, or any other music creation types. There's no APL, Pascal, COBOL, assembly source file formats etc.

Well, what they used this for at Google was apparently scanning their user's files for things they shouldn't store in the cloud. Probably they don't care much about MIDI.

Re: Magika: AI powered fast and efficient file type identification

#87

Post by vintermann »

lopkeny12ko wrote: post #51

I don't understand why this needs to exist. Isn't file type detection inherently deterministic by nature? A valid tar archive will always have the same first few magic bytes. An ELF binary has a universal ELF magic and header. If the magic is bad, then the file is corrupted and not a valid XYZ file. What's the value in throwing in "heuristics" and probabilistic inference into a process that is black and white by desi…

Consider, it's perfectly possible for a file to fit two or more file formats - polyglot files are a hobby for some people.

And there are also a billion formats that are not uniquely determined by magic bytes. You don't have to go further than text files.

Re: Magika: AI powered fast and efficient file type identification

#88

Post by Vogtinator »

samtheprogram wrote: post #13
rfl890 wrote: post #11

We have had file(1) for years

This is beyond what file is capable of. It’s also mentioned in the third paragraph. RTFA.

FWICT file is more capable, predictable and also faster while being more energy-efficient at the same time.

Re: Magika: AI powered fast and efficient file type identification

#89

Post by true_religion »

djxfade wrote: post #81
alephnan wrote: post #78

Earlier quoted context omitted.

That sounds like a nit / premature optimization. Electricity is cheap. If this is sufficiently or actually important for your org, you should measure it yourself. There are too many variables and factors subject to your org’s hardware.

Totally disagree. Most end users are on laptops and mobile devices these days, not desktop towers. Thus power efficiency is important for battery life. Performance per watt would be an interesting comparison.

What end users are working with arbitrary files that they don’t know the identification of?

This entire use case seems to be one suited for servers handling user media.

Re: Magika: AI powered fast and efficient file type identification

#90

Post by underdeserver »

djxfade wrote: post #81
alephnan wrote: post #78

Earlier quoted context omitted.

That sounds like a nit / premature optimization. Electricity is cheap. If this is sufficiently or actually important for your org, you should measure it yourself. There are too many variables and factors subject to your org’s hardware.

Totally disagree. Most end users are on laptops and mobile devices these days, not desktop towers. Thus power efficiency is important for battery life. Performance per watt would be an interesting comparison.

In general you're right, but I can't think of a single local use for identifying file types by a human on a laptop - at least, one with scale where this matters. It's all going to be SaaS services where people upload stuff.
Post reply on HN