Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Extracting AI models from mobile apps

altayakkus.substack.com

Post reply on HN
Print view View original

201–210 of 250 posts

Re: Extracting AI models from mobile apps

#201

Post by biosboiii »

nthingtohide wrote: post #24

One thing I noticed in Gboard is it uses homeomorphic encryption to do federated learning of common words used amongst public to do encrypted suggestions. E.g. there are two common spelling of bizarre which are popular on Gboard : bizzare and bizarre. Can something similar help in model encryption?

Author here, no clue about homeomorphic (or whatever) encryption, what could certainly be done is some sort of encryption of the model into the inference engine.

So e.g.: Apple CoreML issues a Public Key, the model is encrypted with that Public Key, and somewhere in a trusted computing environment the model is decrypted using a private key, and then inferred.

They should of course use multiple keypairs etc. but in the end this is just another obstacle in your way. When you own the device, root it or even gain JTAG access to it, you can access and control everything.

And matrix-multiplication is a computationally expensive process, in which I guess they won't add some sort of encryption technique for each and every cycle.

Re: Extracting AI models from mobile apps

#202

Post by littlestymaar »

refulgentis wrote: post #178
mentalgear wrote: post #158

Earlier quoted context omitted.

Would be interesting if someone could detail the approach to decode the pre-post processing steps before it enters the model, and how to find the correct input encoding.

Boils down to "use Frida to find the arguments to the TensorFlow call beyond the model file" Key here is, a binary model is just a bag-of-floats with primitively typed inputs and outputs. It's ~impossible to write up more than what's here because either: A) you understand reverse engineering and model basics, and thus the current content is clear you'd use Frida to figure out how the arguments are passed to TensorFlo…

> TL;Dr a request for more content is asking for a reverse engineering article to give you a full education on modal inference

I don't understand what you mean: I have no clue about anything related to reverse engineering, but I ported the mistral tokenizer to Rust and also wrote a basic CPU Llama training and inference implementation in Rust, so I definitely wouldn't need an intro to model inference…

Re: Extracting AI models from mobile apps

#203

Post by hn8726 »

biosboiii wrote: post #199
hn8726 wrote: post #175

Earlier quoted context omitted.

That wouldn't help against the technique explained in the article, would it? Since the model makes it way into the device, it can be intercepted in a similar fashion. I'm not quite sure I understand the firebase feature btw. From the docs, it's pretty much file storage with a dedicated API? I suppose you can use those models for inference in the cloud, but still, the storage API seems redundant.

I think the comment author means offering inference via Firebase, with the model never leaving the backend. This works, just like ChatGPT works, but has the downside of 1. You have to pay the computing for every inference 2. Your users can't access it offline 3. Your users will have to use a lot of data from their mobile network operator. 4. Your inference will be slower And since SeeingAI infers the model every seco…

That's what I thought, but the link doesn't say anything about off-device inference, it's only about storing and retrieving the model. There's just one off-hand note about cloud inference.

In any case, yeah you can not download the model to the device at all, but then you have to deal with the other angle - making sure the endpoint isn't abused.

Maybe a hybrid approach would work - infer just part of the model (layers?) on the cloud, and then carry on the inference on the device? I'm not familiar with how AI models look like and work like exactly, but I feel like hiding even a tiny portion of the model would make it not usable in practice

Re: Extracting AI models from mobile apps

#204

Post by rob_c »

dang wrote: post #186
rob_c wrote: post #157

Earlier quoted context omitted.

Really... Some people do need to be taken down a peg here at times though.

I know it feels that way, but people's perceptions of each other online are so distorted that this is just a recipe for massive conflict. That's off topic on HN because it isn't interesting.

I'm not referring to people's perceptions. Some people write with clearly inflated self worth built into their arguments. If writing style isn't related to rules of writing then we're just welcoming chaos through the back door.

If we're at the point of defending people's literacy as a society than we've fallen into the Orwellian trap of goodspeek.

I'm not insulting people I'm making a demonstrable statement that most people post with a view that they are always correct online. I see it from undergrad work too and it gets shot down there as well for being either just wrong or pretentious and wrong.

Not allowing people's egos to get a needed correction is a bad thing. Using demonstrable right/wrong conversations as a stick to grind other axes however is unacceptable in any context.

People should always approach a topic with an "I am wrong" approach and work backwards to establish that you're not, but almost nobody does, instead wading in with "my trusted source X knows better than you" which is tantamount to "my holy book Y says you should..." Anti-intellectualism at its finest.

Re: Extracting AI models from mobile apps

#205

Post by garyfirestorm »

Lot of comments here seem to think that there’s no novelty. I disagree. As a new ML engineer I am not very familiar with any reverse engineering techniques and this is a good starting point. Something about ML yet it’s simple enough to follow, and my 17yr old cousin who is ambitious to start cyber security would love this article. Maybe its too advanced for him!

Re: Extracting AI models from mobile apps

#206

Post by miki123211 »

biosboiii wrote: post #201
nthingtohide wrote: post #24

One thing I noticed in Gboard is it uses homeomorphic encryption to do federated learning of common words used amongst public to do encrypted suggestions. E.g. there are two common spelling of bizarre which are popular on Gboard : bizzare and bizarre. Can something similar help in model encryption?

Author here, no clue about homeomorphic (or whatever) encryption, what could certainly be done is some sort of encryption of the model into the inference engine. So e.g.: Apple CoreML issues a Public Key, the model is encrypted with that Public Key, and somewhere in a trusted computing environment the model is decrypted using a private key, and then inferred. They should of course use multiple keypairs etc. but in th…

In principle, device manufacturers could make hardware DRM work for ML models.

You usually inference those on GPUs anyway, and they usually have some kind of hardware DRM support for video already.

The way hardware DRM works is that you pass some encrypted content to the GPU and get a blob containing the content key from somewhere, encrypted in a way that only this GPU can decrypt. This way, even if the OS is fully compromised, it never sees the decrypted content.

Re: Extracting AI models from mobile apps

#207

Post by xdennis »

SunlitCat wrote: post #11
JTyQZSnP3cQGa8B wrote: post #3

> Keep in mind that AI models [...] are considered intellectual property Is it ironic or missing a /s? I can't really tell here.

To be honest, that was my first thought on reading that headline as well. Given that especially those large companies (but who knows how smaller ones got their training data) got a huge amount of backlash for their unprecedented collection of data all over the web and not just there but everywhere else, it's kinda ironic to talk about intellectual property. If you use one of those AI model as a basis for your AI mode…

They got backlash, but (if I'm not mistaken) it was ruled that it's okay to use copyrighted works in your model.

So if a model is copyrighted, you should still be able to use it if you generate a different one based on it. I.e. copyright laundry. I assume this would be similar to how fonts work. You can copyright a font file, but not the actual shapes. So if you re-encode the shapes with different points, that's legal.

But, I don't think a model can be copyrighted. Isn't it the case that something created mechanically can't be copyrighted? It has to be authored by a person.

I find it weird that so many hackers go out of their way to approve of the legal claims of Big AI before it's even settled, instead of undermining Big AI. Isn't the hacker ethos all about decentralization?

Re: Extracting AI models from mobile apps

#208

Post by TeMPOraL »

refulgentis wrote: post #178
mentalgear wrote: post #158

Earlier quoted context omitted.

Would be interesting if someone could detail the approach to decode the pre-post processing steps before it enters the model, and how to find the correct input encoding.

Boils down to "use Frida to find the arguments to the TensorFlow call beyond the model file" Key here is, a binary model is just a bag-of-floats with primitively typed inputs and outputs. It's ~impossible to write up more than what's here because either: A) you understand reverse engineering and model basics, and thus the current content is clear you'd use Frida to figure out how the arguments are passed to TensorFlo…

> It's ~impossible to write up more than what's here

Except you just did - or at least you wrote an outline for it, which is 80% of the value already.

Re: Extracting AI models from mobile apps

#209

Post by biosboiii »

garyfirestorm wrote: post #205

Lot of comments here seem to think that there’s no novelty. I disagree. As a new ML engineer I am not very familiar with any reverse engineering techniques and this is a good starting point. Something about ML yet it’s simple enough to follow, and my 17yr old cousin who is ambitious to start cyber security would love this article. Maybe its too advanced for him!

Thanks a lot :)

My general writing style is directed mainly towards my non-technical colleagues, which I wish to inspire to learn about computers.

This is no novelty, by far, it is a pretty standard use-case of Frida. But I think many people, even software developers, don't grasp the concept of "what runs on your device is yours, you just dont have it yet".

Especially in mobile apps, many devs get sloppy on their mobile APIs because you can't just open the developer tools.

Re: Extracting AI models from mobile apps

#210

Post by biosboiii »

miki123211 wrote: post #206
biosboiii wrote: post #201

Earlier quoted context omitted.

Author here, no clue about homeomorphic (or whatever) encryption, what could certainly be done is some sort of encryption of the model into the inference engine. So e.g.: Apple CoreML issues a Public Key, the model is encrypted with that Public Key, and somewhere in a trusted computing environment the model is decrypted using a private key, and then inferred. They should of course use multiple keypairs etc. but in th…

In principle, device manufacturers could make hardware DRM work for ML models. You usually inference those on GPUs anyway, and they usually have some kind of hardware DRM support for video already. The way hardware DRM works is that you pass some encrypted content to the GPU and get a blob containing the content key from somewhere, encrypted in a way that only this GPU can decrypt. This way, even if the OS is fully c…

But then you could compromise the GPU, probably :)

Look at the bootloader, can you open a console?

If not, can you desolder the flash and read the key?

If not, can you access the bootloader when the flash is not detected anymore?

...

Can you solder off the capacitors and glitch the power line, to do a [Voltage Fault Injection](https://www.synacktiv.com/en/publications/how-to-voltage-fau...)?

Can you solder a shunt resistor to the power line, observe the fluctuations and do [Power analysis](https://en.wikipedia.org/wiki/Power_analysis)?

There are a lot of doors and every time someone closes them a window remains tilted.

Post reply on HN