Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Extracting AI models from mobile apps

altayakkus.substack.com

Post reply on HN
Print view View original

221–230 of 250 posts

Re: Extracting AI models from mobile apps

#221

Post by dijksterhuis »

philwelch wrote: post #135
dijksterhuis wrote: post #125

Earlier quoted context omitted.

i weirdly agree with you, but also want to point out that “influenced by the training data” is doing some very heavy lifting there. exactly how the new work is created is important when it comes to derivative works. does it use a copy of the original work to create it, or a vague idea/memory of the original work’s composition? when i make music it’s usually vague memories. i’d argue that LLMs have an encoded represen…

I don’t really think it matters because you can just compare the output to the input and apply the same standard, treating the process between the two as a black box.

did you just call me a black box? :/

not sure how i feel about being reduced down to that as a human being.

Re: Extracting AI models from mobile apps

#222

Post by philwelch »

throwaway2037 wrote: post #196
philwelch wrote: post #112

Earlier quoted context omitted.

No, you’re missing the point of copyright. The point of copyright is to protect an exclusive right to copy, not the right to produce original works influenced by previous works. If an LLM produces original works that are influenced by the training data, that is not a violation of copyright. If it reproduces the training data verbatim, it is.

> The point of copyright is to protect an exclusive right to copy, not the right to produce original works influenced by previous works. As I understand, the definition of "the right to produce original works influenced by previous works" has been a slowly moving target in my lifetime. Think about the effects of the album Paul's Boutique by Beastie Boys. They went wild with sampling and paid very little (zero?) to li…

> I'm pretty sure if an LLM creates Paul's Boutique 2.0 in 2025 using incredible number of samples, then someone cannot sell it (or use it in a YouTube video) without first licensing those samples. I doubt very much someone could just "hide behind" an LLM and claim, "Oh, it is original, but derivative, work, created by an LLM." I doubt courts would allow that.

This isn’t how LLM’s work though. Samples are just that, literal samples that are copied from one work to another verbatim. LLM’s use training data to construct a predictive model of which tokens follow each other. You probably could get an LLM to use samples deliberately if you wanted to, but this isn’t how they typically work.

Regardless, at that point you’re just evaluating the claim of copyright infringement based on the nature of the work itself, which is exactly what I’m advocating, versus presuming that all LLM output is necessarily copyright infringement if any copyrighted material was used in training.

Re: Extracting AI models from mobile apps

#223

Post by philwelch »

dijksterhuis wrote: post #221
philwelch wrote: post #135

Earlier quoted context omitted.

I don’t really think it matters because you can just compare the output to the input and apply the same standard, treating the process between the two as a black box.

did you just call me a black box? :/ not sure how i feel about being reduced down to that as a human being.

As far as I’m concerned you are a black box. Just as I’m a black box from your perspective. In principle I could come over and vivisect your brain if you’d like, but I doubt you’d be interested, and I wouldn’t really want to incur the legal liability even if you were.

Besides, “black box” just means that your internal mental life and cognitive mechanism is opaque to me. It’s not like I’m calling you a p-zombie.

Re: Extracting AI models from mobile apps

#224

Post by bangaladore »

biosboiii wrote: post #210
miki123211 wrote: post #206

Earlier quoted context omitted.

In principle, device manufacturers could make hardware DRM work for ML models. You usually inference those on GPUs anyway, and they usually have some kind of hardware DRM support for video already. The way hardware DRM works is that you pass some encrypted content to the GPU and get a blob containing the content key from somewhere, encrypted in a way that only this GPU can decrypt. This way, even if the OS is fully c…

But then you could compromise the GPU, probably :) Look at the bootloader, can you open a console? If not, can you desolder the flash and read the key? If not, can you access the bootloader when the flash is not detected anymore? ... Can you solder off the capacitors and glitch the power line, to do a [Voltage Fault Injection]( https://www.synacktiv.com/en/publications/how-to-voltage-fau... )? Can you solder a shunt…

Any company serious about building silicon that has keys, wouldn't just be storing them in flash.

Try getting a private key off a TPM. There have been novel attacks, but they are few and far between.

Try getting a key from Apple's trusted enclave (or whatever buzz-word they call it).

Re: Extracting AI models from mobile apps

#225

Post by bangaladore »

Ekaros wrote: post #146

Can you launder AI model by feeding it to some other model or training process? After all that is how it was originally created. So it cannot be any less legal...

To some extent this is how many models are being produced today.

Basically its just a synthetic loop of using a previously developed SOTA (was) model like GPT-4 to train your model.

This can produce models with seemingly similar performance at a smaller size, but to some extent, less bits will be less good.

Re: Extracting AI models from mobile apps

#226

Post by crazygringo »

cmiles74 wrote: post #159
crazygringo wrote: post #95

Earlier quoted context omitted.

Can you elaborate on how it's not comparable? It seems obvious to me that it is -- they both learn and then create -- so what's the difference? If I can hire an employee who draws on knowledge they learned from copyrighted textbooks, why can't I hire an AI which draws on knowledge it learned from copyrighted textbooks? What makes that argument "wacky" in your eyes?

I don't think this question really makes any sense... In my opinion, it's kind of mish-mashing several things together. "Can you elaborate on how it's not comparable?" The process of individual people interacting with their culture is a vastly different process than that used to train large language models. In what ways to you think these processes have anything in common? "It seems obvious to me that it is -- they b…

Obviously, on the inside, the process that a person goes through in learning and creating, and the process that a LLM goes through in learning and creating, is very different. Nobody will dispute that.

But if they're learning from the same kinds of materials, and producing the same kind of output, then obviously the comparison can be made. And your idea that LLM's don't create seems obviously false.

So I have to conclude the two seem comparable, and someone would have to show why different legal principles around copyright ought to apply, when it's a simple question of input/output. Why should it matter if it's a human or algorithm doing the processing, from a copyright perspective? Nothing "wacky" about the question at all.

Re: Extracting AI models from mobile apps

#227

Post by biosboiii »

UnreachableCode wrote: post #216
biosboiii wrote: post #209

Earlier quoted context omitted.

Thanks a lot :) My general writing style is directed mainly towards my non-technical colleagues, which I wish to inspire to learn about computers. This is no novelty, by far, it is a pretty standard use-case of Frida. But I think many people, even software developers, don't grasp the concept of "what runs on your device is yours, you just dont have it yet". Especially in mobile apps, many devs get sloppy on their mob…

I'm a mobile developer and I'm new to using Frida and other such tools. Do you have any tips or reading material on how to use things like Frida?

I think you are starting off from the perfect direction, being a forward-engineer first, and then a reverse-engineer.

The community around Frida is a a) a bit small and b) a bit unorganized/shadowy. You cannot find that many resources, atleast I have not found them.

I would suggest you to use Objection, explore an app, enumerate the classes with android hooking list classes or android hooking search classes, then dynamically watch and unwatch them. That is the quickest way to start, when you start developing your own scripts you can always check out code at https://codeshare.frida.re/.

For everything else join the Frida Telegram chat, most knowledge sits there, I am also there feel free to reach out to @altayakkus

Oh and btw, I would start with Android, even though iOS is fun too, and I would really really suggest getting a rooted phone/emulator. For the Android Studio Emulator you can use rootAVD (GitHub), just install Magisk Frida. Installing the Frida gadget into APKs is a mess which you wont miss when you go root

Re: Extracting AI models from mobile apps

#228

Post by npteljes »

TeMPOraL wrote: post #212
npteljes wrote: post #56

Earlier quoted context omitted.

I think it's both. It's 1. the current, unproven-in-court legal understanding, 2. standard disclaimer to cover OP's ass 3. tongue-in-cheek reference to the prevalent argument that training AI on data, and then offering it via AI is being a parasite on that original data

> reference to the prevalent argument that training AI on data, and then offering it via AI is being a parasite on that original data Prevalent or not, phrased this way it's clear how nonsense it is. The data isn't hurt or destroyed in the process of being trained on, nor does the process deprive the data owners from their data or opportunity to monetize it the way they ordinarily would. The right terms here are "lea…

I think the bad part of it is stripping consent from the original creators, after they published their work. I personally see it as an unfortunate side-effect of change. The artists of the future can create with AI already in mind, but this was not the privilege of the current, and previous generations.

Getting back to "learning from", I think the issue is not the learning part, but the recreation part. AI can churn content to orders of magnitude higher than before, even in the age of Fiverr and other tools-opportunities. This changes the dynamics of the interaction, because previously, it took someone tens of hours to create something, now it takes AI minutes. That is not participating in the same playing field, it's absolutely dominating it, completely changing it. That is something to have feelings about, especially if one's livelihood is also impacted. Data is not destroyed, and neither is its ownership, but people don't usually want the exact thing, they are content with a good enough thing, and this takes away a lot of power from the artists, whose work is the lifeblood of artistic AI in the first place.

So I don't think it's as nonsense as you state it. But I do understand that it's not cut and dry the other way around either. Gatekeeping culture is definitely not a humane thing to do. Culture comes and goes, intermingles, inspires and changes all the time, and people take from it and add to it all the time. Preserving copyright perfectly would neuter it, and slant the landscape even more towards the already powerful.

Re: Extracting AI models from mobile apps

#229

Post by biosboiii »

bangaladore wrote: post #224
biosboiii wrote: post #210

Earlier quoted context omitted.

But then you could compromise the GPU, probably :) Look at the bootloader, can you open a console? If not, can you desolder the flash and read the key? If not, can you access the bootloader when the flash is not detected anymore? ... Can you solder off the capacitors and glitch the power line, to do a [Voltage Fault Injection]( https://www.synacktiv.com/en/publications/how-to-voltage-fau... )? Can you solder a shunt…

Any company serious about building silicon that has keys, wouldn't just be storing them in flash. Try getting a private key off a TPM. There have been novel attacks, but they are few and far between. Try getting a key from Apple's trusted enclave (or whatever buzz-word they call it).

You're right about the TPM, I won't get the key out of it. It's a special ASIC which doesn't even have the silicon gates to give me the key.

But is the TPM doing matrix-mulitiplication at 1.3 Petaflops?

Or are you just sending the encrypted file to the TPM, getting the unencrypted file back from it, which I can intercept, be it on SPI or by gaining higher privileges on the core itself? Just like with this app but down lower?

Whatever core executes the multiplications will be vulnerable by some way or the other, for an motivated attacker which has the proper resources. This is true for every hardware device, but the attack vector of someone jailbreaking a Nintendo Switch by using a electron microscope and a ion-beam miller is neglectable.

If you are that paranoid about AI models being stolen, they are worth it, so some attacker will have enough motivation to power through.

Stealing the private key out of a GPU which allows you steal a lot of valuable AI models is break-once-break-everywhere.

Apple trusted enclave is also just a TPM with other branding, or maybe a HSM dunno.

Re: Extracting AI models from mobile apps

#230

Post by bangaladore »

biosboiii wrote: post #229
bangaladore wrote: post #224

Earlier quoted context omitted.

Any company serious about building silicon that has keys, wouldn't just be storing them in flash. Try getting a private key off a TPM. There have been novel attacks, but they are few and far between. Try getting a key from Apple's trusted enclave (or whatever buzz-word they call it).

You're right about the TPM, I won't get the key out of it. It's a special ASIC which doesn't even have the silicon gates to give me the key. But is the TPM doing matrix-mulitiplication at 1.3 Petaflops? Or are you just sending the encrypted file to the TPM, getting the unencrypted file back from it, which I can intercept, be it on SPI or by gaining higher privileges on the core itself? Just like with this app but dow…

I'll concede you are correct that whether the key is extractable or not doesn't really matter if the GPU eventually will eventually need to store the decrypted model in memory.

However, if NVidia or similar was serious about securing these models, I'd be pretty sure they could integrate the crypto in hardware multipliers / etc such that the model doesn't need to be decrypted anywhere in memory.

But at this point there isn't much value in deploying models to the edge. Particularly the type of models they would really want to protect as they are too large.

The types of models deployed to edge devices (like the Apple ones) are generally quite small and frankly not too difficult (computationally) to reimplement.

Post reply on HN