“AI” demystified: a decompiler for “artificial neural networks” • Hacker News Board

Re: “AI” demystified: a decompiler for “artificial neural networks”

#11

Post by r-zip » Fri, Sep 03, 2021, 4:10 PM UTC

Shamar wrote: post #3

heyitsguay wrote: post #2

So there's some arguing over terminology, and then the main technical point seems to be that you can reverse-engineer a training dataset from the "virtual machine" built by training a neural network. The decompilation process doesn't just use the neural network though, if I understand correctly it also uses logs from the final training epoch that include error and weight update data. Does this somehow smuggle the tra…

Beyond the global output and error of each sample from the last epoch, the log also includes the weight update of one single (fully connected) node for each layer. During the compilation phase, the training dataset is projected on a complex vector space that is constituted by both the "model" of the "neural network" and these logs. It's just like projecting a shadow over a bidimensional surface: if you discard the da…

The last iteration (or epoch) of SGD is not shipped with the trained model. The point just does not stand. There are other (better) arguments for why such models are derivative works.

Basically the argument starts with a claim (you can reconstruct the training set of model X from its weights alone) and then shows something totally different. Of course you can reconstruct from the gradient updates plus the weights—that's not interesting, nor does it support the claim.

Re: “AI” demystified: a decompiler for “artificial neural networks”

#12

Post by Shamar » Fri, Sep 03, 2021, 4:59 PM UTC

r-zip wrote: post #10

heyitsguay wrote: post #2

So there's some arguing over terminology, and then the main technical point seems to be that you can reverse-engineer a training dataset from the "virtual machine" built by training a neural network. The decompilation process doesn't just use the neural network though, if I understand correctly it also uses logs from the final training epoch that include error and weight update data. Does this somehow smuggle the tra…

Right. While I appreciate the author's skepticism and diction (there is a lot of misleading terminology thrown around by the ML community), his points don't land. In particular, he argues that there's no learning going on, but then says that there is "absorption" of statistical patterns going on. That's just nitpicking over semantics—to people in the field, the two phrases mean the same thing. The only difference is…

The term "absorbed" was not for the people in the field, but for people who don't know what folding means.

IMHO it's a better metaphor then "learning", because learning is a _subjective_ experience that everyone does and using that term lead inevitably to anthropomorphisation.

"Absorb" match the insight of filters and pipelines, that can be easily understood from any CS student, any "ML expert", any lawyer and any other citizen.

____

As for the network, my argument is simple: if I get back the source dataset from the executable, I think we can agree that such dataset is projected on the numerical matrices that such executable record.

Now where is the dataset?

You might argue that it is recorded _only_ into the gradients logged there (the gradients applied to one single "neuron" for each "layer"), but if so you could reconstruct the source dataset from the logs alone, and in fact, you cannot. You need both the "model" and those gradients in the correct order (and the encodings of inputs and outputs, obviously).

You might ask: "fine, but how much of the source dataset is projected into the gradients and how much is projected into the model?"

To answer, we need to consider that

- the vector space that constitutes the executable is non-linear (the "model" part) and hierarchical (the vectors of the gradients are not independent neither between layers nor between samples)

- (initialization apart) all the values (and the operative value) that the "model" contains comes from the source dataset

Thus I argue that a substantial portion of the source dataset is contained in the "model".

This does not exclude that another substantial portion of the source dataset is also contained into the few logged gradients!

And in fact I've never stated that the "model" contained the whole source dataset.

But if the portion contained into the "model" was negligible, you would be able to get back the sources from those logged gradients alone with negligible errors.

AFAIK, it is not possible, but if you can, please teach me how! I'm always more than happy to be proven wrong if I can learn how to do something that I previously thought impossible!

Re: “AI” demystified: a decompiler for “artificial neural networks”

#13

Post by Shamar » Sat, Sep 04, 2021, 6:37 PM UTC

heyitsguay wrote: post #2

So there's some arguing over terminology, and then the main technical point seems to be that you can reverse-engineer a training dataset from the "virtual machine" built by training a neural network. The decompilation process doesn't just use the neural network though, if I understand correctly it also uses logs from the final training epoch that include error and weight update data. Does this somehow smuggle the tra…

> Does this somehow smuggle the training dataset back into the VM?

Turns out you were right about this: http://www.tesio.it/2021/09/01/a_decompiler_for_artificial_n...

Obviously I was not aware of this, so the whole decompilation process was a waste of computation time, but it doesn't prove nor disprove anything about the "model"'s relation with the source dataset.