“AI” demystified: a decompiler for “artificial neural networks”
1–10 of 13 posts
Re: “AI” demystified: a decompiler for “artificial neural networks”
#2So there's some arguing over terminology, and then the main technical point seems to be that you can reverse-engineer a training dataset from the "virtual machine" built by training a neural network.
The decompilation process doesn't just use the neural network though, if I understand correctly it also uses logs from the final training epoch that include error and weight update data. Does this somehow smuggle the training dataset back into the VM? To me, if you're making a statement about the nature of existing ML systems, the statement "reconstruct the source dataset from the cryptic matrices that constitute the software executed by them" would imply that this is possible from trained networks alone.
Re: “AI” demystified: a decompiler for “artificial neural networks”
#3So there's some arguing over terminology, and then the main technical point seems to be that you can reverse-engineer a training dataset from the "virtual machine" built by training a neural network. The decompilation process doesn't just use the neural network though, if I understand correctly it also uses logs from the final training epoch that include error and weight update data. Does this somehow smuggle the tra…
Beyond the global output and error of each sample from the last epoch, the log also includes the weight update of one single (fully connected) node for each layer.
During the compilation phase, the training dataset is projected on a complex vector space that is constituted by both the "model" of the "neural network" and these logs.
It's just like projecting a shadow over a bidimensional surface: if you discard the data pertaining to one dimension you have no hope to guess what projected it: you need both dimensions.
The logs that are preserved in the compilation process is the part of the vector space that is usually discarded during the "training".
But discarding the "model" would have exactly the same effect: you cannot get back the source dataset from those logs alone. That's why this does not "smuggle the training dataset back".
Indeed the fact that the source dataset is obtainable from the couple "these logs" + "final model", but neither from "these logs" alone nor by the final model alone, proves that a substantial portion of the source dataset is always embedded in the "model", that becomes a derivative work of the sources.
Re: “AI” demystified: a decompiler for “artificial neural networks”
#4Such a delightful read!
I'm a bit disappointed to see the word "learning" thrown around as if we actually understand what it means. If you can explain it's working in simple mechanical terms, does it become less valuable?
Re: “AI” demystified: a decompiler for “artificial neural networks”
#5[dead]
Re: “AI” demystified: a decompiler for “artificial neural networks”
#6So there's some arguing over terminology, and then the main technical point seems to be that you can reverse-engineer a training dataset from the "virtual machine" built by training a neural network. The decompilation process doesn't just use the neural network though, if I understand correctly it also uses logs from the final training epoch that include error and weight update data. Does this somehow smuggle the tra…
Beyond the global output and error of each sample from the last epoch, the log also includes the weight update of one single (fully connected) node for each layer. During the compilation phase, the training dataset is projected on a complex vector space that is constituted by both the "model" of the "neural network" and these logs. It's just like projecting a shadow over a bidimensional surface: if you discard the da…
This does not prove that the source dataset is embedded in the model. You could do this with a random model and get the same result...
Re: “AI” demystified: a decompiler for “artificial neural networks”
#7Earlier quoted context omitted.
Beyond the global output and error of each sample from the last epoch, the log also includes the weight update of one single (fully connected) node for each layer. During the compilation phase, the training dataset is projected on a complex vector space that is constituted by both the "model" of the "neural network" and these logs. It's just like projecting a shadow over a bidimensional surface: if you discard the da…
This does not prove that the source dataset is embedded in the model. You could do this with a random model and get the same result...
I strongly encourage you to prove your statement with a script that use the logs saved and a random "model" and get back the exact source dataset.
Re: “AI” demystified: a decompiler for “artificial neural networks”
#8Earlier quoted context omitted.
This does not prove that the source dataset is embedded in the model. You could do this with a random model and get the same result...
I strongly encourage you to prove your statement with a script that use the logs saved and a random "model" and get back the exact source dataset.
[deleted]
Re: “AI” demystified: a decompiler for “artificial neural networks”
#9I think of neural networks as "smooth compression" - something that compresses a key-value database into a continuous non-linear function. Previously Daniel Holden wrote about something similar in his blog (https://theorangeduck.com/page/machine-learning-kolmogorov-c...)
I agree with the author's sentiment that "learning" might be a misleading term. Neural networks can just be seen as specialized programs that just compress certain kinds of data extremely well (at the cost of extremely high pre-computation). But it's still a very useful technique to tackling previously intractable problems.
Re: “AI” demystified: a decompiler for “artificial neural networks”
#10So there's some arguing over terminology, and then the main technical point seems to be that you can reverse-engineer a training dataset from the "virtual machine" built by training a neural network. The decompilation process doesn't just use the neural network though, if I understand correctly it also uses logs from the final training epoch that include error and weight update data. Does this somehow smuggle the tra…
Right. While I appreciate the author's skepticism and diction (there is a lot of misleading terminology thrown around by the ML community), his points don't land.
In particular, he argues that there's no learning going on, but then says that there is "absorption" of statistical patterns going on. That's just nitpicking over semantics—to people in the field, the two phrases mean the same thing. The only difference is whether you anthropomorphize a piece of software.
The second place the author stumbles is that he makes the (quite grave) mistake you pointed out. The title insinuates that the network contains the "source dataset" itself. He has shown nothing of the sort by including the training logs in his "decompilation". That's like suggesting you have a Swift decompiler that can recover the exact source code from an optimized binary, but you actually require access to the pre-optimized LLVM IR.