Another noob Question: So a 50% size reduction in BERT? let's see if I am getting these numbers right. At inference time you need a fraction of the neurons in the FF layer to do the inference based on the input data and the previous dot product. Here some quick math for BERT-Base which has 110M params according to the original paper:
----
L (Number of Layers): 12 transformer blocks.
H (Hidden Size): 768 units in the hidden layers.
A (Number of Attention Heads): 12 attention heads.
Embedding Layers:
WordPiece Embeddings: 768 (hidden size) * 30,522 (vocab size) = 23,440,896 parameters.
Positional Embeddings: 768 * 512 (max sequence length) = 393,216 parameters.
Segment Embeddings: 768 * 2 (number of segments) = 1,536 parameters.
Total Embedding Parameters: 23,440,896 + 393,216 + 1,536 = 23,835,648 parameters.
Transformer Blocks:
Each transformer block has the following components:
Self-Attention Layer: Each attention head has 768 / 12 = 64 units.
Query (Q), Key (K), Value (V) matrices: 3 * (64 * 768) = 147,456 parameters per head.
Across 12 heads: 147,456 * 12 = 1,769,472 parameters.
Output layer of the attention mechanism: 768 * 768 = 589,824 parameters.
Feed-Forward Network (FFN):
First layer: 768 (input) * 3,072 (intermediate size) = 2,359,296 parameters.
Second layer: 3,072 * 768 = 2,359,296 parameters.
Total FFN parameters per block: 2,359,296 + 2,359,296 = 4,718,592 parameters. -----------------> *This is the number to keep in mind.*
Total Parameters per Block: 1,769,472 (self-attention) + 589,824 (output) + 4,718,592 (FFN) = 7,077,888 parameters.
Total for 12 Blocks: 7,077,888 * 12 = 84,934,656 parameters.
Layer Norm and Other Parameters:
Each transformer block also includes layer normalization and other small components, which add a relatively small number of parameters.
Total Parameters:
Embeddings: 23,835,648
Transformer Blocks: 84,934,656
Layer Norm and Others: A small number, completing the total to around 110 million.
--------------------------------------
4.718M FF Params per block * 12 ~ 56.6 Million/110M Params which is a staggering ~50% reduction in size at inference time if you use 0.3% of the FF neurons for FFF??