Rika
2020-12-07 13:26:13
In DeepLearning literature, an X layer network simply refers to the usage of learnable layers that constitute the representational capacity of the network.
Activation layers, normalization layers (such as NLR, BatchNorm, etc), Downsampling layers (such as Maxpooling, etc) are not considered.
Layers such as CNN, RNN, FC, and the likes that are responsible for the representational capacity of the network are counted.
That is where my question arises. I saw that BatchNorm has trainable parameters in Keras. How does that work? Thank you Rika!
It has learnable parameters gamma and beta that are used specifically for normalization. they do not add to the network's representational capacity, they make training easier! let's put it this way, if you only use BN layers in a network will you learn any features that describe your input data? on the other hand, if you use any form of layers such as CNN, FC, etc in your network, you can indeed reach a set of features that represent your input data in a new form. this is what's considered in the literature.