Fix to unscaling/rescaling of bias in QKeras case with alpha != 1
Created by: thesps
For QKeras Dense/Conv layers with alpha != 1
, there is an optimization pass to factorize the scale out of the layer weights, then apply the scale back in an extra layer (ApplyAlpha
) afterwards. This brings the values of the weights back into the range given by the chosen type.
I observed an issue that the biases were being rescaled by the ApplyAlpha
, but they had not been unscaled in the first place. This fix moves the addition of the bias from the Dense/Conv layer into the ApplyAlpha
to be numerically correct. I think adding the bias here rather than also unscaling-then-rescaling it like the weights makes sense to reduce rounding errors.
The operation y = ApplyAlpha(Dense(x))
was doing y = (w / s * x + b) * s != w * x + b
. Now we do y = (w / s * x) * s + b
.