Skip to content

Fix to unscaling/rescaling of bias in QKeras case with alpha != 1

Javier Duarte requested to merge github/fork/thesps/qkeras_alpha_fix into master

Created by: thesps

For QKeras Dense/Conv layers with alpha != 1, there is an optimization pass to factorize the scale out of the layer weights, then apply the scale back in an extra layer (ApplyAlpha) afterwards. This brings the values of the weights back into the range given by the chosen type.

I observed an issue that the biases were being rescaled by the ApplyAlpha, but they had not been unscaled in the first place. This fix moves the addition of the bias from the Dense/Conv layer into the ApplyAlpha to be numerically correct. I think adding the bias here rather than also unscaling-then-rescaling it like the weights makes sense to reduce rounding errors.

The operation y = ApplyAlpha(Dense(x)) was doing y = (w / s * x + b) * s != w * x + b. Now we do y = (w / s * x) * s + b.

Merge request reports

Loading