Skip to content

Fix writer precision when fp bits >= 14

Javier Duarte requested to merge github/fork/calad0i/fp_write_length_fix into main

Created by: calad0i

A# Description

In the current version, #decimals to write is determined by

lsb = 2**-new_precision.fractional
decimal_spaces = len(str(lsb).split('.')[1])

However, str(2**-14)='6.103515625e-05', which has 13 instead of 14 digits, leads to bit inaccuracy in some cases.

Indeed, #decimal points required is exactly #fp bits (easy to check: x., x.5, x.x5, x.xx5, ...)

Type of change

  • Bug fix (non-breaking change that fixes an issue)

Tests

test/pytest/test_weight_writer.py

Checklist

  • I have read the guidelines for contributing.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have made corresponding changes to the documentation.
  • My changes generate no new warnings.
  • I have installed and run pre-commit on the files I edited or added.
  • I have added tests that prove my fix is effective or that my feature works.

Merge request reports