Skip to content

LSTM model with Strategy "resource" results in siginificant AUC loss

Created by: Leopard777

Prerequisites

Please make sure to check off these prerequisites before submitting a bug report.

  • Test that the bug appears on the current version of the master branch. Make sure to include the commit hash of the commit you checked out.
  • Check that the issue hasn't already been reported, by checking the currently open issues.
  • If there are steps to reproduce the problem, make sure to write them down below.
  • If relevant, please include the hls4ml project files, which were created directly before and/or after the bug.

Quick summary

Created an LSTM neural network with Keras which achieves 98.40% AUC score on the test set. However, the HLS-model only achieves 54.75% when the "Strategy" is set to "Resource". If "Strategy" is set to "Latency", the expected AUC scores are received, i.e. 98.40% (Keras) and 98.16% (HLS4ML).

Details

The problem occurs with the latest release (v0.7.0) as with the main branch 51be56ee .

With strategy=resource:

model = load_model('LSTM_32_units_32_seq_length.h5')
model.summary()
prj_name = "hls4ml_LSTM_test"
hls_config = hls4ml.utils.config_from_keras_model(model, granularity="name")

hls_config['Model']['Strategy'] = 'Resource'
hls_config['Model']['ReuseFactor'] = 16384
for layer in hls_config['LayerName'].keys():
    hls_config['LayerName'][layer]['Strategy'] = 'Resource'
    hls_config['LayerName'][layer]['ReuseFactor'] = 16384

print(hls_config)
config = hls4ml.converters.create_config(backend="VivadoAccelerator")
config["HLSConfig"] = hls_config
config["KerasModel"] = model
config["OutputDir"] = prj_name
config["IOType"] = "io_stream"
hls_model = hls4ml.converters.keras_to_hls(config)
print(config)
hls_model.compile()

predictions = model.predict(X_test)
prediction_errors = np.mean(np.abs(Z_test - predictions), axis=1)
print(f"AUC Keras: {roc_auc_score(Y_test, prediction_errors)}")

predictions = hls_model.predict(np.ascontiguousarray(X_test))
prediction_errors = np.mean(np.abs(Z_test - predictions), axis=1)
print(f"AUC HLS4ML: {roc_auc_score(Y_test, prediction_errors)}")

Results:

AUC Keras: 0.9840026507030819
AUC HLS4ML: 0.5475289911373609

With strategy=latency:

model = load_model('LSTM_32_units_32_seq_length.h5')
model.summary()
prj_name = "hls4ml_LSTM_test"
hls_config = hls4ml.utils.config_from_keras_model(model, granularity="name")

hls_config['Model']['Strategy'] = 'Latency'
# hls_config['Model']['ReuseFactor'] = 16384
for layer in hls_config['LayerName'].keys():
    hls_config['LayerName'][layer]['Strategy'] = 'Latency'
    # hls_config['LayerName'][layer]['ReuseFactor'] = 16384

print(hls_config)
config = hls4ml.converters.create_config(backend="VivadoAccelerator")
config["HLSConfig"] = hls_config
config["KerasModel"] = model
config["OutputDir"] = prj_name
config["IOType"] = "io_stream"
hls_model = hls4ml.converters.keras_to_hls(config)
print(config)
hls_model.compile()

predictions = model.predict(X_test)
prediction_errors = np.mean(np.abs(Z_test - predictions), axis=1)
print(f"AUC Keras: {roc_auc_score(Y_test, prediction_errors)}")

predictions = hls_model.predict(np.ascontiguousarray(X_test))
prediction_errors = np.mean(np.abs(Z_test - predictions), axis=1)
print(f"AUC HLS4ML: {roc_auc_score(Y_test, prediction_errors)}")

Results:

AUC Keras: 0.9840026507030819
AUC HLS4ML: 0.9815923809354193

Overview of the LSTM network:

_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_3 (InputLayer)        [(None, 32, 95)]          0         
                                                                 
 lstm_2 (LSTM)               (None, 32)                16384     
                                                                 
 activation_4 (Activation)   (None, 32)                0         
                                                                 
 dense_2 (Dense)             (None, 95)                3135      
                                                                 
 activation_5 (Activation)   (None, 95)                0         
                                                                 
=================================================================
Total params: 19,519
Trainable params: 19,519
Non-trainable params: 0
_________________________________________________________________

Steps to Reproduce

See previous section.

Expected behavior

As I understood the strategy shouldn't influence the AUC score/performance of the model.

Actual behavior

Different strategies result in different AUC scores.