New backend development framework
Created by: vloncar
This is the big PR that was months in the making. The general idea behind it was to decouple the internal model representation from the backend-specific requirements and to be able to create new backends that don't behave exactly the same nor even target the same type of processor (the FPGA).
The changes are extensive, and I'll try to briefly summarize them here:
- Layers now have only inputs, outputs and attributes. All information about the layer's state and configuration is stored in the attributes. All weights, variables and data types are attributes and there are mapping views to sort through them. Layers can define expected attributes and can be verified for correctness, or to produce a list of configurable attributes that user can tweak. Note that the list of expected attributes is not complete for every layer ATM (I was lazy).
- All model/layer transformations should happen in the optimizers. To support the many new uses of optimmizers, the framework was expanded to support many new types of optimizers (layer-specific, backend-specific, whole-model, templates, decorators etc).
- Since there is now a lot more optimizers now, a concept of "flows" is introduced. A flow is just an ordered set of optimizers that may depend on other flows. There are flows shared by each backend (related to proper conversion and common optimizations) and each backend provides its own flows, with one being the default. For example, Vivado backend defaults to an
ip
flow that applies all other flows and produces an IP (essentially the same functionality as we have now but broken into smaller chunks that can be controlled and hooked into) - The internal model is initially backend-agnostic and doesn't define anything FPGA-specific. Backend-specific changes (including introductions of new layer attributes, such as the venerable
reuse_factor
) to the layers are applied via the flows/optimizers, which replace the layers, types and variables with their versions that can be converted to C++/HLS. For example, we start from simple objects representing tensors that are turned into stream- or array-based variables and the precision is converted from the starting data type to the specific implementation (ap_
orac_
) - The
function_cpp
andconfig_cpp
have been removed from the layers and replaced with backend-specific optimizers (templates) that contain the same functionality. This is the single biggest change by affected line numbers and it came in the very last commit. - In light of these changes, the code has seen some shuffling.
templates
submodule now only contains the big C++ templates, the accompanying python classes have been moved tobackends
submodule. The folders of this module contain the backend implementations and their optimizers. The three (and the 4th that will come in 2022) FPGA backends share the same ancestorFPGABackend
that defines the common stuff. -
(Not yet implemented) Since we're moving away from using the term
HLS
in the layers/types/variables I would propose we drophls_
from the file names as well, sohls_model.py
becomesgraph.py
andHLSModel
becomesModelGraph
,hls_layers.py
becomes justlayers.py
,hls_attributes.py
becomesattributes.py
,hls_types.py
becomestypes.py
etc
(I'll expand this description over time and also will follow up with a dedicated document describing each feature and how to use it)
Known issues:
- Flows broke the
Optimizers
andSkipOptimizers
feature of the config -
QKeras alpharelated stuff has not been wired correctly, so most of the QKeras models don't work.Fixed as of 35bf771d -
GarNet is broken since it wasn't updated to make use of new featuresFixed as of ea7e1d0f - The new VivadoAccelerator backend has seen very little testing in the new branch since it was merged only this week so I would expect it to break, but shouldn't be too hard to fix
- Probably many, many more issues I forgot to mention. There are TODOs scattered around the code pointing to the fixes/enhancements that will be made soon