Implement InplaceVariables with a reference
Created by: jmitrevs
This is an alternate of #469 to fix the issues caused by InplaceVariable
s and optimzers removing their reference. This is how I see it:
- The implementation in the master branch creates the reference of an
InplaceVariable
when it is created. This runs into problems with optimizers changing the graph and making the reference no longer valid. Adding hooks toremove_node
andreplace_node
seemed too much machinery for variables that are used only inReshape
layers. - The fix in #469 completely removes
InplaceVariable
s and changes the shape of the preceding node's output. However, the cnn_mnist test showed that this isn't really safe to do, becausefunction_cpp
calls often make use of the output variable shape, so changing the shape breaks the code generation. - This attempt tries to localize the
InplaceVariable
code without adding more machinery.InplaceVariable
s become more like regular variables, but they have a blankdefinition_cpp
. Instead theReshape
layer gets afunction_cpp
that makes the connection between the input at output variable at that stage, after the optimzers have run. Thefunction_cpp
basically becomes a typedef for the type and defines the output variable to be a referece to the input variable. Note, unlike the previous option, this allows for the shapes to differ while effectively becoming a nop in the code. Of course inio_stream
cases theReshape
is often changed to aRepack
node. This PR has no effect on that case.
Both the new test_reshape
and the existing test_cnn_mnist
tests succeed.
Let me know what you think. I left #469 open because I am not sure what is the best direction, though of course this and that are mutually exclusive.