https://github.com/deeplearning4j/deeplearning4j-examples/blob/b87cb4169d0fb7653427bafa58d0c551a46513d9/dl4j-examples/src/main/java/org/deeplearning4j/examples/advanced/features/externalerrors/MultiLayerNetworkExternalErrors.java#L80
It would be great to have an actual working example rather that this one with random generated error. No PR because I could not make it work, this is my proposal:
- calculate error from model output, maybe this for convergence to 0.4 :
List<INDArray> activations = model.feedForward(true, false);
INDArray output = activations.get(activations.size()-1);
INDArray target = Nd4j.ones(n_batch, nIn).mul(0.4);
INDArray externalError = output.sub(target);
externalError.muli(externalError);
externalError = externalError.mean() // this code suggests that the mean should also be divided by minibatch size https://github.com/deeplearning4j/deeplearning4j/blob/4c22ac5fe4a8350d05d224e7f4499429f7f69c93/deeplearning4j/deeplearning4j-nn/src/main/java/org/deeplearning4j/nn/layers/BaseOutputLayer.java#L83
-
Multiply error by activations, as suggested by
|
* @param epsilon Errors (technically errors .* activations). Not used if withOutputLayer = true |
Pair<Gradient, INDArray> p = model.backpropGradient(error.mul(output), null);
-
The above is not enough to get convergence, other things that might be needed after looking at the source code
a) maybe flatten the gradient before updating model : INDArray g = gradient.gradient(); INDArray fullGrad = g.reshape(g.length());
b) maybe add rather than subtract, since updater has a step of 1 not -1 (Nd4j.getBlasWrapper().level1().axpy(model.params().length(), 1.0, fullGrad, model.params());): model.params().addi(fullGrad);
Again, the above changes are not enough to make the example work.
https://github.com/deeplearning4j/deeplearning4j-examples/blob/b87cb4169d0fb7653427bafa58d0c551a46513d9/dl4j-examples/src/main/java/org/deeplearning4j/examples/advanced/features/externalerrors/MultiLayerNetworkExternalErrors.java#L80
It would be great to have an actual working example rather that this one with random generated error. No PR because I could not make it work, this is my proposal:
Multiply error by activations, as suggested by
deeplearning4j/deeplearning4j/deeplearning4j-nn/src/main/java/org/deeplearning4j/nn/multilayer/MultiLayerNetwork.java
Line 1875 in 4c22ac5
Pair<Gradient, INDArray> p = model.backpropGradient(error.mul(output), null);The above is not enough to get convergence, other things that might be needed after looking at the source code
a) maybe flatten the gradient before updating model :
INDArray g = gradient.gradient(); INDArray fullGrad = g.reshape(g.length());b) maybe add rather than subtract, since updater has a step of 1 not -1 (Nd4j.getBlasWrapper().level1().axpy(model.params().length(), 1.0, fullGrad, model.params());):
model.params().addi(fullGrad);Again, the above changes are not enough to make the example work.