Skip to content

Example with external loss is nonfunctional #10461

Description

@LanaSina

https://github.com/deeplearning4j/deeplearning4j-examples/blob/b87cb4169d0fb7653427bafa58d0c551a46513d9/dl4j-examples/src/main/java/org/deeplearning4j/examples/advanced/features/externalerrors/MultiLayerNetworkExternalErrors.java#L80

It would be great to have an actual working example rather that this one with random generated error. No PR because I could not make it work, this is my proposal:

  1. calculate error from model output, maybe this for convergence to 0.4 :
List<INDArray> activations = model.feedForward(true, false);
INDArray output = activations.get(activations.size()-1);
INDArray target = Nd4j.ones(n_batch, nIn).mul(0.4);
INDArray externalError = output.sub(target);
externalError.muli(externalError);
externalError = externalError.mean() // this code suggests that the mean should also be divided by minibatch size  https://github.com/deeplearning4j/deeplearning4j/blob/4c22ac5fe4a8350d05d224e7f4499429f7f69c93/deeplearning4j/deeplearning4j-nn/src/main/java/org/deeplearning4j/nn/layers/BaseOutputLayer.java#L83
  1. Multiply error by activations, as suggested by

    * @param epsilon Errors (technically errors .* activations). Not used if withOutputLayer = true

    Pair<Gradient, INDArray> p = model.backpropGradient(error.mul(output), null);

  2. The above is not enough to get convergence, other things that might be needed after looking at the source code
    a) maybe flatten the gradient before updating model : INDArray g = gradient.gradient(); INDArray fullGrad = g.reshape(g.length());
    b) maybe add rather than subtract, since updater has a step of 1 not -1 (Nd4j.getBlasWrapper().level1().axpy(model.params().length(), 1.0, fullGrad, model.params());): model.params().addi(fullGrad);

Again, the above changes are not enough to make the example work.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions