Complexity and early stopping
A network uses four inputs, six sigmoid hidden units and three softmax outputs. Every non-input unit has a bias.
Count the fitted weights and biases, showing how the total is obtained.
Explain how a validation set can select the stopping epoch and why its best recorded score is not an independent performance estimate.
Explain how an L^2 weight penalty changes training and why it does not usually produce exact zero weights.
Input to hidden: 4\times 6+6=30. Hidden to output: 6\times 3+3=21. The total is 51 stored trainable parameters. Softmax has a common-logit shift that makes the parameters nonidentifiable, but the stated unconstrained implementation still stores these 51 coefficients. A baseline-constrained parametrisation has a different count.
Track validation loss after epochs of training, retain the parameters at the selected minimum, possibly using a prespecified patience rule, then assess once on an independent test set. The epoch is chosen using validation outcomes, so that minimum is selected and can be optimistic.
Adding \lambda\sum w^2 adds 2\lambda w to the gradient for each penalised weight and discourages large weights. This smooth penalty typically shrinks weights continuously. The kink in an L^1 penalty is what can yield exact zeros. State whether biases are included. Either convention is valid when it is explicit.