Counting for NETtalk

Topic: Networks in practice · more questions on this topic
‹ Sum of squares versus accuracy Complexity and early stopping ›

Verify the 18{,}426 weights of NETtalk. How many weights would a network with 50 hidden units have? If the training set has 5000 patterns and 26 outputs, how many numbers (targets) is it trying to fit, and what does the ratio of weights to targets suggest about overfitting?

Worked answer (click to show)

(p+1)l+(l+1)m=204\times 80+81\times 26=18426.

With l=50 the count becomes 204\times 50+51\times 26=11526. A table with 5000 rows and 26 output entries has 130{,}000 target entries, about seven per weight in the larger network. Those entries are not necessarily independent information, and the count supplies no generalisation guarantee. The dependence structure of the patterns and the outputs, the architecture, and the fitting procedure still matter.