The magnitude of the diagonal elements in neural networks

The weights of completely connected neural networks are usually derived from the sum-of-outer products rule, with zero diagonal in the weight matrix. In this paper, we calculate what the magnitude of the diagonal elements should be in order to obtain a capacity that is linear in the number of neurons, with the proportionality factor chosen by the user of the network. The theoretical results are verified with simulations. We also show how to obtain the optimal magnitude for the diagonal elements and we investigate the number of spurious patterns. We assume several statistical independence conditions. The theoretical calculations are valid in the limit for an infinite number of neurons. The simulations provide results for networks of a finite size.