# If Neural Networks Are Allowed To Sleep And Dream, Their Performance Sensibly Increases

The harmonic oscillator for associative memory and pattern recognition in Artificial Intelligence is certainly the Hopfield model [1] (or, equivalently [2], its dual representation, i.e. the Restricted Boltzmann Machine (RBM) [3]). In a nutshell, we can store information (consisting in a set of *P* digital words or -generally speaking- *patterns of information*) by suitably modifying the synaptic interactions among neurons in the Hopfield neural network by means of the so-called Hebbian learning (or by using *contrastive divergence* algorithms for training RBMs).

However, since the seminal AGS theory [5], we know that -for this model- the critical capacity α_{c} (namely the maximal amount of patterns *P _{max}* a network equipped with N neurons can handle, i.e. α

_{c}=

*P*/

_{max}*N)*, is α

_{c}∼ 0.14. On the other side, general information theoretical considerations (i.e., Gardner Theory [4]) prescribe that the maximal critical capacity for networks equipped with symmetric couplings (as the Hopfield model) is precisely α

_{c}= 1 (i.e.

*P*=

_{max}*N*), significantly better than the Hopfield case.

Recently, in our work [6], we have shown that forcing the network to sleep -namely, mathematically implementing dreaming processes (mimicking *slow wave sleep* (SWS) and *random eye movements* (REM) phases in real mammal’s brain), after a proper rest the Hopfield model has impressive enhanced skills. In particular, it reaches the maximal bound for the critical storage, i.e. α_{c} = 1, further, stored patterns are very stable configurations (technically speaking, they are *free energy absolute minima*) up to α ∼ 0.85 (while in standard Hopfield picture, this is true solely for α < 0.05). This is a remarkable property as, in the long time limit, any stochastic neural dynamics for the standard Hopfield model, if α > 0.05, almost surely ends up in a (spin-glass) state that does not overlap significantly with any of the stored patterns; while -if the network can sleep- this happens only for α > 0.85 conferring to network’s functionalities an enhanced robustness with respect to severe loads of patterns.

Finally, thanks to the equivalence between RBMs and Hopfield networks [2], we know that the ratio α = *P/N* in the Hopfield retrieval matches the ratio among the size of the hidden layer (built of P hidden neurons) over that of the visible one (build of N visible neurons) for RBMs. In the standard scenario, if the hidden layer is larger than ∼ 0.14 times the visible one, learning can be prone to over-fitting (as its dual Hopfield model would lie already outside the working regime, in the spin-glass phase). Allowing the network to sleep, we can learn without over-fitting even by keeping the two layers roughly of the same size (and this obviously allows to extract much more features from the data-sets presented to the network during the training stage).

Summarizing, we believe that in the process of *cognition* -while certainly *learning* and *retrieval* keep covering a pivotal role – also *sleeping* is mandatory for Artificial Intelligence, as it is for the biological one.

These findings are described in the article entitled Dreaming neural networks: Forgetting spurious memories and reinforcing pure ones, recently published in the journal *Neural Networks*.

**References:**

- J.J. Hopfield, Neural networks and physical systems with emergent collective computational abilities, Proc. Natl. Acad. Sci. 79(8):2554, (1982).
- A. Barra, et al., On the equivalence of Hopfield networks and Boltzmann machines, Neural Networks 34:1, (2012).
- D.H. Ackley, G.E. Hinton, T.J. Sejnowski, A learning algorithm for Boltzmann machines, Cognitive science 9(1):147, (1985).
- E. Gardner, The space of interactions in neural network models, J. Phys. A 21(1):257, (1998).
- D.J. Amit, H. Gutfreund, H. Sompolinsky, Storing infinite numbers of patterns in a spin-glass model of neural networks, Phys. Rev. Lett. 55(14):1530, (1985).
- A. Fachechi, E. Agliari, A. Barra, Dreaming neural networks: forgetting spurious memories and reinforcing pure ones, Neural Networks (2019) in press