If Neural Networks Are Allowed To Sleep And Dream, Their Performance Sensibly Increases

The harmonic oscillator for associative memory and pattern recognition in Artificial Intelligence is certainly the Hopfield model [1] (or, equivalently [2], its dual representation, i.e. the Restricted Boltzmann Machine (RBM) [3]). In a nutshell, we can store information (consisting in a set of P digital words or -generally speaking- patterns of information) by suitably modifying the synaptic interactions among neurons in the Hopfield neural network by means of the so-called Hebbian learning (or by using contrastive divergence algorithms for training RBMs).

However, since the seminal AGS theory [5], we know that -for this model- the critical capacity Ī±c (namely the maximal amount of patterns Pmax a network equipped with N neurons can handle, i.e. Ī±c = Pmax/N), is Ī±c āˆ¼ 0.14. On the other side, general information theoretical considerations (i.e., Gardner Theory [4]) prescribe that the maximal critical capacity for networks equipped with symmetric couplings (as the Hopfield model) is precisely Ī±c = 1 (i.e. Pmax = N), significantly better than the Hopfield case.

Recently, in our work [6], we have shown that forcing the network to sleep -namely, mathematically implementing dreaming processes (mimicking slow wave sleep (SWS) and random eye movements (REM) phases in real mammalā€™s brain), after a proper rest the Hopfield model has impressive enhanced skills. In particular, it reaches the maximal bound for the critical storage, i.e. Ī±c = 1, further, stored patterns are very stable configurations (technically speaking, they are free energy absolute minima) up to Ī± āˆ¼ 0.85 (while in standard Hopfield picture, this is true solely for Ī± < 0.05). This is a remarkable property as, in the long time limit, any stochastic neural dynamics for the standard Hopfield model, if Ī± > 0.05, almost surely ends up in a (spin-glass) state that does not overlap significantly with any of the stored patterns; while -if the network can sleep- this happens only for Ī± > 0.85 conferring to networkā€™s functionalities an enhanced robustness with respect to severe loads of patterns.

Finally, thanks to the equivalence between RBMs and Hopfield networks [2], we know that the ratio Ī± = P/N in the Hopfield retrieval matches the ratio among the size of the hidden layer (built of P hidden neurons) over that of the visible one (build of N visible neurons) for RBMs. In the standard scenario, if the hidden layer is larger than āˆ¼ 0.14 times the visible one, learning can be prone to over-fitting (as its dual Hopfield model would lie already outside the working regime, in the spin-glass phase). Allowing the network to sleep, we can learn without over-fitting even by keeping the two layers roughly of the same size (and this obviously allows to extract much more features from the data-sets presented to the network during the training stage).

Summarizing, we believe that in the process of cognition -while certainly learning and retrieval keep covering a pivotal role – also sleeping is mandatory for Artificial Intelligence, as it is for the biological one.

These findings are described in the article entitled Dreaming neural networks: Forgetting spurious memories and reinforcing pure ones, recently published in the journal Neural Networks.

References:

  1. J.J. Hopfield, Neural networks and physical systems with emergent collective computational abilities, Proc. Natl. Acad. Sci. 79(8):2554, (1982).
  2. A. Barra, et al., On the equivalence of Hopfield networks and Boltzmann machines, Neural Networks 34:1, (2012).
  3. D.H. Ackley, G.E. Hinton, T.J. Sejnowski, A learning algorithm for Boltzmann machines, Cognitive science 9(1):147, (1985).
  4. E. Gardner, The space of interactions in neural network models, J. Phys. A 21(1):257, (1998).
  5. D.J. Amit, H. Gutfreund, H. Sompolinsky, Storing infinite numbers of patterns in a spin-glass model of neural networks, Phys. Rev. Lett. 55(14):1530, (1985).
  6. A. Fachechi, E. Agliari, A. Barra, Dreaming neural networks: forgetting spurious memories and reinforcing pure ones, Neural Networks (2019) in press

About The Author

Adriano Barra

Adriano is a senior researcher in Mathematical Physics at the University of Salento. He specializes in statistical mechanics, artificial intelligence, and biological complexity.

Speak Your Mind!

READ THIS NEXT

Pressure Temperature Chart: Example r22 PT Chart

A pressure-temperature chart gives information on how the temperature and pressure of a substance are related to one another. Pressure and temperature have a relationship with one another, a relationship referred to as Gay-Lussacā€™s Law. Gay Lussacā€™s Law states that for a fixed mass of gas, the pressure of the gas has a directly proportional […]

Thermal Comfort, Built Environments, And Seasonal Outdoor Activities

Why is weather so important in the use of outdoor spaces? Sustainable outdoor built environments play a key role in the lives of residents in urban environments. Therefore, outdoor built environments are important from social, environmental, and economic points of view. A wide range of factors come together to form a sustainable built environment, among […]

The Sunniest Spots On The Moon Are Not Where You’d Expect Them

The 23.4Ā° tilt of the Earth’s rotation axis with respect to the ecliptic plane is the reason for our pronounced seasons on Earth. The ecliptic plane itself is defined by Earth’s orbit around the Sun. Contrary to Earth the lunar spin axis is almost perpendicular (1.5Ā°) to that plane leaving the Moon with no distinct […]

What Is A German Cockroach?

The German cockroachĀ (Blatella germanica) is a species of cockroach under the order Blattodea commonly found all over the world. Related to the common wood termite, the GermanĀ cockroach is the most commonly encountered pest variant of cockroach in the United States, next to the AsianĀ cockroach, American cockroach, and the brown-banded cockroach. The cosmopolitan nature of the […]

If You Burn Them, They Will Return: A Case Of Ant-Plant Interactions In A Fire-Prone Brazilian Ecosystem

Climate changes and anthropogenic disturbances are current threats to biodiversity and ecosystem functioning. Ecosystem health depends not only on its biodiversity per se but also on services and functions played by species and their interactions. Ant-plant interactions are an outstanding model to study ecosystem structure and functioningĀ since they each fill distinct ecological roles. Thus, the […]

How Do Sports Organizations Become Ecologically Conscious?

In order for sport and sports organizations to operate, materials, venues, and athletes must exist.Ā  Typically, this is been done with little regard to the natural environment, as evidenced by the sizable ecological footprint that results from sport organization practices. While some sports organizations have taken some of the responsibility in addressing their past, current, […]

Adopting Renewable Energy And Nuclear Power In Japan

The self-sufficiency rate of energy in Japan (nuclear energy is considered semi-domestic energy) was 7.0% in 2015. This is the worst level in the history for Japan. Japan depends strongly on fossil fuels for its primary energy supply (more than 90% in the time of writing), and most are imported. More than 80% of Japanā€™s […]