ADVERTISEMENT

If Neural Networks Are Allowed To Sleep And Dream, Their Performance Sensibly Increases

The harmonic oscillator for associative memory and pattern recognition in Artificial Intelligence is certainly the Hopfield model [1] (or, equivalently [2], its dual representation, i.e. the Restricted Boltzmann Machine (RBM) [3]). In a nutshell, we can store information (consisting in a set of P digital words or -generally speaking- patterns of information) by suitably modifying the synaptic interactions among neurons in the Hopfield neural network by means of the so-called Hebbian learning (or by using contrastive divergence algorithms for training RBMs).

However, since the seminal AGS theory [5], we know that -for this model- the critical capacity Ī±c (namely the maximal amount of patterns Pmax a network equipped with N neurons can handle, i.e. Ī±c = Pmax/N), is Ī±c āˆ¼ 0.14. On the other side, general information theoretical considerations (i.e., Gardner Theory [4]) prescribe that the maximal critical capacity for networks equipped with symmetric couplings (as the Hopfield model) is precisely Ī±c = 1 (i.e. Pmax = N), significantly better than the Hopfield case.

ADVERTISEMENT

Recently, in our work [6], we have shown that forcing the network to sleep -namely, mathematically implementing dreaming processes (mimicking slow wave sleep (SWS) and random eye movements (REM) phases in real mammalā€™s brain), after a proper rest the Hopfield model has impressive enhanced skills. In particular, it reaches the maximal bound for the critical storage, i.e. Ī±c = 1, further, stored patterns are very stable configurations (technically speaking, they are free energy absolute minima) up to Ī± āˆ¼ 0.85 (while in standard Hopfield picture, this is true solely for Ī± < 0.05). This is a remarkable property as, in the long time limit, any stochastic neural dynamics for the standard Hopfield model, if Ī± > 0.05, almost surely ends up in a (spin-glass) state that does not overlap significantly with any of the stored patterns; while -if the network can sleep- this happens only for Ī± > 0.85 conferring to networkā€™s functionalities an enhanced robustness with respect to severe loads of patterns.

Finally, thanks to the equivalence between RBMs and Hopfield networks [2], we know that the ratio Ī± = P/N in the Hopfield retrieval matches the ratio among the size of the hidden layer (built of P hidden neurons) over that of the visible one (build of N visible neurons) for RBMs. In the standard scenario, if the hidden layer is larger than āˆ¼ 0.14 times the visible one, learning can be prone to over-fitting (as its dual Hopfield model would lie already outside the working regime, in the spin-glass phase). Allowing the network to sleep, we can learn without over-fitting even by keeping the two layers roughly of the same size (and this obviously allows to extract much more features from the data-sets presented to the network during the training stage).

Summarizing, we believe that in the process of cognition -while certainly learning and retrieval keep covering a pivotal role – also sleeping is mandatory for Artificial Intelligence, as it is for the biological one.

These findings are described in the article entitled Dreaming neural networks: Forgetting spurious memories and reinforcing pure ones, recently published in the journal Neural Networks.

ADVERTISEMENT

References:

  1. J.J. Hopfield, Neural networks and physical systems with emergent collective computational abilities, Proc. Natl. Acad. Sci. 79(8):2554, (1982).
  2. A. Barra, et al., On the equivalence of Hopfield networks and Boltzmann machines, Neural Networks 34:1, (2012).
  3. D.H. Ackley, G.E. Hinton, T.J. Sejnowski, A learning algorithm for Boltzmann machines, Cognitive science 9(1):147, (1985).
  4. E. Gardner, The space of interactions in neural network models, J. Phys. A 21(1):257, (1998).
  5. D.J. Amit, H. Gutfreund, H. Sompolinsky, Storing infinite numbers of patterns in a spin-glass model of neural networks, Phys. Rev. Lett. 55(14):1530, (1985).
  6. A. Fachechi, E. Agliari, A. Barra, Dreaming neural networks: forgetting spurious memories and reinforcing pure ones, Neural Networks (2019) in press

Comments

READ THIS NEXT

Solutions For Pumped Hydro Energy Storage Plants

The safety and stability of the electricity grid is strictly related to the achievement of a perfect balance between supply […]

Functions Of The Golgi Body

TheĀ Golgi body, also sometimes referred to as the Golgi apparatus or Golgi complex, is an intracellularĀ organelle that is responsible for […]

Your Subjective Age Is More Important Than Your Actual Age, Study Finds

Physical and mental health is a factor of subjective age (the age you feel). As one age naturally, the subjective […]

Becoming Picasso Or Gauss: Predictions From Twin ResearchĀ 

Do children who can draw well perform better in math later on? There are indications of this and also of […]

How Chemical Compounds Affect Fruit Bats’ Plant Interactions

Fruit bats are known to be able to discriminate, select, and track the essential oils of their preferred fruits. A […]

Back To The Past For Management Of Large Carnivores In Alaska

In the 19th and early 20th centuries, North American bears,Ā wolves, and mountain lions were viewed as threats to human welfare […]

Those Who Use Alternative Medicines Over Traditional Medicines More Than Twice As Likely To Die Of Cancer

A recent study released in the Journal of the American Medical Associationā€™s JAMA Oncology has reported that cancer patients who […]

Science Trends is a popular source of science news and education around the world. We cover everything from solar power cell technology to climate change to cancer research. We help hundreds of thousands of people every month learn about the world we live in and the latest scientific breakthroughs. Want to know more?