The Balance Between Independent Progress and Alignment of Deep Learning and Neuroscience

 
Callahan Wilde interned with us over the summer—we hope you enjoy this guest post from her:

As an intern at Indico Data as well as a neuroscience major at Connecticut College, the correlation between neuroscience and deep learning is of significant interest to me. I wanted to focus my research with the goal of answering one main question: “is biological inspiration worthwhile to the deep learning community and is deep learning/computer modeling worthwhile to the neuroscience community?” It became clear that there are two separate camps surrounding this question: those who believe that deep learning and biology are useful to each other and interconnected, and those who believe that deep learning is separate and not necessarily related to or benefiting from neuroscience.

After gaining a broad understanding of the two groups’ opinions I decided to focus more narrowly on the following questions:

1. In what ways are neuroscience and deep learning intertwined and how have they benefited one another?
2. Whether neuroscience should be the main avenue for AI inspiration and advancement?
3. Is aligning the two communities mutually beneficial or does it hinder progress?

Through both conducting my research and composing this post I not only gained insight into the workings of the two communities, but also the knowledge that neuroscience and deep learning are two rapidly advancing fields that will continue to improve and impact society.

The interests of the neuroscience and deep learning communities have not always been aligned, and have recently had more limited overlap (Hassabis et al., 2017). The origins of deep learning are heavily based in neuroscience, and as deep learning has progressed it has in turn inspired neuroscience development. Maintaining open communication and “a common language between the two fields would create a virtuous cycle whereby research is accelerated” (Fan, 2017). From an AI specific perspective, it is crucial to scrutinize the human brain as it is the only proof that advanced intelligence exists (Hassabis et al., 2017). The most effective path to success for both communities is to maintain open passage of knowledge, while ensuring that aligning the AI and neuroscience fields does not become burdensome. “By focusing on the computational and algorithmic levels, we gain transferrable insights into general mechanisms of brain function, while leaving room to accommodate the distinctive opportunities and challenges that arise when building intelligent machines in silico” (Hassabis et al. 2017). This comment from Neuroscience Inspired Artificial Intelligence indicates that both communities will experience the most advancement if the universal benchmark of success is progress rather than convergence.

The inception of deep learning was largely inspired by knowledge of how the brain works. The overarching idea that “if a known algorithm is subsequently found to be implemented in the brain then that is strong support for its plausibility as a integral component of an overall general intelligence system” has provided reason to pursue neuroscience inspired AI (Hassabis et al., 2017). Beginning in 1943 Pitts and McCulloch created the first mathematical model of a neural network, the backbone of deep learning. This first neural network was able to successfully use mathematics and algorithms to mimic the human brain (Fogg, 2018). Ten years later psychologist Frank Rosenblatt sought to “construct an electronic or electromechanical system which would learn to recognize similarities or identities between patterns of optical, electrical, or tonal information, in a manner which may be closely analogous to the perceptual processes of a biological brain” (Fog, 2010).

In 1979 another step forward was taken for deep learning when Kunihiko Fukushima developed the first convolutional neural networks, with inspiration from biological neural networks (Foote, 2017). The neural networks were designed with multiple pooling and convolutional layers (Foote, 2017). The networks designed by Fukushima, and now an integral part of deep learning, replicate the organization of the mammalian cortical systems with both convergent and divergent information flow in successive, nested processing layers (Hassabis et al., 2017). The next success for deep learning was back propagation in the late 1980s, which allowed networks to learn non-linear functions (Beam, 2017). Unlike other developments a backpropagation equivalent does not occur in the brain (Bhatia, 2017). In 1999 computers began to process data faster and GPUs were developed, making deep learning algorithms commercially viable for the first time (Foote, 2017).

Source: https://beamandrew.github.io/deeplearning/2017/02/23/deep_learning_101_part1.html

The next 10 years saw continuous improvement and by 2011 “the speed of convolutional neural networks had increased significantly, making it possible to train CNN’s without layer-by-layer training (Foote, 2017). Another area of deep learning which has taken cues from neuroscience is reinforcement learning. The recent use of experience replay buffers for reinforcement learning was inspired by the observation that sequences of pyramidal cell activity are replayed by the hippocampus during sleep and periods of inactivity (Wierzynski, 2018). As concluded by Hassabis et al., once researchers gain more information on the wiring of the brain it is likely that new connectivity motifs will become apparent and inspire even more productive architectural variety in engineered systems (2017).

Collaboration between the neuroscience and deep learning fields not only serves to progress AI, but also provides insights into the workings of biological systems. Deep learning has enabled progress in the biological field due to the ability of algorithms to dive into data in ways that humans can’t, therefore detecting features that would have otherwise been impossible to catch (Webb, 2018). One significant benefit of shared knowledge between the two fields is that neuroscientists are now using brain-inspired algorithms to study the brain itself. One such example is connectomics, a recent effort to map large volumes of the brain at nanometer scale. Given that neurons are huge at this scale there is potential to trace their inputs and outputs (Wierzynski, 2018). Artificial neural networks also provide valuable insights to understanding the brain. Through observing the mechanisms of ANNs in conjunction with the workings of the brain, it was determined that biological systems, like many machine learning systems, are able to optimize cost functions. Neurons in the brain have the ability to change their properties to become better fit for their role, proving that the brain is an optimization machine (Marbleston et al., 2017).

The deep learning and neuroscience communities will experience the most success if a balance is struck. A better understanding of the brain will allow new structures and algorithms to be created for electronic intelligence. In addition, lessons learned from building and testing artificial intelligence models can help to better define what intelligence is (Condliffe, 2017). Given the importance of communication between the two fields it is important to recognize that both deep learning and neuroscience are also separate relevant entities. Success in the deep learning field is obtained regardless of biological connections, as long as it results in better speech recognition, image search and recommendations (Condliffe, 2017). In terms of neuroscience, success comes from a better understanding of the brain and psychiatric disorders as well as methods to treat diseases. The two communities are valuable to each other and the passage of knowledge often causes progress, yet the importance of individual success should not be discarded.

References

1. Beam, A. (2017). Deep learning 101 – part 1: History and background. Machine Learning and Medicine, Retrieved from https://beamandrew.github.io/deeplearning/2017/02/23/deep_learning_101_part1.html
2. Bernini, N. (2017). Artificial neural networks and neuroscience  Towards Data Science, Retrieved from https://towardsdatascience.com/artifician-neural-networks-and-neuroscience-e4852b10d7a9
3. Bhatia, R. (2017). Back-propagation: Is it the achilles heel of today’s AI. Analytics India, Retrieved from https://analyticsindiamag.com/back-propagation-is-it-the-achilles-heel-of-todays-ai/
4. Condliffe, J. (2017). Intelligent machines
   google’s AI guru says that great artificial intelligence must build on neuroscience. MIT Technology Review, Retrieved from https://www.technologyreview.com/s/608317/googles-ai-guru-says-that-great-artificial-intelligence-must-build-on-neuroscience/
5. Fan, S. (2017). Why neuroscience is the key to innovation in AI  Singularity Hub, Retrieved from https://singularityhub.com/2017/08/02/why-neuroscience-is-the-key-to-innovation-in-ai/#sm.0000198kd6zolidvsrwbsgvkm3rxk
6. Fogg, A. (2017). A history of deep learning  Import.Io, Retrieved from https://www.import.io/post/history-of-deep-learning/
7. Foote, K. (2017). A brief history of deep learning  Dataversity, Retrieved from http://www.dataversity.net/brief-history-deep-learning/
8. Hassabis, D., Kumaran, D., Summerfield, C., & Botvinick, M. (2017). Neuroscience-inspired artificial intelligence. Neuron, 95(2), 245-258. 10.1016/j.neuron.2017.06.011 Retrieved from https://www.sciencedirect.com/science/article/pii/S0896627317305093
9. Marblestone, A., Wayne, G., & Kording, K. (2016). Towards and integration of deep learning and neuroscience  Frontiers in Computational Neuroscience
10. Webb, S. (2018). Deep learning for biology  Nature International Journal of Science, Retrieved from https://www.nature.com/articles/d41586-018-02174-z
11. Wierzynski, C. (2018). Deep learning to study the brain to improve deep learning  Intel AI, Retrieved from https://ai.intel.com/deep-learning-study-brain-improve-deep-learning/