Simei Gomes Wysoski & Lubica Benuskova
Knowledge Engineering and Discovery Research Institute, Auckland University of Technology, New Zealand
swysoski@aut.ac.nz , lbenusko@aut.ac.nz
 

Wysoski, S.G. & Benuskova, L. (2006, October), Biologically Realistic Neural Networks and Adaptive Visual Information Processing. Bulletin of Applied Computing and Information Technology Vol. 4, Issue 2. ISSN 1176-4120. Retrieved from

ABSTRACT

This work aims to review the basic concepts of biologically realistic neural networks when applied to visual pattern recognition. A new simple model of biologically realistic visual pattern recognition that adaptively learns by example through synaptic plasticity and changes in structure is also presented. This system uses a spiking neural network composed of integrate-and-fire neurons and Hebbian-based learning. The event driven approach used in the implementation optimizes processing speed in order to simulate networks with large number of neurons.

Keywords

Biologically realistic neural networks, spiking neural networks, pattern recognition, adaptive systems.

1. INTRODUCTION

It is an oversimplification to say that it is not possible to artificially emulate the human brain because of the limitations of current computational resources. In reality, the key issue for failing to properly emulate the human way of processing is the existence of many undeciphered details of the brain structure and behaviour.

Briefly, with respect to current knowledge on the brain, so far it is possible to accurately:

• measure electrophysiological properties of a single cell with high accuracy (Kandel, Schwartz, & Jessell, 2000);
• measure the average activity of the neurons located in a certain area of the brain (electroencephalograms and fMRI);
• divide the human brain into compartments and perform a systematic analysis on how they are interconnected. Here we can map and build a flow chart of signal activity, define structural functions for each compartment, and deduce the relationship and dependencies among compartments.

However, little is known about connectivity among neurons, how the neurons behave as an ensemble, and how information is encoded. Attempts to reproduce connectivity of biologically realistic cells have been limited to consider the ensembles of neurons placed homogeneously in a two dimensional array where the connectivity may follow a given criteria (all-to-all connections or neighbouring connections) or be chosen randomly. Some more careful investigations claim the brain is connected through many “small world networks” (Gong & van Leeuwen, 2004). This theory, which is currently under experimental validation, basically suggests that neurons in the brain are strongly interconnected with other neighbouring neurons, while there are fewer connections among neurons located outside of the immediate neighbourhood.

In an attempt to discover what is behind the reasoning of human minds, one of the most challenging aspects for neuroscientists is that current technologies cannot keep track of and measure all the signals used for inter-neural communication, even in a small portion of the brain. If this were possible, it could enable us to accurately understand the emergence of intelligence from an ensemble of neurons. In an attempt to overcome this limitation, a common practice is to complement the experimental brain study with the development of computational models based on neurobiological data and to study their properties by theoretical and simulation methods. To this end there are computational models of information processing with biological relevance - the biologically realistic neural networks. Of particular interest to this work is the modelling of visual information processing to perform visual pattern recognition. 

Thus, in this paper we will first review the basic concepts of biologically realistic neural networks and explore brain-like techniques for information processing. In addition, we will review how the visual system is formed and the emergence of knowledge through experience and structural development. A simple adaptive brain-like model of a visual system is exemplified and its correspondence with traditional computational models is highlighted.

2. BIOLOGICALLY REALISTIC NEURAL NETWORKS

2.1. Definition

Biologically realistic neural networks or brain-like neural networks are networks that have a closer association with what is known about the way brains process information. In a simplified manner, these networks incorporate the following properties of the brain:

1. A neuron as a processing unit. The brain is basically composed of neurons and glial cells. Despite the number of glial cells being 10 to 50 times greater than the number of neurons, the role of information processing is given exclusively to the neurons. For this reason the glial cells are most of the time left aside in computational models;
2. neurons are constituted of 4 parts. Input, trigger, conduction and output (Kandel, Schwartz, & Jessell, 2000) are commonly represented in neuronal models. In Figure 1 we can see a representation of a biological neuron and its respective computational model.
3. Neurons interact using spikes. In biological brains, neurons transmit information through synaptic junctions in complex chemical reactions using neurotransmitters. This operation can be represented in electrical terms by electric pulses (short term high amplitude signals). To imitate this property, in biologically realistic neural networks the transfer of information is done through spikes. The use of spikes brings together the definition of time varying postsynaptic potential, PSP, firing threshold, J, and spike latencies, D , as depicted in Figure 1.
4. Neurons organize in ensembles with divergent and convergent connections. In a very simplified manner, the neurons connect to each other in two basic ways: through divergent and convergent connections. Divergent connection occurs when the output of a neuron is split and is connected to the input of many other neurons. Convergent connections are those where a certain neuron receives input from many neurons. It is with the organization of the neurons in ensembles that functional compartments emerge.

 Figure 1.Left: Representation of biological neuron. Right:  Basic artificial spiking unit.

2.2 Information Processing in Traditional and Biologically Realistic Neural Networks

In many times it is possible to directly associate traditional ways of performing information processing with the operations executed by spiking neural network models. The visual system, for instance, can be modelled using well established image processing techniques that already have a well described mathematical background (Fourier, Wavelets, etc).  Spiking networks can perform similar tasks, however they are often more complex and difficult to evaluate, mainly because all the processing is done based on spike time.

Nevertheless, it has been proven that spiking networks can, similarly to traditional networks, act as a radial basis function, perform weighted sum of temporal code, and be used for universal approximation of continuous functions (Gerstner & Kistler, 2002). Besides which, a single neuron can act as a coincidence detector, what cannot be easily achieved with traditional neural networks. After a detailed comparison, Maass (1998) concluded that the traditional and biologically realistic neural networks are computationally equivalent.

Further analysis showed that any sigmoidal neural networks can be successfully reproduced using almost the same number of spiking neurons, while simulating spiking neurons requires many more sigmoidal units. (Maass, 1998). As a final remark, in addition to the ability of the spiking neural networks to perform complex mathematical operations, being closer to known biology can help discover principles and rules that govern the brain’s way of processing patterns that are yet unknown.

2.3 Types of Spiking Neural Networks (SNN) and Their Usage

SNN have been traditionally used in computational neuroscience, usually in an attempt to model and evaluate the dynamics of neurons and how they interact in an ensemble. In terms of artificial neurons, the Hodgkin-Huxley model (Hodgkin & Huxley, 1952) can be considered to be the pioneering approach describing the action potentials in terms of ion channels and current flow (Nelson & Rinzel, 1995). Further studies expanded this work and revealed the existence of a wide number of ion channels which vary from one neuron to another. Simulation with this type of neuron can be done using simulators like GENESIS and NEURON (Genesis, n.d.; Neuron, n.d.). The use of numerous compartments to describe a neuron has been shown to be very computationally expensive for simulations of networks with large numbers of units, to such a degree that new models have been developed to produce similar behaviour (similar action potentials) at a lower computational cost. As an example of the simplified models, the integrate-and-fire neuron (Gerstner & Kistler, 2002) ultimately has the properties of a single capacitor and the Izhikevich model (Izhikevich, 2003) combines the Hodgkin-Huxley model with the integrate-and-fire model using a two-dimensional system of ordinary differential equations. Table 1 shows the different models of neuronal units.

For engineering applications, on the other hand, a neural network is normally comprised of rather simple linear/non-linear processing elements (Haykin, 1999; Bishop, 2000), where the sigmoidal function is widely used. SNN has often been considered too complex and cumbersome for this task. However, recent discoveries in the information processing capabilities of the brain and technical advances related to massive parallel processing, are bringing back the idea of using biologically realistic neural networks as a  machine learning technique. A recent pioneering work has shown that the primate (including human) visual system can analyse complex natural scenes in about 100-150 ms (Delorme & Thorpe, 2001). Considering the average spiking rate of the neuronal cells in the brain, this result is very impressive. This theory suggests that neurons exchange only one or few spikes before the information processing is completed. Following this work, the authors proposed a multilayer feedforward network of integrate-and-fire neurons that can successfully track and recognize faces in real time (Delorme, Gautrais, van Rullen, & Thorpe, 1999).

Table 1 - Classification of artificial neural network models according to the biological relevance.

2.4 Emergence of Knowledge in Biologically Realistic Neural Networks

Widely accepted as being the most similar to the brain’s way of learning, the Hebbian rule basically consists of strengthening the connections among neurons that are more active and decreasing the importance of less active connections. The Hebbian learning rule is intrinsically unsupervised, as the connection weights are updated based only on the inner activity levels. The learning is not influenced by feedback or back propagation of errors, which typify supervised learning.

Spike-Timing-Dependent Synaptic Plasticity (STDP) is a rule based on Hebbian learning that uses the temporal order of presynaptic and postsynaptic spikes to determine whether a synapse is potentiated or depressed (Song, Miller, & Abbott, 2000). Another variation of the Hebbian learning is the 'BCM' rule that modifies synaptic weights dynamically depending on the current and previous spiking rates (Bienenstock, Cooper, & Munro, 1982). All these works justify the biological relevance of different methods of unsupervised learning and self organization. Doya (1999) further hypothesizes that the cerebellum performs supervised learning, basal ganglia perform reinforcement learning and cerebral cortex unsupervised learning. In the same work, it is emphasized that all three parts of the brain are able to learn rather than being genetically pre-programmed for static behaviours.

3. BIOLOGICAL VISUAL INFORMATION PROCESSING

On the subject of biological approaches for processing incoming information, Hubel and Wiesel (1962) received many awards for their description of the human visual system. Through neurophysiological experiments, they were able to distinguish types of cells that have different neurobiological responses depending on the pattern of light stimulus. They identified the role that the retina has as a contrast filter as well as the existence of direction selective cells in the primary visual cortex (Figure 2). Their results have been widely implemented in biologically realistic image acquisition approaches. The idea of contrast filters and direction selective cells can be considered a feature selection method that holds a close correspondence with traditional ways of image processing, such as wavelets and Gabor filters (Sonka, Hlavac, & Boyle, 1998).

Figure 2. Contrast cells and direction selective cells.

3.1 Learning to Search Visual Patterns: An Example

In this section we will explain an example of a biologically realistic system capable of performing visual pattern recognition. The SNN is composed of 3 layers of a modified version of integrate-and-fire neurons. The neurons have a latency of firing that depends upon the order of spikes received. Each neuron acts as a coincidence detection unit and the postsynaptic potential for neuron i at a time t is calculated as:

where mod [0,1] is the modulation factor, j is the index for the incoming connection and w is the corresponding synaptic weight. The system is implemented based on principles introduced in (Delorme & Thorpe, 2001; Delorme, Gautrais, van Rullen, & Thorpe, 1999; Delorme, Perrinet, & Thorpe, 2001; Thorpe & Gautrais, 1998).

Each layer is composed of neurons that are grouped in two dimensional arrays forming neuronal maps. Connections between layers are purely feedforward and each neuron can spike at most once when an external source (image) excites the input neurons. The first layer represents cells of the retina, basically enhancing the high contrast parts of the input image. The output values of the first layer are encoded to spikes in the time domain. High amplitude values of the output of the first layer are translated as a spike with a low time delay while low amplitude output values generate spikes with higher delays. This technique is called Rank Order Coding (Thorpe & Gautrais, 1998) and basically prioritizes the pixels with high contrast which are consequently processed first.

The second layer is composed of orientation maps, each one selective to a different direction (0^o, 45^o, 90^o, 135^o, 180^o, 225^o, 270^o, and 315^o). Each orientation can have several maps sensitive to different frequencies. It is important to notice that in the first two layers there is no learning, so that the structure can be considered simply as time domain encoders and passive filters.

The third layer is where the learning takes place. Maps in the third layer represent the complex cells that are selective to more complex shapes and patterns. For learning, the weights of the third layer are adjusted in such a way that the relative order of firing will maximize the response of a given integrate-and-fire neuron using:

The network can, for instance, learn to discriminate objects, detect and recognize faces, facial expressions, hand postures, etc. The only requirement is the visual pattern needs to have well defined orientations and/or boundaries in order to properly excite the contrast and orientation maps.

During learning, the number of neuronal maps in each category changes according to the on-line learning algorithm proposed in (Wysoski, Benuskova, & Kasabov, 2006). According to this algorithm, neuronal maps can be created or adaptively updated in an on-line manner. There are also inhibitory connections among neuronal maps in the third layer, so that when a neuron fires in a certain map, other maps receive inhibitory spikes in an area centred at the same spatial position. An input pattern belongs to a certain class if a neuron in the corresponding neuronal map spikes first.

After learning is completed (weights trained), the network is configured to process test images. Here, each map in each layer is composed of the same number of neurons as the number of pixels in the input image. The spikes generated when an image is presented to the system are propagated to the second and third layer. A pattern is classified if neurons in one of the maps of the third layer generate output spikes. Further, if output spikes occur in layer 3, the spatial position of the neuron that generated the output spikes can be directly associated with the pattern’s position in the input image. Figure 3 shows the complete network architecture.

Figure 3. Adaptive spiking neural network architecture for visual pattern recognition.

An interesting property of this system is the overall low activity of the neurons. It enables the simulation of networks with large numbers of neurons where only few effectively take part in retrieving information. In this case, the computational performance of the simulation can be optimized using an event driven approach (Delorme, Gautrais, van Rullen, & Thorpe, 1999; Mattia & del Giudice, 2000). In addition, the processing can be interrupted many times before the simulation is completed. Once a single neuron of the output layer (layer 3) reaches the threshold and generate an output spike, the simulation can be finished.

4. DISCUSSION

We have described that biologically realistic neural network models can be used for a wide range of purposes, from the pure modelling of brain signals to industrial applications. Biologically realistic neural networks have some interesting properties that make them unique and interesting to analyze. To mention only a few:  

1. While spiking networks can perform all possible operations associated with traditional neural networks, new ways of connectivity and temporal coding yet to be discovered can bring new insights into the creation of artificial systems with performance closer to the human brain.
2. The event driven approach can speed up simulations due to the intrinsically low level of activity of the networks. The early interruption of the simulation can enable optimization of processing time, i.e., it is possible to optimize the network not only to maximize accuracy but also to minimize information processing time.
3. Suitability for hardware implementation, since the operation of spiking neurons is inherently similar to the operational character of elements the of electronic circuits.

A simple example of network architecture and behaviour is presented that can detect and recognize patterns. In the example, a high degree of sparseness is reached in the final layer, which can ultimately be considered to be maps of “grandmother” cells (Connor, 2005). From the pattern recognition view point it seems to be a waste of allocated memory since most of the neurons are rarely used (for each visual pattern to be recognized a neuronal map or a set of neuronal maps is/are allocated). However, memory usage does not affect processing speed due to the event driven approach applied to the computation.

5. ACKNOWLEDGEMENTS

The work has been supported by the Tertiary Education Commission - NZ (S.G.W.) and the NERF grant X0201 funded by FRST (L.B.). We also acknowledge Prof. Nikola Kasabov for his supervision and guidance.

REFERENCES

Bienenstock, E. L., Cooper, L. N.  & Munro, P. W. (1982). Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex, J. Neurosci., vol. 2, pp. 32-48.

Bishop, C. (2000). Neural networks for pattern recognition, Oxford University Press, Oxford.

Connor, C. E. (2005). Friends and grandmothers, Nature, vol. 435, pp. 1036-1037.

Delorme, A., Gautrais, J., van Rullen, R. & Thorpe, S. (1999). SpikeNet: a simulator for modeling large networks of integrate and fire neurons, Neurocomputing, vol. 26-27, pp. 989-996.

Delorme, A., Perrinet, L., & Thorpe, S.J. (2001). Network of integrate-and-fire neurons using Rank Order Coding B: spike timing dependant plasticity and emergence of orientation selectivity. Neurocomputing, vol. 38-40, pp. 539-545.

Delorme, A. & Thorpe, S. (2001). Face identification using one spike per neuron: resistance to image degradation, Neural Networks, vol. 14, pp. 795-803.

Doya, K. (1999). What are the computations of the cerebellum, the basal ganglia, and the cerebral cortex, Neural Networks, vol. 12, pp. 961-974.

Genesis. http://www.genesis-sim.org/GENESIS/.

Gerstner, W. & Kistler, W. M. (2002). Spiking neuron models, Cambridge Univ. Press, Cambridge, MA.

Gong, P. & van Leeuwen, C. (2004). Evolution to a small-world network with chaotic units, Europhysics Letters, vol. 67, pp. 328-333.

Haykin, S. (1999). Neural networks - a comprehensive foundation, Prentice Hall, Engelwood Cliffs, NJ.

Hodgkin, A. L. & Huxley, A. F. (1952). A quantitative description of membrane current and its application to conduction and excitation in nerve, J. Physiol, vol. 117, pp. 500-544.

Hubel, D.H. & Wiesel, T.N. (1962). Receptive fields, binocular interaction and functional architecture in the cat's visual cortex, J. Physiol, vol. 160, pp. 106-154.

Izhikevich, E. M. (2003). Simple model of spiking neurons. IEEE Trans. of Neural Networks, vol. 14, pp. 1569.

Kandel, E. R., Schwartz, J.H. & Jessell, T.M. (2000). Principles of neural science, ed. 4, McGraw-Hill, New York.

Maass, W. (1998). Computing with spiking neurons, In Maass, W. & Bishop, C.M. (eds), Pulsed neural networks, chap. 2, pp. 55-81, MIT Press, Cambridge, MA.

Mattia, M. & del Giudice, P. (2000). Efficient Event-Driven Simulation of Large Networks of Spiking Neurons and Dynamical Synapses, Neural Computation, vol. 12, pp. 2305-2329.

Nelson, M. & Rinzel, J. (1995). The Hodgkin-Huxley model, In Bower, J.M. & Beeman (eds), The book of Genesis, chap. 4, pp. 27-51, Springer, New York.

Neuron. http://www.neuron.yale.edu/neuron/.

Song, S., Miller, K. D. & Abbott, L. F. (2000). Competitive Hebbian learning through spike-timing-dependent synaptic plasticity, Nature Neurosci., vol. 3, pp. 919-926.

Sonka, M., Hlavac, V. & Boyle, R. (1998). Image processing, analysis, and machine vision, ed. 2.

Thorpe, S. & Gaustrais, J. (1998). Rank Order Coding, In Bower, J. (ed), Computational Neuroscience: Trends in Research, Plenum Press, New York.

Wysoski, S.G., Benuskova, L., Kasabov, N. (2006). On-line learning with structural adaptation in a network of spiking neurons for visual pattern recognition. In: S. Kollias et al (eds), Intl. Conf. Art. Neural Net. - ICANN 2006, Lecture Notes in Computer Science, vol. 4131, pp. 61-70, Springer, Berlin/Heidelberg.

Copyright © 2006 Simei Gomes Wysoski and Lubica Benuskova
The author(s) assign to NACCQ and educational non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The author(s) also grant a non-exclusive licence to NACCQ to publish this document in full on the World Wide Web (prime sites and mirrors) and in printed form within the Bulletin of Applied Computing and Information Technology. Authors retain their individual intellectual property rights.

Home | Issue Index | About BACIT

 Copyright © 2006 NACCQ. Krassie Petrova, Michael Verhaart, Andy Connor and Judith Symonds (Eds.). An Open Access Journal, DOAJ # 11764120