Constraint programming in spiking neural networks

Spiking Neural Networks (SNNs) are a class of event-driven and low-power Artificial Neural Networks which aim to closely mimic the computational dynamics which are observed in biological nervous systems. These networks employ different architectural designs and computational characteristics, when compared to the more commonly used Analogue Neural Networks (ANNs). As a consequence, existing tried-and-true Machine Learning methods which have proven effective for training ANNs may not directly work on SNNs, but require new interpretations or approximations to be applicable. It is currently still unclear how SNNs can deliver on the promises of high-performance computing at reduced energy costs.

The work in this thesis addresses the problem of efficiently training SNNs on traditional von Neumann hardware platforms. Although supervised learning rules that allow SNNs to learn spatio-temporal spike-pattern mappings have been developed and studied for a variety of problem domains, the computational paradigm of these methods can be broadly categorised into iterative or one-batch methods, with their own advantages and limitations. The research conducted here aims to combine computational properties from both of these two families of methods, in order to derive hybrid learning algorithms which exhibit improved learning efficiency.

First, we introduce a novel learning rule for supervised training of single-layer SNNs to solve precise input-output spike train mapping problems. This algorithm first converts the learning task into the form of a Constrained Satisfaction Problem (CSP), with the aim of computing the precise \emph{step size} with which the spike-mapping problem can be solved with a single update step. In practice, the constraints of performing computation in continuous time means that the method still require a number of updates to converge, however the required number of learning iterations will be several times fewer than with traditional iterative learning. We will show that the proposed algorithm is viable and efficient through extensive numerical simulations.

Next, we apply the proposed learning rule to supervised learning tasks with spike-count encoding, in which only the number of output spikes are specified and not their exact timings. Encoding the network output with a spike-count have become the norm in classification tasks, since it has reduced computational requirements for inference. Here, our algorithm demonstrates competitive generalisation accuracy and improved convergence speed on common data classification benchmarks, in comparison to existing methods in the literature. Additionally, we perform an experiment in order to measure the maximal learning capacity of the algorithm in spike-count learning problems.

Finally, we present an extension of the proposed algorithm to perform unsupervised feature extraction in networks with convolutional layers. When used sequentially with the original algorithm, we are able to partially address the main weakness of the CSP weight update approach, which is its inability to train multi-layer architectures. The application of our algorithms to the convolutional network architecture is examined in depth, highlighting some of their strengths and weaknesses. Using these findings, we apply the methods to three well-known image classification benchmark problems.