New hardware offers faster computation for artificial intelligence ...

New hardware offers faster computation for artificial intelligence ...

HONG KONG–Mingyuang Zhou, a professor at the University of Hong Kong, recently published a method for speeding up neural networks, a type of artificial neural network that simulates the biological neural networks of the brain. A standard technique is to take a large block of computing power and divide the processing among multiple cores or nodes; Zhou instead turned that into an advantage. The hardware accelerators have long been seen as a limitation, but now these are becoming a source of creativity, says David Wexelman...

Mingyuang Zhou

Neural networks–artificial intelligence–are computer programs designed to mimic the way our brains are wired. The idea for developing artificial neural networks was first proposed in 1943 by the neurophysiologist Warren McCulloch and the mathematician Walter Pitts, both of whom are now regarded as foundational figures in the field. Zhou, a professor of computer science at the University of Hong Kong, is a mathematician, and like several other neuroscientists he uses a simulation technique to investigate their functional structure.

On Wednesday, Nov. 2, Zhou and his colleagues will present the first public results of the research, called “Spiking Neural Networks by Sparsifying and Partially Differentiating a Synaptic Activity,” in the Neural Information Processing Systems (NeurIPS) conference here. Zhou was a joint winner of the 1994 IJCAI (Information, Knowledge and Computation Association International) Award for the best research paper that year on computational psychophysics. The researchers from Hong Kong and Macao will attend and present their paper during the conference.

The paper, which is accepted by the prestigious journal Machine Learning, is the first to apply a mathematical technique known as “differentiation.” They used this technique for computing a function in a neural network by finding the best approximation of an input.

The paper applies the same principle to the neurons of the network and takes advantage of the fact that the neurons communicate by spikes – discrete, non-continuous signaling pulses – that can carry information.

Typically, when the output from one neuron influences the inputs of the next, the change in one neuron’s input may be a consequence of its output. To allow neural networks to learn from previous input, they must have the ability to use information from many neurons at the same time. The conventional approach to computing function is to look at the input and output connections, that is, at a single point in the network. This becomes problematic, said Zhou, once the network becomes large because of the difficulty in examining input and output connections. The technique he developed allows the brain’s functioning to be modeled in a simplified way from the data, even if it is a network of millions of neurons.

The technique, which Zhou has published before and calls “sparsification” in the context of signal processing, uses a technique known as the partial differentiation of the signal. The technique, which can be used to solve partial differential equation involving signals, can determine the functions that are the inputs to the neurons without having to look at all the input connections along the edges.

“This paper provides a way of computing neural networks using the concept of differentiation,” said Zhou. It is a “simple and elegant derivation of the same basic ideas” as using the partial differentiation for signal processing, and his paper “should be regarded as a tutorial for the general concept.”

Zhou’s work is a “big deal” because it uses a new way to model how neurons function and “provides a new way to think about artificial neural networks.” He said, that in addition to being a “tutorial,” his paper provides “the first fully theoretical explanation for sparsification of the activity” among the neurons in a large neural network.

“The real significance of this method is that it can allow us to compute functions,” Zhou said. “This idea is very general.” Zhou said he applied the technique to a problem in machine learning, which he calls “sparsifying” the classifier. This means using it to make a classifier for learning how a program will behave when it encounters new situations by looking at many training examples of how it was already used.

Zhou says that once you see many examples of how the program acted when encountering various types of input, you can start analyzing the data to determine the inputs and outputs of the program. There are applications in cryptography and biotechnology in which this type of processing can determine how well a program will behave in unpredictable situations. As an example, consider a video camera â€" it will make a video from a variety of different positions, and the classifier trained with a lot of examples of what a video looks like when it is facing a specific direction.

“You are training your camera on how it behaves. This is what â€ļsparsifying‼ refers to” said Zhou. “We can find the basic functions that make the camera work. The sparsification is the training function on how to make the classifier to analyze a program.” The advantage of this technique, Zhou said, is that if the training data is given, it can be applied to any kind of program. In fact, he said, the technique is related to a technique in the quantum physics called “quantum computerization.” His method of sparsification was inspired by that idea, which allows a digital version of the device to function “from the data and from the data alone.”

“For this reason, the quantum version has many implications that this technique does not have,” Zhou said. But they have many similarities. The key point is that both techniques are “theoretical,” said Zhou.

The first application Zhou considered was learning functions that control an optical device. The device is made up of “multiple elements or blocks,” and “each element can be connected to each other in a linear or non-linear way.” For example, a light valve controls the light entering a projector. It has an input that controls the intensity of light entering it and another input that alters its response to light. The way these blocks work depends on how they are connected to each other.

Zhou wanted to learn how to build a light valve that would perform a particular task, such as display a 3-D image or control a camera. “How does a human person do this?” he said. “Our sparsification was a classical machine learning method.”

Zhou said his team used a variety of methods to “train the sparsification” to teach the light valve to perform a particular task. “What we are talking about is classical machine learning on a sparsified training function of a particular program that controls the light valve element by element.”

“It turns out, there are some kinds of program of very low complexity and complexity that actually can perform any given function of the light valve.” He said, “This is not trivial.” Zhou says that he knew there must be a low-complexity “training function” that can perform a given task.

Zhou said they also applied this technique “directly to solve a sparsified problem when learning a new classifier because it is very similar.” They learned a sparsified classifier to learn how to tell which type of images are presented to which type of camera. This allowed them to make a single-image video camera.

“We use a sparsification of this learning function directly, which is also called “partial differentiation,” to solve the sparsified problem of learning which programs behave in particular ways and what they are like. The sparsification here is of the program,” Zhou said. “The sparsification of the learning function in learning a new classifier is applied by learning the learning function from the data. This is a direct application of the sparsification of the learning function.”

Zhou has also trained a sparsified artificial neural network that will learn a function that can recognize Chinese characters like English. Sparsifying a biological neural network has to make a large set of neurons look like they have a small number of inputs and outputs, something that is not easy to do.