Activity 17: Neural Networks

🕑04:41, 28 Nov 2019

Function fitter

For this activity [1], we will first explore the applications of a neural network---or more accurately, a multilayer perceptron (MLP)---as a universal function fitter. Designing a neural network architecture can be quite tedious as there are a lot of hyperparameters to tune in order for you to get desired results. After several attempts of trial-and-error, I ended up with the network shown in Fig. 1 (all network visualizations were done using [2]). For the training data, I generated 1000 random numbers distributed uniformly in the range as the input, and the corresponding function as the output. Specifically, the functions used were:

• linear:
• quadratic:
• cubic:
• logistic sigmoid:

The first hidden layer of the network contains 500 nodes, which are all activated by a rectified linear unit (ReLU), defined by

The second hidden layer contains 10 nodes, also activated by ReLU. The output layer is a single node which maps everything calculated so far to a single output value. Thus, the function arguments (random numbers ) comprise an entire dataset . The network objective is to minimize the loss, defined as

where are the ground truth and predicted outputs, respectively, and is the norm operator. The network is trained until the loss drops below a value of 0.01. I opted to use the loss instead of the sum-of-squares error or mean-squared error because, from experience, the former is more perceptually accurate.

For the test data, I generated 1000 equally-spaced numbers in the range . Figure 2 shows the predictions for each function.

Fruit classification

For this section, I used my extracted color features of bananas, apples, and oranges from previous activities. The architecture I came up with is shown in Fig. 3. The dataset consists of 50 feature vectors for each fruit (total of 150 feature vectors), which were split 80% for training and 20% for testing.

The network was designed such that the first hidden layer contains as many nodes as data points. Layers are added which contain many nodes as the previous layer, until the number of nodes in the current layer is on the order of magnitude of 10¹. The output layer contains as many nodes as the classes. The hidden layers are activated by a ReLU except for the last, which is activated by a logistic sigmoid, defined as

and the output layer is activated by a softmax, defined by

which outputs probabilities such that the sum of the output nodes should equal unity. Thus, for this dataset, the network topology is 120-12-3 (two hidden layers, 3 nodes in output layer).

This time, the network objective is to minimize the mean-squared error (MSE), and the network is trained until it drops below a value of 0.01. Feeding in the test data shows a test accuracy of 100%, and the decision boundaries are shown in Fig. 4. This is not surprising since the fruit feature vectors form distinct clusters and do not overlap.

Now, let’s consider adding more samples to the training/testing sets. Using the fruits dataset from [3], I added more varieties of apples, bananas, and oranges, as well as an additional class of cherries. Figure 5 shows the distribution of their feature vectors in space. Notice now that the clusters occupy more area and are almost touching each other.

With a total of 4752 feature vectors, I split the data 50-50 among the train-test sets. Therefore, the network topology now is 2376-237-23-4 (3 hidden layers, 4 output nodes). Plugging in the test data afterwards yields a test accuracy of 99.96%, and the decision boundaries are shown in Fig. 6.

References

1. M. N. Soriano, A17 - Neural networks (2019).

2. A. LeNail, Publication-ready neural network architecture schematics, Journal of Open Source Software 4(33) 747 (2019).

3. H. Muresan, and M. Oltean, Fruit recognition from images using deep learning. Acta Univ. Sapientiae, Informatica 10(1), 26-42 (2018).

Keywords