MSc/MEng代做、代写Data Mining、MatLab程序语言调试、MatLab代做

日期：2019-12-08 07:37

MSc/MEng Data Mining and Machine Learning (2019)

Lab 4 – Neural Networks

Problem

The challenge is to implement the Error Back-Propagation (EBP) training algorithm for a multilayer

perceptron (MLP) 4-2-4 encoder [1] using MatLab (or your language of choice). Intuitively

the structure of the encoder is as shown in Figure 1.

The MLP has an input layer with 4 units, a single hidden layer with 2 hidden units, and an output

layer with 4 units. Each unit has a sigmoid activation function. The task of the encoder is to map

the following inputs onto outputs:

Input pattern Output pattern

1,0,0,0 1,0,0,0

0,1,0,0 0,1,0,0

0,0,1,0 0,0,1,0

0,0,0,1 0,0,0,1

The problem is that this has to be achieved through the 2-unit “bottle-neck” hidden layer.

Rumelhart, Hinton and Williams demonstrate that to achieve this, the MLP learns binary

encoding in the hidden layer.

Input (and target) pattern

There are 4 input patterns, and the targets are equal to the inputs (Table 1). Recall that the

output ??j of the j

th unit in the network is given by:

???? = ??(????????) =1(1 + ???????????)

where ??????j

is the input to the j

th unit. The values of ??j converge towards 0 and 1 as the

magnitude of ??????j becomes large, but the values 0 and 1 are never realised. Hence for practical

purposes it is better to replace, for example, 1, 0, 0, 0 in Table 1 with 0.9, 0.1, 0.1, 0.1.

Since there are only these 4 input/output pairs, the training set consists of just 4 input/output

pairs.

Structure of the program

The program needs to run the EBP weight updating process multiple times. So you will need a

variable N for the number of iterations and an outer loop (for n=1:1:N). You could

Figure 1: MLP structure for 4-2-4 encoder

Table 1: Input-output pairs for the 4-2-4 encoder

2

terminate the process when the change in error drops below a threshold but this is simpler for

the moment. In addition you will need a second inner loop (for d=1:1:4) to cycle through

the 4 input patterns in each iteration. But before you do this you need to set up some basic

structures:

? You will need two arrays W1 and W2 to store the weights between the input and hidden,

and hidden and output layers, respectively. I suggest that you make W1 4x2 and W2 2x4.

You will need to initialize these arrays (randomly?). Given an output ?? from the input layer,

the input ?? to the hidden layer is given by:

y = W1’*x;

Note the transpose!

? The output from the hidden layer is obtained by applying the sigmoid function to y, so you

will need to write a function to implement this function.

? Once you have propagated the input to the output layer you can calculate the error. In fact

the only use you have for the error is to plot it to help confirm that your code is working.

? Now you need to back-propagate to calculate ??j

for every unit ?? in the output and hidden

layers. First you need to calculate ??j

for every output unit (see equation (12) in the slides).

Then you need to apply back-propagation to calculate ??j

for every hidden unit (again see

equation (12) in the slides). To back-propagate the vector of ??j

s from the output layer to

the hidden layer you just need to multiply by W2 (no transpose this time):

deltaH = W2*deltaO;

where deltaO and deltaH are the deltas in the output and hidden layers, respectively.

Once you have calculated the ????

s for the output and hidden layers you can calculate

???ij = ?????j??i

(see slide 12 from the lecture).

? Finally, you can update the weights: ??ij → ??ij + ???ij

I suggest you do all this about 1,000 times (n=1:1:N, N=1000) and plot the error as a

function of n.

Practical considerations

If you implement all of this properly you will see that the error decreases as a function of n. You

should play with the learning rate, different initialisations of the weight matrices, different

numbers of iterations etc. However, you will find that whatever you do you cannot get the error

to reduce to zero. To do this I think you need to add bias units to the input and hidden layers.

Lab-Report Submission

Submit your source code and comments on experimental evaluations and results obtained.

References

[1] D. E. Rumelhart, G. E. Hinton, and R. J. Williams (1986), “Learning Internal Representations

by Error Propagation”, In: Rumelhart, D.E., McClelland, J.L. and the PDP Research Group, Eds.,

Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. 1:

Foundations, MIT Press, Cambridge, MA, 318-362.