代写COMP5328程序、代写Python编程

日期：2021-11-03 09:38

COMP5328 - Advanced Machine Learning

Assignment 2

Due: 11 November 2021, 23:59PM

This assignment is to be completed in groups of 2 to 3 students. It is worth

25% of your total mark.

Introduction

The objective of this assignment is to build a transition matrix estimator and two

classification algorithms that are robust to label noise.

Three input datasets are given. For each dataset, the training and validation

data contains class-conditional random label noise, whereas the test data is clean.

You need to build at least two different classifiers trained and validated on

the noisy data, which can have a good classification accuracy on the clean test

data. You are required to compare the robustness of the two algorithms to label

noise.

For the first two datasets, the transition matrices are provided. You can directly

use the given transition matrices for designing classifiers that are robust to label

noise.

For the last dataset, the transition matrix is not provided. You are required to

build a transition matrix estimator to estimate the transition matrix. Then,

employ your estimated transition matrix for classification. Your estimated tran-

sition matrix must be included in your final report. Note that to validate the

effectiveness of your transition matrix estimator, you could use your estimator on

the first two datasets and compare your estimation to the given transition matri-

ces. The code contained in tutorial 9 could be a good starting point.

Data prepossessing is allowed, but please remember to clarify and justify it in

the report carefully.

1

1 A Guide to Using the Datasets

Three image datasets with .npz format are provided. You can download them via

canvas.

1.1 Attributes Contained in a Dataset

The following code is used to load a dataset and check the shape of its attributes.

import numpy as np

# Remember to r ep l a c e the $FILE PATH

datase t = np . load ($FILE PATH)

Xtr va l = datase t [ ’ Xtr ’ ]

S t r v a l = datase t [ ’ Str ’ ]

Xts = datase t [ ’ Xts ’ ]

Yts = datase t [ ’ Yts ’ ]

print ( Xtr va l . shape )

print ( S t r v a l . shape )

print ( Xts . shape )

print ( Yts . shape )

1.1.1 Training and validation data

The variable Xtr val contains the features of the training and validation data.

The shape is (n, image shape) where n represents the total number of the in-

stances.

The variable Str val contains the noisy labels of the n instances. The shape

is (n, ). For all datasets, the class set of the noisy labels is {0, 1, 2}.

Note that do not use all the n examples to train your models. You are re-

quired to independently and randomly sample 80% of the n examples to train a

model and use the rest 20% examples to validate the model.

1.1.2 Test data

The variable Xts contains features of the test data. The shape is (m, image shape),

where m represents the total number of the test instances.

2

The variable Yts contains the clean labels of the m instances. The class set

of the clean labels is also {0, 1, 2}.

1.2 Dateset Description

1.2.1 FashionMINIST0.3.npz

Number of the training and validation examples n = 18000.

Number of the test examples m = 3000.

The shape of each example image shape = (28× 28).

The transition matrix T =0.7 0.3 00 0.7 0.3

0.3 0 0.7.

1.2.2 FashionMINIST0.6.npz

Number of the training and validation examples n = 18000.

Number of the test examples m = 3000.

The shape of each example image shape = (28× 28).

The transition matrix T =0.4 0.3 0.30.3 0.4 0.3

0.3 0.3 0.4 .

1.2.3 CIFAR.npz

Number of the training and validation examples n = 15000.

Number of the test examples m = 3000.

The shape of each example image shape = (32× 32× 3).

The transition matrix T is unknown.

3

2 Performance Evaluation

The performance of each classifier will be evaluated with the top-1 accuracy metric,

that is,

top-1 accuracy =

number of correctly classified examples

total number of test examples 100%.

To have a rigorous performance evaluation, you need to train each classifier

at least 10 times with the different training and validation sets gener-

ated by random sampling. Then report both the mean and the standard

derivation of the test accuracy.

3 Tasks

You need to implement at least two label noise robustness classifiers with at

least one not taught in this course and test their performance on the three datasets.

You need to implement an estimator to estimate the transition matrix. The

code must be written in Python 3. You are allowed to use external libraries for

optimization and linear algebraic calculation. If you have any ambiguity about

whether you can use a particular library or a function, please post your question

on canvas or Ed.

3.1 Image Classification with Known Flip Rates

For the first two datasets, the transition matrices are provided. You can directly

use the given transition matrices for designing classifiers that are robust to label

noise. As mentioned in the section 2, for each classifier, you should report the

mean and the standard derivation of the test accuracy.

3.2 Image Classification with Unknown Flip Rates

For the last dataset, Since the transition matrix is not provided, you need to imple-

ment an estimator to estimate the transition matrix. Then use the estimated

transition matrix to build a noise robust classifier. Note that you can use the

provided transition matrices of the first two datasets to validate the effectiveness

of your transition matrix estimator. You need to include your estimated transition

matrix in the final report. You also need to report the mean and the standard

derivation of the test accuracy for each of your designed noise robustness classi-

fiers. Both estimation accuracy of the transition matrix and the test accuracy on

the last dataset contribute to the final mark.

4

3.3 Report

The report should be organized similar to research papers, and should contain the

following sections:

In abstract, you should briefly introduce the topic of this assignment, your

methods, and describe the organization of your report.

In introduction, you should first introduce the problem of learning with

label noise, and then its significance and applications. You should give an

overview of the methods you want to use.

In related work, you are expected to review the main idea of related label

noise methods (including their advantages and disadvantages).

In methods, you should describe the details of your classification models,

including the formulation of the cost functions, the theoretical foundations

or views (if any) of the cost functions, and the optimization methods. You

should describe the details of the transition matrix estimation methods, the-

oretical foundations (if any), and optimization algorithms.

In experiments, you should introduce your experimental setup (e.g., datasets,

algorithms, evaluation metric, etc.). Then, you should show the experimen-

tal results, compare, and analyze your results. If possible, give your personal

reflection or thoughts on these results.

In conclusion, you should summarize your methods, results, and your in-

sights for future work.

In references, you should list all references cited in your report and format-

ted all references in a consistent way.

In appendix, you should provide instructions on how to run your code.

The layout of the report:

Font: Times New Roman; Title: font size 14; Body: font size 12

Length: Ideally 10 to 15 pages - maximum 20 pages

Note: Submissions must be typeset in LaTex using the provided template.

5

4 Submissions

Detailed instructions are as follows:

1. The submission contains two parts: report and source code.

(a) report (a pdf file): the report should include each member’s details

(student id and name).

(b) code (a compressed folder)

i. algorithm (a sub-folder): your code could be multiple files.

ii. data (an empty sub-folder): although two datasets should be inside

the data folder, please do not include them in the zip file. We will

copy those datasets to the data folder when we test the code.

2. The report (file type: pdf) and the codes (file type: zip) must be named

as student ID numbers of all group members separated by underscores. For

example, “xxxxxxxx xxxxxxxx xxxxxxxx .pdf”.

3. Only one student needs to submit your report (file type: pdf) to Assignment

1 (report) and upload your codes (file type: zip) to Assignment 1 (codes).

4. Your submission should include the report and the code. A plagiarism

checker will be used.

5. You need to clearly provide instructions on how to run your code in the

appendix of the report.

6. Indicate the contribution of each group member.

7. A penalty of minus 5% marks per each day after due (email late submissions

to TA and confirm late submission dates with TA). The maximum delay is

10 days, after that assignments will not be accepted.

8. Remember, the submission deadline is 11 November 2021, 23:59PM.

6

5 Marking scheme

Category Criterion Marks Comments

Report [80]

Abstract [3]

problem, methods, and organization

Introduction [6]

the problem you intend to solve

the importance of the problem

Previous work [8]

previous relevant methods used in literature

their advantages and disadvantages

Label noise methods with known flip

rates [23]

pre-processing (if any)

label noise methods’ formulation

cross-validation method for model selection

or avoiding overfitting (if any)

experiments

discussions

Noise rate estimation method [12]

noise rate estimation method’s formulation

experiments

discussions

Label noise methods with unknown flip

rates [10]

pre-processing (if any)

label noise methods’ formulation (if different

from above)

cross-validation method for model selection

or avoiding overfitting (if any)

experiments

discussions

Conclusions and future work [3]

meaningful conclusions based on the results

meaningful future work suggested

7

Presentation [8]

academic style, grammatical sentences, no

spelling mistakes

good structure and layout, consistent format-

ting

appropriate citation and referencing

use graphs and tables to summarize data

Other [7]

at the discretion of the assessor: illustrate

outstanding comprehensive theoretical analy-

sis, demonstrate the insightful and compre-

hensive assessment of the significance of their

results, provide descriptions and explanations

that have depth but clarity, and are concisely

worded

Code [20]

reasonable code running time

well organized, commented and documented

Note: Marks for each category is indicated in square brackets. The minimum mark for the assignment will be 0