DS-UA 202代写、代写Python，Java程序语言

日期：2022-03-01 11:33

DS-UA 202, Responsible Data Science, Spring 2022

Homework 1: Algorithmic Fairness

Due on Thursday, March 3 at 11:59pm EST

Objectives

This assignment consists of written problems and programming exercises on algorithmic

fairness.

After completing this assignment, you will:

● Understand that different notions of fairness correspond to points of view of different

stakeholders, and are often mutually incompatible.

● Gain hands-on experience with incorporating fairness-enhancing interventions into

machine learning pipelines.

● Learn about the trade-offs between fairness and accuracy.

● Observe the effect of hyperparameter tuning on performance, in terms of both accuracy

and fairness.

You must work on this assignment individually. If you have questions about this assignment,

please send an email to all instructors.

Grading

The homework is worth 75 points, or 10% of the course grade. Your grade for the programming

portion (Problem 2) will be significantly impacted by the quality of your written report for that

portion. In your report, you should explain your observations carefully.

You are allotted 2 (two) late days over the term, which you may use on a single homework, or

on two homeworks, or not at all. If an assignment is submitted at most 24 hours late: one day is

used in full. If it’s submitted between 24 and 48 hours late: two days are used in full.

Submission instructions

Provide written answers to Problems 1, 2, and 3 in a single PDF file. (It is recommended that

you Google Docs to prepare this PDF, but you may instead use Word or LaTeX). Provide code

in answer to Problem 2 in a Google Colaboratory notebook. Both the PDF and the notebook

should be turned in as Homework 1 on Brightspace. Please clearly label each part of each

question.

Problem 1 (20 points): Fairness from the point of view of different stakeholders

(a) (5 points) Consider the COMPAS investigation by ProPublica and Northpointe’s response.

(You may also wish to consult Northpointe’s report.) For each metric A-E below, explain in 1-2

sentences which stakeholders would benefit from a model that optimizes that metric, and why.

If you believe that it would not be reasonable to optimize that metric in this case, state so.

A. Accuracy

B. Positive predictive value

C. False positive rate

D. False negative rate

E. Statistical parity (demographic parity among the individuals receiving any prediction)

(b) (6 points) Consider a hypothetical scenario in which TechCorp, a large technology company,

is hiring for data scientist roles. Alex, a recruiter at TechCorp, uses a resume screening tool

called Prophecy to help identify promising candidates. Prophecy takes applicant resumes as

input and returns them in ranked (sorted) order, with the more promising applicants (according

to the tool) appearing closer to the top of the ranked list. Alex takes the output of the Prophecy

tool under advisement when deciding whom to invite for a job interview.

In their 1996 paper “Bias in computer systems”, Friedman & Nissenbaum discuss three types of

bias: A. pre-existing, B. technical, and C. emergent. We also discussed these types of bias in

class and in the “All about that Bias” comic.

For each type of bias:

● Give an example of how this type of bias may arise in the scenario described above;

● Name a stakeholder group that may be harmed by this type of bias; and

● Propose an intervention that may help mitigate this type of bias.

(c) (9 points) Consider a hypothetical scenario in which an admissions officer at Best University

is evaluating applicants based on 3 features: SAT score, high school GPA, and family income

bracket (low, medium, high). We discussed several equality of opportunity (EO) doctrines in

class and in the “Fairness and Friends” comic: formal, substantive / luck egalitarian, and

substantive / Rawlsian.

A. In a selection procedure that is fair according to formal EO, which of these features

would the admissions officer use? Briefly justify your answer.

B. Suppose that income-based differences are observed in applicants’ SAT scores: the

median SAT score is lower for applicants from low-income families, as compared to

those from medium- and high-income families. Which EO doctrine(s) is/are consistent

with the goal of correcting such differences in the applicant pool? Briefly justify your

answer.

C. Describe an applicant selection procedure that is fair according to luck-egalitarian EO.

Problem 2 (40 points): Fairness-enhancing interventions in machine learning pipelines

In this part of the assignment you will use AIF360 to incorporate fairness-enhancing

interventions into binary classification pipelines. You should use the provided Google

Colaboratory notebook as the starting point for your implementation (Note: you must use

Google Colab, not Jupyter Notebooks). Your grade will be based on the quality of your code and

of your report: explain your findings clearly, and illustrate them with plots as appropriate.

Working with the ASCIncome dataset, you will predict high vs low income using individuals’

attributes. We have preloaded the ASCIncome dataset from Folktables. We select sex as the

sensitive attribute to analyze throughout this question.

For the below tasks, split your data into 80% training and 20% test. Report all results on the

withheld test dataset.

You will evaluate performance using the following metrics:

(i) Overall accuracy

(ii) Accuracy for the privileged group

(iii) Accuracy for the unprivileged group

(iv) Disparate Impact

(v) False positive rate difference

(a) (10 points) Train a baseline random forest model to predict income. Use the

hyperparameters in the provided notebook. Report performance on the metrics listed

above on the test set. Discuss your results in the report.

(b) (15 points) Consider Disparate Impact Remover (DI-Remover), a pre-processing

fairness-enhancing intervention by Feldman et al., 2015 (here) that is implemented in

AIF360. This algorithm provides a parameter called the repair level that controls the

trade-off between fairness and accuracy. In this question, you will measure the impact of

the repair level on fairness and accuracy.

Transform the original dataset using DI-Remover with five different values of the

repair level: [0, 0.25, 0.5, 0.75, 1]. Train a random forest model on each transformed

dataset using the same hyperparameters and train/test split proportions that you used in

part (a). Report the same five metrics again for each trained model.

Discuss in your report how these results compare with the metrics from the baseline

random forest model in part (a), paying particular attention to the impact of repair level.

You may wish to plot each metric against the repair level.

(c) (15 points) Train a model using the Prejudice Remover in-processing technique by

Kamishima et al. 2012 (link) that is implemented in AIF360. This algorithm provides a

parameter called eta, which controls the fairness regularization weight. Use the values

[0.01, 0.1, 1] for the eta parameter. Plot both the accuracy and disparate impact as you

adjust this parameter and discuss the results.

Discuss in your report how the effect of the eta parameter compares to what you

observed for DI-Remover. (Remember: Prejudice Remover is not a pre-processing

method that is combined with an existing Random Forest model. It’s a different model

altogether, which fits a Logistic Regression under the hood.)

Conclude your report with any general conclusions about the trends and trade-offs you

observed in the performance of the fairness enhancing interventions with respect to the

accuracy and fairness metrics.

Problem 3 (15 points)

In the final part of the assignment, you will watch a lecture from the AI Ethics: Global

Perspectives course and write a memo (500 words maximum) reflecting on issues of fairness

raised in the lecture. You can watch either:

● “AI for whom?” (watch the lecture)

● “AI Powered Disability Discrimination: How Do You Lipread a Robot Recruiter“ (watch

the lecture)

● “Ethics in AI: A Challenging Task” ( watch the lecture)

● “Alexa vs Alice: Cultural Perspectives on the Impact of AI” (watch the lecture)

Before watching the lecture, please register for the course at

https://aiethicscourse.org/contact.html, specify “student” as your position/title, “New York

University” as your organization, and enter DS-UA 202 in the message box.

Your memo should include the following information:

● Identify and describe a data science application that is discussed in the lecture. What is

the stated purpose of this data science application?

● Identify the stakeholders. In particular, which organization(s), industry, or population(s)

could benefit from the data science application? Which population(s) or group(s) have

been adversely affected, or are most likely to be adversely affected, by the data science

application?

● Option 1: If applicable, identify examples of disparate treatment and/or disparate impact

in the data science application and describe how these examples of disparate treatment

or disparate impact relate to pre-existing bias, technical bias, and/or emergent bias.

● Option 2: If option 1 is inapplicable, give examples of harms that may be due to the use

of the data science application, and explain or hypothesize about the data-related or

other technical reasons that these harms may arise.

You may also discuss any other issue of fairness raised in the lecture.