代写program、代做Python/c++编程

日期：2025-03-13 08:20

Homework2

Release v5.0 16/02/2025

CONTENTS

1 TicTacToe 1

2 Contents 3

2.1 TicTacToe Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2.2 Reinforcement Learning Player . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.3 Requirements (60 points) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.4 Submission Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.5 Marking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

i

ii

CHAPTER

ONE

TICTACTOE

Gameland is a small independent game development studio. Gameland develops 8-bit games for Android and Apple

products. They have decided to create a series of games for children ages 6 to 12. Their first game will be tic-tac-toe.

However, to make their version of tic-tac-toe different from their competitors they’ve chosen to include an AI player

for the builtin opponent. You have been hired to design and implement an agent that learns to play tic-tac-toe using

Reinforcement Learning.

For this homeworking assignment, you will be working in pairs. Find a partner and complete the registration quiz on

KEATS; make sure both of you complete the Partner Selection quiz on KEATS. It’s best to find someone in your lab

to work with rather than someone from another lab session. A group of three can be formed if there’s an odd number

of students in a lab.

Tic-tac-toe is a game played on a 3 by 3 grid. The game is played with two players, who take turns putting an X or an

O in the grid. The player who has formed a horizontal, vertical, or diagonal sequence of three marks wins. Here’s a

sample game where player X wins. You can learn more about tic-tac-toe here.

1

Homework2, Release v5.0 16/02/2025

2 Chapter 1. TicTacToe

CHAPTER

TWO

CONTENTS

2.1 TicTacToe Module

The TicTacToe module, available on KEATS, contains the functions and classes that you will need to complete this

homework assignment. Be mindful that there are several classes in the module, always make sure you’re editing the

right segment of code. You are not allowed to modify any of the method headers or return types. Changing either

will result in a failing mark. The TicTacToe module has a text-based version of tic-tac-toe already implemented. The

game looks like this

3

Homework2, Release v5.0 16/02/2025

Each square on the tic-tac-toe board has a unique value from 0 to 8. For example, the square at the top left is zero.

4 Chapter 2. Contents

Homework2, Release v5.0 16/02/2025

In the following image, player one has entered 4 as his or her move. An ‘X’ has been placed in the centre of the board.

2.1. TicTacToe Module 5

Homework2, Release v5.0 16/02/2025

2.1.1 Two Human Players

You’ll also find sample code on KEATS. The twoHumans code demonstrates how to use the module for 2-human

players. Download the sample and get the game running. After you get code for 2-human players working, take a look

at the full API. Familiarise yourself with all the functions, classes and methods before you begin implementing the RL

player.

1 import TicTacToe as ttt

2

3 def main():

4

5 # Players

6 player1 = ttt.createPlayer('X', ttt.HUMAN_AGENT)

7 player1.name = 'Alice'

8

9 player2 = ttt.createPlayer('O', ttt.HUMAN_AGENT)

10 player2.name = "Bob"

11

12 # Create board

13 board = ttt.TicTacToe()

14 board.setPlayers(player1, player2)

15 board.drawBoard()

16

17 # Run Game

18 while not board.isGameOver():

19

20 player = board.next()

21 player.makeMove(board)

22 board.drawBoard()

23

24 winner = board.getWinner()

25

26 if winner:

27 print(f'Congratulations {winner.name}')

28

29 main()

2.2 Reinforcement Learning Player

The first step in implementing your reinforcement learning agent is to provide a random number seed, without it you’ll

have a hard time tuning your parameters. Find the following code at the top of the TicTacToe module:

RANDOM_NUMBER_SEED = 9999

random.seed(RANDOM_NUMBER_SEED)

Replace the 9999 with a unique number, for example, your birthday, 10242006.

In the RLPlayer class you will be implementing three methods, makeMove(board), rewardState(board,

prevBoard=None) and getReward(board). Pseudocode is provided for makeMove(board) and

rewardState(board, prevBoard=None). Notice in the pseudocode that the self keyword is sometimes

omitted, it’s your responsibility to add it where necessary.

6 Chapter 2. Contents

Homework2, Release v5.0 16/02/2025

The next three sections explain the methods you will have to implement in the RLPlayer class. Before entering your

code, make sure you’re in the right class.

2.2.1 Make Move

This method is responsible for making a move for the RL agent. It follows the Epsilon-Greedy Algorithm which

balances the need for exploitation and exploration. The RL agent will perform an optimal move (1 - epsilon) percent

of the time, otherwise she will randomly select a move. However, if the agent is not training, she will always perform

an optimal move.

def makeMove(board):

if mode is TRAINING_MODE then do the following

Generate a random number, n, between 0 and 1 (2)

if n is less than epsilon then

anyMove = random.choice(board.remainingMoves)

moveLegal = board.makeMove(anyMove, self.letter)

if not moveLegal:

print('*** WARNING ILLEGAL MOVE BY RL ***')

else

getRLMove(board)

Call rewardState (make sure to pass board and previousState as arguments)

Set previousState equal to a copy of the board

else

getRLMove(board)

2.2.2 Reward State

In class, we discussed the difference between short and long term value of performing an action. In the example we

used in class, we utilised the average reward as the long term value of performing an action. We will take a slightly

different approach for our RL playing agent. We are going to use temporal-difference learning (𝑇 𝐷(0)) which is the

simplest variant of Reinforcement Learning. A full explanation of this Reinforcement Learning approach is outside

the scope of this module. To learn how to play tictactoe, the agent will predict the value of game states. That is, if the

agent knows the value of game states then it can perform the move that will maximise the value of the next game state.

For example, if the current state is 𝑆𝑡, the agent should make a move that maximises the value of 𝑉 (𝑆𝑡 + 1). where

𝑆𝑡+1 is the next game state.

We’re going to use the following equation to predict the value of states:

𝑉 (𝑆𝑡) = 𝑉 (𝑆𝑡) + 𝛼[𝑅𝑡+1 + 𝛾𝑉 (𝑆𝑡+1) − 𝑉 (𝑆𝑡)] (2.1)

In Equation (2.1), the game state 𝑆𝑡 is the previous game state and 𝑆𝑡+1 is the next game state after a move has been

made. Alpha, 𝛼, is the learning rate. The learning rate affects the rate of learning; it determines how much the old state

value, 𝑉 (𝑆𝑡), will change. Gamma, 𝛾, is the discount rate. The discount rate expresses the value of future rewards to

the agent —if it’s set to zero then the RL agent is more concerned about the immediate reward than future rewards.

Alpha and gamma should be set to a number between 0 and 1.

2.2. Reinforcement Learning Player 7

Homework2, Release v5.0 16/02/2025

The RLPlayer class has a an attribute, valueFunction, a dictionary, to implement the function 𝑉 (𝑆𝑡). The dictionary

serves as a look up table for values of states. The dictionary keys are game states and the values are determined by Eq.

(2.1). Use the method board.getKey(letter) (in the TicTacToe class) to convert the game state to a dictionary key.

For example, if you have a board and would like to get the value of that board (or game state), you would execute the

following statements:

1 key = board.getKey(self.letter)

2 value = self.valueOfState(key)

Assuming the board is 𝑆𝑡 then line 2 of this code segment would give you 𝑉 (𝑆𝑡). Here’s the pseudocode for

rewardState(board, prevBoard=None)

def rewardState(board, prevBoard=None):

if prevBoard is None then

Get the reward for board, call getReward

Get the key for the board (board.getKey(self.letter))

Get the value associated with the key, call valueOfState(key)

Update the dictionary, valueFunction, using key to: value + self.learningRate *␣

˓→reward

else:

Get the reward for board, call getReward

Get the key for the current board (board.getKey(self.letter))

Get the value associated with the current board key, call valueOfState(key)

Get the key for the previous board (prevBoard.getKey(self.letter))

Get the value associated with the previous board key, call valueOfState(previous␣

˓→key)

Update the dictionary, valueFunction, using the previous board key to: valuePrev␣

˓→+ self.learningRate * (reward +(self.discountRate * valueCurrent) - valuePrev)

2.2.3 Get Reward

The reward is the immediate feedback that an agents receives about its action(s). There are no exact rules about how

to formulate rewards in Reinforcement Learning. However, the reward will affect learning; a poorly designed reward

scheme can slow the learning process or prevent the agent from learning altogether. For the getReward(board)

method, you will have to design and implement your own reward scheme. For example, you may have a simple scheme

such as if the player wins, the reward is 10, otherwise the reward is -10. The player can also get a positive reward for

drawing the game. Regardless of how your scheme works, the method must return an integer value.

2.2.4 Training

Congratulations, you’re almost done with your RL Agent. Now you must train it. The RL agent will learn to play by

playing hundreds of games against a random player. Download and run the training.py program.

Once you’ve fixed all your syntax errors, you have one final step: fine tune your hyper-parameters. The learning rate,

discount rate, epsilon are set to 0.1, 0.2, 0.3 (line 13). These are not good values. It’s up to you to tweak these values

until your RL agent is performing well. Also, the number of episodes or the number of games the RL gets to play, is

only set to 500 (line 14). You can increase that number up to 50,000. It’s important that you do not share the values

you use with other students.

You may have noticed that everytime the training script runs, it creates a file, cse_policy_hw2.txt Do not edit this file

or change its name. This file contains your RL agent’s policy; its brain. You will submit this text file along with your

8 Chapter 2. Contents

Homework2, Release v5.0 16/02/2025

code.

You might be wondering, How do you tell if your agent is getting better? The training program produces stats that you

can use to evaluate your agent. For example,

Agents Won Lost Draws Rating

RL Agent 4 6 0 1172.4

Random 6 4 0 1227.6

In these results, the RL agent won 4 games but lost 6 (she had no draws). The final value, 1172.4 is her ELO rating

(learn more about the ELO rating system here). These stats tell me the RL Agent did not learn very well. First, she lost

more games then she won. Second, her rating is below 1200; Agents begin with an ELO of 1200 and her ELO went

down. At the very least, your RL agent should do better than an agent that randomly picks moves.

1 import TicTacToe as ttt

2

3 def main():

4

5 # Players

6 rlAgent = ttt.createPlayer('X', ttt.RL_AGENT)

7 rlAgent.name = 'RL Agent'

8

9 partner = ttt.createPlayer('O', ttt.RANDOM_AGENT)

10 partner.name = "Random"

11

12 # Training Session 1

13 rlAgent.initTraining(0.1, 0.2, 0.3)

14 ttt.train(rlAgent, partner, 1000)

15 rlAgent.save()

16

17 # Training Session 2 Optional

18 # rlAgent.initTraining(0.1, 0.2, 0.3)

19 # ttt.train(partner, rlAgent, 1000)

20

21 # Evaluation

22 rlAgent.setMode(ttt.PLAYING_MODE)

23 tournament = ttt.Tournament()

24 tournament.start(rlAgent, partner, 5)

25 tournament.start(partner, rlAgent, 5)

26 tournament.printStats([rlAgent, partner])

27

28 main()

2.2. Reinforcement Learning Player 9

Homework2, Release v5.0 16/02/2025

2.3 Requirements (60 points)

1. Implemented the 3 missing methods, makeMove(board), rewardState(board, prevBoard=None), and getRe ward(board) as described in the pseudocode and brief. See sections, 2.2.1, 2.2.2 and 2.2.3. Do not submit

code from the Internet and expect full credit. (30 points)

2. The RL agent selects a moved based what it has learned through Reinforcement Learning. Your agent must not

use a fixed policy for making decisions. (15 pts)

3. Did not add new code to the bottom of the module such as main(), testing or training code; did not add any print

statements to the module (5 pts)

4. Selected and registered a partner on KEATS (5 pts)

5. Followed all submission requirements (5 pts)

2.4 Submission Instructions

2.4.1 Nominate Someone to Submit

• Only one group member has to submit the group’s assignment —you must discuss and decide which group

member will be responsible for submitting the assignment.

• The person submitting the assignment. This person will take responsibility for uploading the correct files,

submitting everything before the deadline, and for adding all other group members to the submission record.

• Other group members. If you are not submitting the assignment on behalf of the group, keep an eye out for a

notification email that will let you know when your partner has submitted the assignment. The group member

submitting must add you to the submission record. Only after they have done this will you become part of the

assignment group on Gradescope.

2.4.2 Submission

1. Submit the TicTacToe module & policy file on Gradescope by Wednesday, March 26𝑡ℎ, 16:00.

2. You must submit your modified TicTacToe module on Gradescope — the TicTacToe.py file.

3. You must submit your policy file on Gradescope — the cse_policy_hw2.txt file.

4. At the top of your program put the following comments. You must also include the hyperparameters that you

used to train your RL agent in your submission.

1 # [Your Full Name]

2 # [Your K number]

3 # Homework 2

4 #

5 #rlAgent.initTraining(0.1, 0.2, 0.3) PUT YOUR NUMBERS NOT THESE!

6 #ttt.train(rlAgent, partner, 500) PUT YOUR NUMBERS NOT THESE!

5. You do not need to include a cover sheet

6. If asked for a submission title, enter your full name.

10 Chapter 2. Contents

Homework2, Release v5.0 16/02/2025

2.4.3 Late Policy

All coursework must be submitted on time. If you submit coursework late and have not applied for an extension or

have not had a mitigating circumstances claim upheld, you will have an automatic penalty applied. If you submit late,

but within 24 hours of the stated deadline, the work will be marked, and 10 raw marks will be deducted. All work

submitted more than 24 hours late will receive a mark of zero.

• Any program submitted within 24 hours after the deadline will lose 10 points

• Any program submitted after the 24 hour grace period will receive a 0

2.5 Marking

For this lab, your mark will be based on the program’s correctness and its performance. Your RL agent will play in a

tournament against another agent. This other agent plays using random moves and a fixed policy.

Correctness is worth 60% and that’s described under the Requirements section. Performance is worth 40% and depends

on how well your agent performs on one of the following three categories, ELO, win rate or success rate (percent of

games won or drawn). The amount you’ll earn for performance, depends on the class average, i.e., a performance above

average will earn more than a performance below average. Points will be given based on the category where your agent

had the best performance. If your submission crashes (during play) you will receive a 55.

2.5. Marking 11