140 lines
5.4 KiB
Org Mode
140 lines
5.4 KiB
Org Mode
#+TITLE: Quack-TD
|
|
|
|
Quack-TD is a backgammon playing algorithm based upon neural networks trained
|
|
through TD(\lambda)-learning. The algorithm is implemented using Python 3 and
|
|
Tensorflow.
|
|
|
|
|
|
* Setup
|
|
** Pubeval
|
|
To use Pubeval for evaluation the Python module =pubeval= must first be
|
|
installed. The necessary source files should be distributed alongside the main
|
|
application and located in the =pubeval= directory. The installation can be done
|
|
by entering the directory and running =python3 setup.py install= or =pip install
|
|
.=.
|
|
* Usage
|
|
|
|
The main executable is =main.py=. Various command-line options and switches can be used to
|
|
execute different stages and modify the behaviour of the program. All
|
|
command-line options and switches are listed by running =main.py= with the argument
|
|
=--help=. The central mode-switches are listed below:
|
|
|
|
- =--train=: Trains the neural network for a set amount of episodes (full games
|
|
of backgammon) set by =--episodes= (defaults to 1,000). Summary results of the
|
|
games played during the training session are written to =models/$MODEL/logs/eval.log=
|
|
|
|
- =--eval=: Evaluates the nerual network using the methods specified by =--eval-methods= for a the amount of episodes set by =--episodes= (defaults to
|
|
1,000). Results are written to =models/$MODEL/logs/eval.log=.
|
|
|
|
- =--play=: Allows the user to interactively play a game of backgammon against
|
|
the algorithm.
|
|
|
|
- =--list-models=: Lists the models stored on in the =models= folder.
|
|
|
|
** Evaluation methods
|
|
|
|
Currently, the following evaluation methods are implemented:
|
|
|
|
- =pubeval=: Evaluates against a Python extension based on the =pubeval=
|
|
backgammon benchmark developed by Gerald Tesauro. The source code is included
|
|
in the =pubeval= directory and needs to be installed before use. This can be
|
|
done by running =python3 setup.py install= or =pip install .= from the source
|
|
directory.
|
|
- =random=: Evaluates by playing against a player that makes random moves drawn
|
|
from the set of legal moves. Should be used with high episode counts to lower
|
|
variance. *TODO*: Doesn't even work currently
|
|
|
|
** Examples
|
|
|
|
The following examples describe commmon operations.
|
|
|
|
*** Train default model
|
|
=python3 --train=
|
|
|
|
*** Train perpetually
|
|
=python3 --train --train-perpetually=
|
|
|
|
*** Train model named =quack=
|
|
|
|
=python3 --train --model=quack=
|
|
|
|
*** Train default model in sessions of 10,000 episodes
|
|
|
|
=python3 --train --episodes=10000=
|
|
|
|
*** Train model =quack= and evaluate after each training session
|
|
|
|
=python3 --train --eval-after-train --model=quack=
|
|
|
|
*** Evaluate model named =quack= using default evaluation method (currently =random=)
|
|
|
|
=python3 --eval --model=quack=
|
|
|
|
*** Evaluate default model using evaluation methods =random= and =pubeval=
|
|
=python3 --eval --eval-methods random pubeval=
|
|
|
|
* Model storage format
|
|
|
|
Models are stored in the directory =models=. If no model is specfied with the
|
|
=--model= option, the model is stored in the =models/default=
|
|
directory. Otherwise, the model is stored in =models/$MODEL=.
|
|
|
|
** Files
|
|
|
|
Along with the Tensorflow checkpoint files in the directory, the following files
|
|
are stored:
|
|
|
|
- =episodes_trained=: The number of episodes of training performed with the
|
|
model
|
|
- =logs/eval.log=: Log of all completed evaluations performed on the model. The
|
|
format of this file is specified in [[Log format]].
|
|
- =logs/train.log=: Log of all completed training sessions performed on the
|
|
model. If a training session is aborted before the pre-specified episode
|
|
target is reached, nothing will be written to this file, although =episodes_trained= will be updated every time the model is saved to disk. The
|
|
format of this file is specified in [[Log format]].
|
|
|
|
** Log format
|
|
|
|
The evaluation and training log files (=logs/eval.log= and =logs/train.log=
|
|
respectively) are CSV-foramtted files with structure as described below. Both
|
|
files have semicolon-separated columns (=;=) and newline-separated rows (=\n=).
|
|
|
|
*** Evaluation log (=eval.log=)
|
|
|
|
Columns are written in the following order:
|
|
|
|
- =time=: Unix time (Epoch time) timestamp in local time (*TODO*: should be UTC
|
|
instead?) describing when the evaluation was finished.
|
|
- =method=: Short string describing the method used for evaluation.
|
|
- =trained_eps=: Amount of episodes trained with the model before evaluation
|
|
- =count=: Amount of episodes used for evaluation
|
|
- =sum=: Sum of outcomes of the games played during evaluation. Outcomes are
|
|
integers in the range of -2 to 2. A sum of 0 indicates that the evaluated
|
|
algorithm scored neutrally. (*TODO*: Is this true?)
|
|
- =mean=: Mean of outcomes of the games played during evaluation. Outcomes are
|
|
integers in the range of -2 to 2. A mean of 0 indicates that the evaluated
|
|
algorithm scored neutrally. (*TODO*: Is this true?)
|
|
*TODO*: Add example of log row
|
|
|
|
*** Training log (=train.log=)
|
|
|
|
Columns are written in the following order:
|
|
|
|
- =time=: Unix time (Epoch time) timestamp in local time (*TODO*: should be UTC
|
|
instead?) describing when the training session was finished.
|
|
- =trained_eps=: Amount of episodes trained with the model /after/ the training
|
|
session
|
|
- =count=: Amount of episodes used for training
|
|
- =sum=: Sum of outcomes of the games played during training. Outcomes are
|
|
integers in the range of -2 to 2. A sum of 0 indicates that the evaluated
|
|
algorithm scored neutrally. (*TODO*: Is this true?)
|
|
- =mean=: Mean of outcomes of the games played during training. Outcomes are
|
|
integers in the range of -2 to 2. A mean of 0 indicates that the evaluated
|
|
algorithm scored neutrally. (*TODO*: Is this true?)
|