Recently, the Arcade Learning Environment (ALE) has become really popular in the machine learning field. Google's Deepmind group integrated reinforcement learning and convolutional neural networks into an AI that can learn to play Atari Games. Original article is here. There are some really cool YouTube videos of their approach learning to play games, I'd suggest checking them out.
This post will be the first in the series of replicating their approach. In this tutorial, we will get ALE running and get it set up with Python. Of course, it's not going to be as easy as using Linux, but we all use Windows for our own reasons.
Thankfully, a lot of the legwork for this integration has been done for us by various people and this tutorial should take much less time than the last.
Note: Almost all of the work on the Visual Studio project comes from Martin Brazdil's (my code is forked from his library). The integration from C++ to python (Ale_python_interface) comes from Ben Goodrich at https://github.com/bbitmaster/ale_python_interface. My code here just combines these and makes necessary modifications for it to work in Windows (and some bug fixes).
- Download the updated implementation of ALE 0.4.4 in Visual Studio 2013
- Extract, go into the src folder and load the ALE.sln file
- NOTE: You'll need visual studio 2013... should have it from the last tutorial.
- Build with CTRL+SHIFT+B or under Build->Rebuild Solution
- This will create the needed .dlls under the src/x64/Debug folder
- We'll come back to this in Step 2.2
- Download the Python deep q learning project. Note: this code is work in progress, the files we will be looking at are fully functional but others may confuse and misdirect.
- Extract, then copy the .dll files from the Visual Studio folder (from src/x64/Debug) into the libs folder.
- You can put the .dll's anywhere you want you'll just have to change the ale_python_interface.py file.
- Change ale_lib = cdll.LoadLibrary('ALE.dll') to the location of your .dlls
- Download the breakout.bin ROM file.
- Note: ROMs are a legal gray area, I don't condone breaking the law. Do some of your own research to determine what is legally acceptable for your uses.
- Once you choose your download location you'll need to load the rom using this code
- ale.loadROM(b'd:\_code\\breakout.bin') # do a double \\ if the next character is an escape character
Here are the current best results I've gotten after 4000 games. Note that losing a life is -1 reward so add +5 (total number of lives) to the score to get the number of bricks broken. The best is 17 bricks which happens twice, once before 2000 games and once right after.