Friday, July 31, 2015

Deep Q-Learning (DeepMind) Arcade Learning Environment

Last time I wrote about configuring Theano with Windows 7&8 64-bit. Now lets do something fun with it.

Recently, the Arcade Learning Environment (ALE) has become really popular in the machine learning field. Google's Deepmind group integrated reinforcement learning and convolutional neural networks into an AI that can learn to play Atari Games. Original article is here. There are some really cool YouTube videos of their approach learning to play games, I'd suggest checking them out.

This post will be the first in the series of replicating their approach. In this tutorial, we will get ALE running and get it set up with Python. Of course, it's not going to be as easy as using Linux, but we all use Windows for our own reasons.

Thankfully, a lot of the legwork for this integration has been done for us by various people and this tutorial should take much less time than the last.

Note: Almost all of the work on the Visual Studio project comes from Martin Brazdil's (my code is forked from his library). The integration from C++ to python (Ale_python_interface) comes from Ben Goodrich at https://github.com/bbitmaster/ale_python_interface. My code here just combines these and makes necessary modifications for it to work in Windows (and some bug fixes).
  1. Download the updated implementation of ALE 0.4.4 in Visual Studio 2013
    1. https://github.com/Islandman93/The-Arcade-Learning-Environment-0.4.4-Visual-Studio-2013-Windows-8.1-x64
    2. Extract, go into the src folder and load the ALE.sln file
      1. NOTE: You'll need visual studio 2013... should have it from the last tutorial.
    3.  Build with CTRL+SHIFT+B or under Build->Rebuild Solution
    4. This will create the needed .dlls under the src/x64/Debug folder
      1. We'll come back to this in Step 2.2
  2. Download the Python deep q learning project. Note: this code is work in progress, the files we will be looking at are fully functional but others may confuse and misdirect. 
    1. https://github.com/Islandman93/python-deep-q-learning
    2. Extract, then copy the .dll files from the Visual Studio folder (from src/x64/Debug) into the libs folder.
      1. You can put the .dll's anywhere you want you'll just have to change the ale_python_interface.py file.
      2. Change ale_lib = cdll.LoadLibrary('ALE.dll') to the location of your .dlls
  3. Download the breakout.bin ROM file.
    1. Note: ROMs are a legal gray area, I don't condone breaking the law. Do some of your own research to determine what is legally acceptable for your uses.
    2. Once you choose your download location you'll need to load the rom using this code
      1. ale.loadROM(b'd:\_code\\breakout.bin') # do a double \\ if the next character is an escape character
This wraps up this tutorial. To run, start the breakout_dqn.py program and let it run (currently it's configured for 2000 games, it takes ~1 day to run on GTX 970 and i7. I'm planning on doing a livecoding.tv walkthrough of the project soon.

Here are the current best results I've gotten after 4000 games. Note that losing a life is -1 reward so add +5 (total number of lives) to the score to get the number of bricks broken. The best is 17 bricks which happens twice, once before 2000 games and once right after. 




5 comments:

  1. Hey there,

    Great articles! I am trying to build the simple python interface that comes with 0.5.1 before moving onto your blog here. I cannot figure out what I'm suppose to do to get this to work on Windows 7 :/. Any chance you'd be willing to spend a few minutes explaining that part in detail? I have built the Arcade learning environment dll using visual studio 2015 c++. Just want to understand how to build the python wrapper and get the example that is included with the source, python_example.py, to work.

    ReplyDelete
  2. From ESP (and looking at your question in the ALE-Group) I think you have all the steps right but need to export the functions in ale_c_wrapper.h. Windows requires the export flag to know which functions are public when creating the DLL (Linux does not I guess). Look at https://github.com/Islandman93/Arcade-Learning-Environment/blob/master/ale_python_interface/ale_c_wrapper.h for a working example

    ReplyDelete
  3. I had the same problem with .dll library files. Want to share with you how to solve it fast http://fix4dll.com/steam_api_dll try to, and I'm sure you'll solve it fast. Just find missed file you need. Good luck.

    ReplyDelete
  4. When you settle on the item, the consequent choice is about the sort of candy machine, that is, an electric or an auto movement machine. machine learning course

    ReplyDelete
  5. It is not uncommon for small businesses with limited resources to be challenged at the thought of facing their bank manager to apply for business funding. The reason is simple; regardless of how long you have been with your bank, you will still have to comply with formalities when it comes to funding your business start-up or business growth. Fundamentally, you will be asked to write a business plan for funding which must be presented with your application form. You may wonder why you need to present a business plan to lenders or investors. Let's think about why banks want you to prepare a business plan and then you will fully understand why investors ask for this precious document that will cost you https://adesk.cheapsoftwaredownload.net/autodesk-maya.html time and money to put together, but ultimately, if done well, will help you raise the much needed finance.

    ReplyDelete