Friday, July 31, 2015

Deep Q-Learning (DeepMind) Arcade Learning Environment

Last time I wrote about configuring Theano with Windows 7&8 64-bit. Now lets do something fun with it.

Recently, the Arcade Learning Environment (ALE) has become really popular in the machine learning field. Google's Deepmind group integrated reinforcement learning and convolutional neural networks into an AI that can learn to play Atari Games. Original article is here. There are some really cool YouTube videos of their approach learning to play games, I'd suggest checking them out.

This post will be the first in the series of replicating their approach. In this tutorial, we will get ALE running and get it set up with Python. Of course, it's not going to be as easy as using Linux, but we all use Windows for our own reasons.

Thankfully, a lot of the legwork for this integration has been done for us by various people and this tutorial should take much less time than the last.

Note: Almost all of the work on the Visual Studio project comes from Martin Brazdil's (my code is forked from his library). The integration from C++ to python (Ale_python_interface) comes from Ben Goodrich at My code here just combines these and makes necessary modifications for it to work in Windows (and some bug fixes).
  1. Download the updated implementation of ALE 0.4.4 in Visual Studio 2013
    2. Extract, go into the src folder and load the ALE.sln file
      1. NOTE: You'll need visual studio 2013... should have it from the last tutorial.
    3.  Build with CTRL+SHIFT+B or under Build->Rebuild Solution
    4. This will create the needed .dlls under the src/x64/Debug folder
      1. We'll come back to this in Step 2.2
  2. Download the Python deep q learning project. Note: this code is work in progress, the files we will be looking at are fully functional but others may confuse and misdirect. 
    2. Extract, then copy the .dll files from the Visual Studio folder (from src/x64/Debug) into the libs folder.
      1. You can put the .dll's anywhere you want you'll just have to change the file.
      2. Change ale_lib = cdll.LoadLibrary('ALE.dll') to the location of your .dlls
  3. Download the breakout.bin ROM file.
    1. Note: ROMs are a legal gray area, I don't condone breaking the law. Do some of your own research to determine what is legally acceptable for your uses.
    2. Once you choose your download location you'll need to load the rom using this code
      1. ale.loadROM(b'd:\_code\\breakout.bin') # do a double \\ if the next character is an escape character
This wraps up this tutorial. To run, start the program and let it run (currently it's configured for 2000 games, it takes ~1 day to run on GTX 970 and i7. I'm planning on doing a walkthrough of the project soon.

Here are the current best results I've gotten after 4000 games. Note that losing a life is -1 reward so add +5 (total number of lives) to the score to get the number of bricks broken. The best is 17 bricks which happens twice, once before 2000 games and once right after. 


  1. Hey there,

    Great articles! I am trying to build the simple python interface that comes with 0.5.1 before moving onto your blog here. I cannot figure out what I'm suppose to do to get this to work on Windows 7 :/. Any chance you'd be willing to spend a few minutes explaining that part in detail? I have built the Arcade learning environment dll using visual studio 2015 c++. Just want to understand how to build the python wrapper and get the example that is included with the source,, to work.

    1. Machine Learning Projects for Final Year machine learning projects for final year

      Deep Learning Projects assist final year students with improving your applied Deep Learning skills rapidly while allowing you to investigate an intriguing point. Furthermore, you can include Deep Learning projects for final year into your portfolio, making it simpler to get a vocation, discover cool profession openings, and Deep Learning Projects for Final Year even arrange a more significant compensation.

      Python Training in Chennai
      Python Training in Chennai

      Angular Training Project Centers in Chennai

  2. From ESP (and looking at your question in the ALE-Group) I think you have all the steps right but need to export the functions in ale_c_wrapper.h. Windows requires the export flag to know which functions are public when creating the DLL (Linux does not I guess). Look at for a working example

  3. I had the same problem with .dll library files. Want to share with you how to solve it fast try to, and I'm sure you'll solve it fast. Just find missed file you need. Good luck.

  4. When you settle on the item, the consequent choice is about the sort of candy machine, that is, an electric or an auto movement machine. machine learning course

  5. It is not uncommon for small businesses with limited resources to be challenged at the thought of facing their bank manager to apply for business funding. The reason is simple; regardless of how long you have been with your bank, you will still have to comply with formalities when it comes to funding your business start-up or business growth. Fundamentally, you will be asked to write a business plan for funding which must be presented with your application form. You may wonder why you need to present a business plan to lenders or investors. Let's think about why banks want you to prepare a business plan and then you will fully understand why investors ask for this precious document that will cost you time and money to put together, but ultimately, if done well, will help you raise the much needed finance.

  6. Debt consolidation loans are a popular way for people to free up money each month by consolidating several monthly credit card payments into a single lower interest loan. But, the question is whether it's best to consolidate those debts into a home equity loan or an unsecured debt consolidation loan. buymodafinilonline

  7. Do you need investors? Looking for a loan? Do you want to apply for a grant? Or has the time just come to do a self-analysis of your business? Are you expanding your business? Looking for new markets? Seeking the next level in your business? These are all times that you need a business plan? What are the nuts and bolds of a business plan? download lumion 9.5 pro

  8. This is a great blog” and i want to visit this every day of the week .   putlocker

  9. digital marketing courses in hyderabad by 360DigiTMG is the best one with quality training, best support and more than 9,000 Students Enrolled
    digital marketing courses in hyderabad