• "Civilization" is a strategy game in which players build empires by, among other things, deciding where to found cities and deploy armies.

    Image courtesy of Sid Meier's Civilization V

    Full Screen

Computer learns language by playing games

By basing its strategies on the text of a manual, a computer infers the meanings of words without human supervision.


Press Contact

Caroline McCall
Email: newsoffice@mit.edu
Phone: 617-253-2700
MIT News Office

Computers are great at treating words as data: Word-processing programs let you rearrange and format text however you like, and search engines can quickly find a word anywhere on the Web. But what would it mean for a computer to actually understand the meaning of a sentence written in ordinary English — or French, or Urdu, or Mandarin?

One test might be whether the computer could analyze and follow a set of instructions for an unfamiliar task. And indeed, in the last few years, researchers at MIT’s Computer Science and Artificial Intelligence Lab have begun designing machine-learning systems that do exactly that, with surprisingly good results.

In 2009, at the annual meeting of the Association for Computational Linguistics (ACL), researchers in the lab of Regina Barzilay, associate professor of computer science and electrical engineering, took the best-paper award for a system that generated scripts for installing a piece of software on a Windows computer by reviewing instructions posted on Microsoft’s help site. At this year’s ACL meeting, Barzilay, her graduate student S. R. K. Branavan and David Silver of University College London applied a similar approach to a more complicated problem: learning to play “Civilization,” a computer game in which the player guides the development of a city into an empire across centuries of human history. When the researchers augmented a machine-learning system so that it could use a player’s manual to guide the development of a game-playing strategy, its rate of victory jumped from 46 percent to 79 percent.

Starting from scratch

“Games are used as a test bed for artificial-intelligence techniques simply because of their complexity,” says Branavan, who was first author on both ACL papers. “Every action that you take in the game doesn’t have a predetermined outcome, because the game or the opponent can randomly react to what you do. So you need a technique that can handle very complex scenarios that react in potentially random ways.”

Moreover, Barzilay says, game manuals have “very open text. They don’t tell you how to win. They just give you very general advice and suggestions, and you have to figure out a lot of other things on your own.” Relative to an application like the software-installing program, Branavan explains, games are “another step closer to the real world.”

The extraordinary thing about Barzilay and Branavan’s system is that it begins with virtually no prior knowledge about the task it’s intended to perform or the language in which the instructions are written. It has a list of actions it can take, like right-clicks or left-clicks, or moving the cursor; it has access to the information displayed on-screen; and it has some way of gauging its success, like whether the software has been installed or whether it wins the game. But it doesn’t know what actions correspond to what words in the instruction set, and it doesn’t know what the objects in the game world represent.

So initially, its behavior is almost totally random. But as it takes various actions, different words appear on screen, and it can look for instances of those words in the instruction set. It can also search the surrounding text for associated words, and develop hypotheses about what actions those words correspond to. Hypotheses that consistently lead to good results are given greater credence, while those that consistently lead to bad results are discarded.

Proof of concept

In the case of software installation, the system was able to reproduce 80 percent of the steps that a human reading the same instructions would execute. In the case of the computer game, it won 79 percent of the games it played, while a version that didn't rely on the written instructions won only 46 percent. The researchers also tested a more-sophisticated machine-learning algorithm that eschewed textual input but used additional techniques to improve its performance. Even that algorithm won only 62 percent of its games.

“If you’d asked me beforehand if I thought we could do this yet, I’d have said no,” says Eugene Charniak, University Professor of Computer Science at Brown University. “You are building something where you have very little information about the domain, but you get clues from the domain itself.”

Charniak points out that when the MIT researchers presented their work at the ACL meeting, some members of the audience argued that more sophisticated machine-learning systems would have performed better than the ones to which the researchers compared their system. But, Charniak adds, “it’s not completely clear to me that that’s really relevant. Who cares? The important point is that this was able to extract useful information from the manual, and that’s what we care about.”

Most computer games as complex as “Civilization” include algorithms that allow players to play against the computer, rather than against other people; the games’ programmers have to develop the strategies for the computer to follow and write the code that executes them. Barzilay and Branavan say that, in the near term, their system could make that job much easier, automatically creating algorithms that perform better than the hand-designed ones.

But the main purpose of the project, which was supported by the National Science Foundation, was to demonstrate that computer systems that learn the meanings of words through exploratory interaction with their environments are a promising subject for further research. And indeed, Barzilay and her students have begun to adapt their meaning-inferring algorithms to work with robotic systems.


Topics: Computer Science and Artificial Intelligence Laboratory (CSAIL), Computer science and technology, Electrical engineering and electronics, Video games, Civilization

Comments

As an avid player of Cvilization, I am curious which version of the game (they're up to Civ5 now), and what difficulty level? Also, did the program do better playing as or against particular civilizations (the game "personalizes" different historical civilizations by giving them all slightly different capabilities)?
Bit of a tangent, but I'm curious about which victory conditions the AI tended towards. I'm hoping it was space colonization and not world domination.
The AI they include in the game itself does notoriously badly against human players at warfare. In fact, human players who want to win on higher difficulty levels, which give the AI players advantages in terms of production and research and happiness levels, tend to have to compensate for those AI advantages by adopting a more war-centered strategy than they might otherwise prefer. So, yes, it would be interesting if this more high-powered approach to a Civ-playing AI does what human players do against the game-supplied AI, and tends to play a conquest-centered more than a developement-centered game.
The methods were tested on FreeCiv (version 2.2) - the open-source reimplementation of Civilization II. This was because we needed access to the code to modify the game user interface, and let our algorithm programmatically observe and play the game in place of a human. The game manual of Civilization II was used as the strategy guide. FreeCiv doesn't have the concept of special traits for different historical civilizations. So we didn't test how such traits would affect the performance of our method. Having said that, our algorithm learns to adapt its game play based on what the opponent is doing - so it will learn different strategies against opponents with different strategy traits. S.R.K. Branavan
Our algorithm is inherently biased towards finding the fastest way to win the game. Against the built-in AI, this happens to be world domination via an "early rush" strategy - so that is what the method ends up learning. S.R.K. Branavan.
The full details can be found in the paper: http://people.csail.mit.edu/branavan/papers/acl2011.pdf It seems they player Civ II.
They should have given a link to the actual paper. I believe it's here: http://people.csail.mit.edu/regina/my_papers/civ11.pdf In answer to Glen's question, the paper says they're playing Civilization II.
Even more than details such as version and difficulty, it would be extremely useful for the article to link to the paper itself. You can read Regina Barzilay's paper at: http://people.csail.mit.edu/regina/my_papers/civ11.pdf
This is really awesome, and hopefully will lead to better AIs in video games. People like the devs of Dominions 3 should be all over this.
So i guess this algorithm learned from some better info source than the official game manual, considering how un-informative it is ...
Have you tried what happens if two or more algorithms would play against each other? It would be interesting to know what kind of strategies they then would use?
Very interesting you have tackled the problem of learning and not the one of information that seems to distract most researchers in the AI field. IMO It's not so much about telling it what to do, as to what to want.
As a person fascinated by the game industry I was curious to find out whether your algorythm would work playing against a human opponent or wheather it depends on the scripted intelligence of the computer civilization it is playing against. If the former is true than could your work lead to creating a tool for designing artificial intelligence in games based not on scripts but on adapting to what the opposite party is doing (as I understand is happening now vs a AI player)?
With all the similarities of this work to that of Peter Bock at GWU over the past 25-30 years, I'm surprised not to see any references to his work. Bock has also developed a rich taxonomy in this area that doesn't need to be reinvented.
I respect the accomplishments of the authors of this paper, however I think the title for this summary article is grandiose and misleading. "Computer learns language..." suggests that, well, a computer learned a language. It is quite clear from reading the article that nothing of the sort happened.
My guess is the learning algo would not work well against a human player because repeated responses to the same input would be too inconsistent.
It would seem that MIT (at least in this case) bypassed a very important step of the Scientific Method. A body of literature on Collective Learning Systems Theory (as pioneered by Professor Peter Bock and his graduate students at The George Washington University) has existed for several decades. There is no reference to this body of literature and no use of the resulting, finely-tuned governing propositions. This article would lead the uninformed to believe that MIT has developed a new form of machine learning: This is simply not true.
Could the A.I. be tested on a game with more moral complexity such as fallout 3, new Vegas, or Oblivion? The instruction manuals in such games give hints and true instructions on how to manipulate the controls. I am more interested in the choices something that learns can make rather than just the fact that it LEARNS which could be quite frightening. Still Awesome though!
Someone's will have to tell me how AI is demonstrated in these tests with games? For instance that a computer "doesn’t know what the objects in the game world represent" indicates to me a lack of understanding. Also that "it doesn’t know what actions correspond to what words in the instruction set" seems to indicate it doesn't know meaning. Now "as it takes various actions" and it looks for "instances of those words" it is merely matching symbols. This correlation does not prove intelligence any more than a machine that sorts parts can. The original test of AI is a game of chess which merely proves a computer can memorize many moves, many more than it's intelligent human counterpart can. Again memory is not the same as intellect. Neither do algorithms prove intelligence even if they are the product of intelligence. The effect can never be greater than the cause. Also that the computer won more games after "reading the same instructions" a human would merely narrows down the possible activities or moves it can make. This limitation in no way hints at intelligence. So someone please explain to me how anyone can see we are closer to AI?
Ratjaw: I believe that you will find it very hard to argue that what you call "understanding" and "intellect" are substantially different from what you call "pattern matching", if you were to give the pattern-matcher vastly better hardware, design, and longer learning time, comparable to the human neuron, brain, and lifetime, respectively.
Back to the top