Introduction:
Welcome to planet Q-Boltz! You have been stranded here after your Millennium Z-Series R23S space craft has crashed. The native beings of this planet survive by draining nuclear energy from mechanical devices. Unfortunately for you the cells in your Z-Series run on nuclear energy. In order to escape the planet you must first destroy all of the energy sucking aliens! But, watch out! They would much rather kill you than allow you to escape!
In order to help you achieve your goal you can collect weapons from the aliens you kill. Also, your B143 Titanium Protection Suit will regenerate health for you if you can avoid being shot for a short time.
A Demo version is available here!
Q-BMW is a modification of Quake 3 created by Emuye Taylor and Brandon Lees for CS196-2 in the spring of 2006 at Brown University. This implementation uses a new AI based on Q-Learning. This method allows the AI to make decisions based on the outcome of previous actions. The result is a more dynamic AI, which can learn and adapt to an opponents style of play.
Additionally, to allow for more varying possible styles of play Q-BMW is a single person campaign game as opposed to a multiplayer game like Quake 3.
Problems with current AI in first person shooters:
Many traditional approaches to AI for first person shooters are based on a Finite State Machine (FSM) design. This approach models the AI's functionality (Wander, Attack, Retreat, Chase, etc) as states in an FSM. In order to create more complicated AI, more states are added to the FSM to represent more fine grained actions. In order to create AI that is more difficult to play against, the AI's accuracy is improved. Another method used to make more challenging AI is to allow the AI to perform actions which the player is unable to. For example, the AI might be given a speed bonus or a damage bonus. From a player's point of view, the shortcoming of this approach is that regardless of difficulty, the AI will play the same. An additional shortcoming is that at higher levels of difficulty, for the player, the game play can feel as if the AI is cheating.The Solution:
To address these problems we implemented a more dynamic AI which is based on learning. The goal of this implementation was to offer a game which provides a more realistic and challenging experience to the gamer. In theory, this AI would be able to react not only to a player's individual actions but also to patterns in these actions. The end result would be a game in which the player would be able to play against NPCS that are able to adapt and react to the player's style of play.
Our Approach:
- Model the game using states
- Create specific actions for the AI
- Use Q-Learning to develop an optimal control policy:
- Create a reward function to reward actions which result in decreasing the AI's target's health, and which penalizes actions which result in decreasing the AI's health
- Assign Q-Values to state action pairs based on a reward function
- Integrate this AI approach with the Quake III engine to create a completed game to demo our AI
More Technically:
- 65,000 states
- 20 actions
- Q-Learning:
- From the current state s, select an action a. This will cause a receipt of an immediate reward r, and arrival at a next state s'.
- Update Q(s,a) based upon this experience as follows:
Q(s,a) = Q(s,a) + x[r + y max(Q(s',b)) - Q(s,a)]
(where x is the learning rate and 0 < y < 1 is the discount factor)