I am not a regular reader of Wired but stumbling across it in Mexico the other day after two weeks offline was refreshing, especially this piece about game testing at Microsoft Games User Research. The article basically describes the work done on Halo 3 by a team of user researcher. Some excerpts i found interesting:
"Pagulayan's team quickly went to work building tools for extracting gameplay data, including the location of each player and when and where they fired weapons, rode vehicles, killed aliens, and died. They ran weekly tests, analyzing 2,300 hours of play by 400 gamers in under two months. Over and over again, they found snags — a mutant alien that was far too powerful, a lava pit that too many players fell into. (...) Some tests include a pop-up box that interrupts the player every few minutes, asking them to rate how engaged, interested, or frustrated they are. Pagulayan also has gamers talk out loud about what they're experiencing, providing a stream-of-consciousness record of their thought process as they play. (...) After each session Pagulayan analyzes the data for patterns that he can report to Bungie. For example, he produces snapshots of where players are located in the game at various points in time — five minutes in, one hour in, eight hours in — to show how they are advancing. If they're going too fast, the game might be too easy; too slow, and it might be too hard. He can also generate a map showing where people are dying, to identify any topographical features that might be making a battle onerous. And he can produce charts that detail how players died, which might indicate that a particular alien or gun is proving unexpectedly lethal or wankishly impotent. The lab also records video footage of every testing session and hyperlinks these clips to the individual progress reports. If the design team wonders why players are having trouble in a particular area, they can just pull up a few test games to see what's going wrong"
Interesting enough, there are some result examples, about spatial behavior that I found pertinent:
"Pagulayan pulls up an early map of Jungle; on it are superimposed the locations of about 30 testers after half an hour of play. The dots are scattered throughout the terrain. This, he says, is bad: It means that people were wandering aimlessly instead of progressing through the level. "People were lost," Pagulayan says. "There wasn't much deep analysis to do here." To solve such problems, the designers must subtly direct player movement by altering the world in small ways. In this case, they decided to change the geography of the Jungle level so that in certain places players had to jump down a steep ledge to reach the next area."
This is very similar to what Ubi Soft employed for Splinter Cell (see here).
See the two examples they give:
The first one shows how "players wandered lost around the Jungle level: Colored dots showing player location at five-second intervals (each color is a new time stamp) were scattered randomly. So Bungie fixed the terrain to keep players from backtracking. Sure enough, the dots clustered by color, showing that players were moving smoothly through the map."
This one is about "player deaths (represented in dark red on this "heat map" of the level) were skewing toward the base on the left, indicating that forces invading from the right had a slight advantage. After reviewing this image, designers tweaked the terrain to give both armies an even chance.
Why do I blog this? although these techniques are common in User Experience, it's good to see how they are applied and which sort of results they lead to.