Data-Driven Studio Decisions
by Laure Haak
Sarah Gagnon is the director of Field Day Studio. In this role she wears many hats, including creative director, story writer, and producer of the learning games. She works across stakeholders - her team of creative designers and coders, as well as game sponsors, subject experts, classroom teachers, and education researchers - to map out projects and manage the game design, build, and release process. The Field Day studio releases 4 to 6 games every year so she is in constant motion.
“I love story telling and making art with a team,” says Sarah. “The synergy I see is how game design is key to reaching large audiences.” Only in this way will learning games be useful in testing research hypotheses about the ways students learn. To get there, the games need to be instrumented, but they also need to be great stories of interest to students, with amazing art and design, all crafted for the age of the intended player audience. And they need to be built and tested with teachers and students to ensure that the game is both communicating the learning topic and also engaging the students.
So when Sarah talks about the data needed to produce learning games, she talks about beautiful art first. Because without captivating kids' imaginations, games don’t get far. Then playtesting: iterations with teachers, students, data collected in the game interface but also coordinated with researchers and designers in the classroom who monitor student reactions when playing the game. “It is a lesson in humility when you test your game with students,” says Sarah. “You find out what they really think and it is often not what you hope and dream.” Each milestone of the game build process has to incorporate testing.
Field Day has a 6-week milestone cycle. Each of the team members is working on a number of projects at one time, so this spacing works better than weekly scrums. And at each milestone, there is an external audience that plays the game. And then the team adjusts the build and goes through the next iteration. They generally start with the subject matter, which inspires the art and story, and then develop the game mechanics and flex the art and story as needed.
When it comes to the learning research, that’s when it gets more tricky. The team has worked with a huge range of researchers. What they have found to be important is at every milestone to revisit and clarify research goals. Even the best written grant is not detailed enough to specify how to integrate research into learning games. “The learning researchers have to be flexible, and they have to be ready for detailed meetings and lots of questions,” explains Sarah.
While research goals are prioritized every step of the way, the design process identifies all kinds of paradoxes and contradictions. The story may start out plotted in one direction, but the actual science is another thing. Sarah’s team will usually flex the game design, but only to a point, because they have learned that kids won't play a game if it's not fun.
There are four types of game test data that are particularly relevant for studio design teams:
Think-alouds. Students play in pairs and talk out loud to each other as they are playing the game. This helps to identify how students experience the game, what they think about it and what they don’t understand.
In-classroom. Researchers and developers watch kids play and can see their reactions. They can inquire when a kid gets stuck, or really likes something. This is a really effective reality check for the design team. Teacher-hosted questionnaires are used with the in-classroom testing, which the teacher and the students fill out after they play.
Surveys. Built into games, surveys assess ease of use, deployment in a classroom, and integration with curriculum. Surveys are purpose-built for each game, and often tailored for different phases of the build process. Often they are embedded directly within the game.
Logging data. Games are built to collect anonymous player data. Students can’t be identified, even while their decisions in gameplay can be logged, including choices they make, and how long they play in a level. The key distinction with other test data is that you don't hear what the player thinks or feels, you only see what they do. With logging data you get a different kind of "true" information.
“Game logging and analytics has been really transformational,” said Sarah. So much so that the team is looking for ways to integrate logging earlier into the game design and test cycle to assist with the prototyping process. At present they have been deploying logging when the game is released in beta, to a pre-release audience that is larger than classroom testing. This helps to identify big picture problems, such as drop-off points where students leave the game, bugs, strange behaviors and emergent properties, or places where incentives in the game need to be rebalanced. During playtesting the logging data is looked at right alongside of the playtesting observations, giving a more complete view of what really happened.
Incorporating logging earlier in the design process means a data schema also needs to be developed earlier. The logging data can reveal different player experiences than verbal feedback, providing a valuable input to the design process. While this may slow down other aspects of the design work in the early iterations, the hope is that the logging enables finer-tuned understanding of how students are using the game, such as, are they pressing certain buttons important to the story or learning objective? Implementing logging earlier may also mean that, as the team learns, the data schema will need to be adjusted. But, this may actually be a time savings: instead of waiting for the schema to be wholly created in later iterations, the team will have the schema implemented and evidence to update it.
Once a game is deployed, telemetry logging is the only way to continue learning. Data from hundreds of thousands of players reveals more subtle patterns and exposes what is happening with even small populations of the players. It also allows the team to run experiments and A/B tests, such as this project that explored the effects of different scripts or this project that explored different types of players.
The Field Day team has been working on a larger project to share what they are learning with the broader education research community. The tools they are developing to instrument games are available on an open code repository. In most cases, depending on the game sponsor, the team also shares game code and art. Each project has a landing page, and the team works out with the sponsor what can be shared.
And now, Field Day is exploring with other researcher teams and studios a shared game repository for making playable learning games, as well as code and artwork, available to the community.
Further Reading
Gagnon, D.J., Baker, R.S., Gagnon, S., Swanson, L., Spevacek, N., Andres, J., Harpstead, E., Scianna, J., Slater, S. and San Pedro, M.O. (2022). Exploring players' experience of humor and snark in a grade 3-6 history practices game. arXiv preprint arXiv:2210.09906. https://doi.org/10.48550/arXiv.2210.09906
Swanson, L., Gagnon, D., Scianna, J., Mccloskey, J., Spevecek, E., Slater, S. & Harpsted, E. (2022). Leveraging cluster analysis to understand educational game player styles and support design. GLS 13.0 Conference Proceedings. Games, Learning, Society, Irvine, CA. https://doi.org/10.48550/arXiv.2210.09911
This article is part of a series on pilot projects carried out by the Field Day team in collaboration with internal and external partners, to explore the opportunities and benefits of creating shared approaches to learning games, through standards, processes, and infrastructure. This work was supported by the National Science Foundation, Grant # 2243668