Microsoft has introduced Muse, a generative AI model designed for gameplay ideation. The model, built on the World and Human Action Model (WHAM), can generate game visuals, controller actions, or both.
The research, published in Nature, was developed by the Microsoft Research Game Intelligence and Teachable AI Experiences (Tai X) teams in collaboration with Xbox Game Studios’ Ninja Theory.
The research aims to refine AI-generated gameplay for game development and interactive storytelling. Microsoft has open-sourced the model’s weights, sample data, and the WHAM Demonstrator, a concept prototype for interacting with WHAM models. These resources are available on Azure AI Foundry.
“I’m incredibly proud of our teams and the milestone we have achieved, not only by showing the rich structure of the game world that a model like Muse can learn but also by demonstrating how to develop research insights to support creative uses of generative AI models,” said Katja Hofmann, senior principal research manager at Microsoft Research.
Muse was trained on human gameplay data from Bleeding Edge, a 4v4 online game by Ninja Theory. The dataset includes visuals and controller actions recorded with user consent. The model has been trained on over 1 billion images and actions, representing more than seven years of continuous gameplay.
Gavin Costello, technical director at Ninja Theory, said, “It’s been amazing to see the variety of ways Microsoft Research has used the Bleeding Edge environment and data to explore novel techniques in a rapidly moving AI industry.”
The research was motivated by the release of ChatGPT in 2022. Microsoft scaled the model’s training from a V100 GPU cluster to H100s, refining its representation of controller actions and images. Early versions struggled with consistency, but iterative training improved the model’s ability to predict accurate game dynamics.
Comparing Muse’s generated visuals with actual gameplay, researchers assessed key capabilities such as consistency, diversity, and persistency. Consistency measures whether generated sequences adhere to game dynamics.
On the other hand, diversity evaluates how gameplay variations evolve from the same prompt. Persistency determines if introduced elements are maintained in subsequent sequences.
Cecily Morrison, senior principal research manager at Microsoft, highlighted the importance of involving game creators from the outset. “It was a great opportunity to join forces at this early stage to shape model capabilities to suit the needs of creatives right from the start, rather than try to retrofit an already developed technology.”
Meanwhile, xAI chief Elon Musk recently announced that the company is launching a game studio to reshape the gaming industry. While announcing xAI’s latest model, Grok-3, Musk said, “We’re launching an AI gaming studio at xAI. If you’re interested in joining us and building AI games, please join xAI.”
