litany against fear
RailsConf 2009 Guitar Hero Behind The Music
This is part of my series of notes on RailsConf 2009. Check them all out here.
This talk is about bringing the data from the game into the web. They’re going to go over the process, the programming, and the infrastructure behind the Guitar Hero titles. They’ve built a pretty huge Rails app that collects stats from the game across all of the currently available titles.
They use feature driven development. This causes plenty of issues: with deployment, with getting customers to use it, and even more. A big part of this is managing expectations of the client. Lots of different stakeholders behind the game, and it’s the project manager’s job to deal with them and the developers.
One huge difference with developing the website is that it’s never actually complete, unlike the game. They use a modified scrum process for iterations. They had plenty of issues with traditional scrum, though:
- Lots of unnecessary overhead
- Sprints were way too long
They changed it so sprints were much shorter, usually a week, and daily standups to help out with communication.
Lots of maintenance they have to deal with: managing refactors and upgrades along with making sure people can still use the game. A huge saving grace for this was using continous integration.
Going over the architecture of their system. The Wii system talks directly to a Rails stack, but the Xbox 360 and PS3 do not, but the data eventually shows up on their network.
They’ve made some serious modifications to Rails. STI was not a good solution, it doesn’t handle millions of rows in a good way. They created the GHConfigurator, which has a lot of crazy features:
- Title specific associations (for example: high scores across different games)
- DB sharding, and making sure that connections aren’t opened everywhere
They used Array#pack and String#unpack to cram a lot of data for sending stats to the GH servers. The Wii and Xbox/PS ways of dealing with the stats are drastically different and required ways to work with this.
They created an event manager to deal with processing stats coming in, but there were a lot of problems. There were too many GH titles, lots of load, the deploys had to wait until the queues were empty, and so on and so forth. It was bad.
Don’t redesign the solution, solve the problem! They had to fix this monster by splitting it into separate parts:
- Reaper: Get events from the DBs and toss them into the queue, using Sparrow. Backed by sqlite3.
- Dispatcher: Check the queues every so often, create consumers. They had to deal with problems that the game developers send them here as well.
- Consumer: Parse the xml, actually create the high scores and other magic.
Huge success, lots of great feedback from the community about how well it’s working. Since it’s separated out it’s a lot easier to test, deploy, and manage overall.
The configurator broke fixtures completely. FactoryGirl saved them from this. Transactions also broke pretty seriously, but it was fixed by patching ActiveRecord. Caching was also hosed. They use fragment caching mostly, and they have to manage expiring them on their own. Caching on pretty much every single query.
They use capistrano along with branches in SVN for deployment. Three environments: development, staging/QA, and production. Pretty standard set of tools for serving: Thin, Monit, HAProxy, NGINX, Memcache, MySQL.