Handling a Beta Test

View the original content here.

By Fabrice Cambounet

Heroes 5 dev letter
#2 : handling a beta test

It seemed quite obvious that we had to include the fans into the development process in some manner. I don’t think anyone can be considered as being the ultimate expert on Heroes questions, but having a few dozens or hundred people sending you ideas or feedback helps getting everything into shape.

The risk however, is to get overrun by the feedback, or to give in to all demands. Most of them are contradictory, and quite a part would be conflicting with other decisions or would alter the direction we are giving to Heroes 5. So it is a huge work to integrate these multiple opinions together and mash something out of that. Simply keeping track of the feedback during the closed beta test was a headache – but most communities have helped getting summaries out.

A disadvantage of making beta tests is the amount of work required to prepare versions and update them in a timely manner. It can disrupt the production process for a long time, and either you get late delivering beta updates, or you get late actually making the rest of the game, and most often you get late in both respects !

Finally, a disadvantage I see now is that, at some point, this has to end. Beta testers, who were frustrated at the start because of the amount of bugs, and also because the changes were not necessarily going to their liking, become frustrated again as they feel they need to be a part of it even longer, until the game is perfect or close to it. But as we enter a polish phase, treating a flow of feedback becomes less interesting and more dangerous : you need to nail down decisions on design points, and constant rediscussions only lead to delay. A game can only be as perfect as production allows.

On the other hand, running a beta test is an exhilirating experience ! You get in touch with hundreds of fans from all over the world, you get lengthy debates, historical summaries, polls, etc, all over your head. It is also a challenge to keep the exchanges running, and to keep the development in check with the requests.

Preparing and selecting people is interesting in itself. With the registration form we have put up, a very precise image of the typical Heroes fan was laid out, thanks to the 15.000 registrants. The average age of the registrants, for instance, was logically quite high, above 24 years old. A large (and untended) community was `discovered` in Israel. The hardcore aspect of the russian fans was proven. The vast fields of casual strategy players in the US, whom we need to get to try Heroes, was underlined. We could also sort out which were the influential communities :)

Then you have to organize the work in advance with communities and internal testers. We had to make out with the constraints and tools we had, in terms of forums, bug reports, etc. The number of spoken languages was a challenge in itself. Each community created language-specific topics to harvest the feedbacks. The update of activation keys had to be taken care of : we added one month of closed beta testing, and thus needed to distribute new keys around. We just ran into a similar problem with the open beta : too many players downloaded the version when compared to our plans :)

In the end, running a beta test is about getting feedback, and we had a truckload in this respect. Even by weeding out the duplicates and the unrealistic suggestions, we had a lot more than a thousand items in the list to browse, in just three months. I think around a large third of this has been integrated, although not quickly enough for the beta testers to actually see their ideas in the game.

Besides suggestions, getting the general feeling on the game was needed. At the time (september) we had been working on the thing for almost two years, and we started to have a blurred vision of the priorities. The E3 and Leipzig shows garnered very positive feedbacks, and all journalists were already enthusiastic, but we needed to get players` opinion. This was also made possible by the discussions on the forums with the fans, and with the communities. Being able to get an opinion prior to release is invaluable.

Finally, running an open beta test after a closed one is about reaching out to more people. You present the game – in its current state, it was not a demo and wasn’t polished – to a vast public besides the first fans of the closed test. I think this challenge was very correctly met, even with the downfalls of the version itself ;)

Back to News and Interviews


Thanks to ThE_HyDrA for creating and looking after the Heroes 5 section of Age of Heroes!