I’ll admit, I think I have a problem when I playtest games and scenarios. It’s a forest-and-trees problem — I’m always erring on the side of the immediate play experience, making play decisions that work in the moment but don’t necessarily reflect the actual Rules As Written, or are otherwise difficult to systematize. I catch myself saying things in D&D like, “I’m going to allow it, for it amuses me.”
That’s great for a home game, but it’s tricky when I’m trying to playtest something I’m hoping to submit to, say, Dungeon magazine.
I’m experiencing the same issues now and again in my playtests for Razed, in which I find myself trying to mix up the storytelling tricks and tactics from story to story, sometimes in ways that I wouldn’t dare to put into the sample adventure in the book. I find myself putting my immediate campaign before the product I should be working on. This used to bother me.
In recent weeks, though, I’ve come around to a happier conclusion: this is sometimes fair game. Playtesting, at least for an RPG, is as much about exercise, at these early stages, as it is about a strict testing of the mechanics. Ideally, an RPG will be dynamic enough to handle things like nonlinear storytelling, flashbacks, and dramatic deviations from the prescribed setting — whatever tricks the GM might want to try at home — without breaking. In the case of my Razed game, that’s what I’ve been testing over the past few sessions: the tensile strength of the game. In the case of D&D, I’ve been testing to see if my dungeon is fun and inspirational.
It’s not as though players at home, further down the line from the original idea, should be expected to say, “Wait, that’s too engaging, that’s too fun, and the game’s about nuance and rigidity rather than fun.” It’s important to test the rules as they will be written, but it’s also important to test an RPG’s dynamism.
Yes, it’s possible to take this too far. Playing a campaign isn’t always suitable playtesting — and GM fiat doesn’t help test mechanics. This kind of playtesting can’t come at the expense of more solid testing of the mechanics themselves.
What it is, though, is a valuable design tool for discovering things that need to be covered by the rules or the setting, for keeping the designer engaged with the material and in sync with the potential GM.
In the case of Razed, for example, the stricter playtesting will come when the game as written gets in front of strangers and I’m not there to improvise the setting or patch over design holes I’ve been fool enough to allow. Then it’s not quite the game that gets tested, but the trinity that makes up the game: the manuscript, the spirit of the text, and the mechanisms described by the text.
Right now, I’m playtesting the spirit of the text, and turning it into a manuscript based on the lessons I’m learning.
True words.
OTOH, the dark side of this attitude is apparently what happened with D&D4 skill challenges: All the designers were apparently playing their own variations on mechanics that never actually made it into the rulebooks. (At least 3 or 4 variations of the mechanics could be found spread across the first half dozen releases of the game, and they’ve never managed to iron out all the probability issues they were ignoring through fiat.)
So, in that vein, I recommend keeping your eyes open for the best-practices that emerge from your actual gameplay and seeing if they can be integrated into the system itself.
The flip-flip-side of that, OTOH, is over-specifying the product to the particular experience of your table. The end result of that process are the modules you see where someone is trying to capture a particular sequence of events that happened in their personal campaign. (I remember one module, for example, where an inexplicably huge amount of text — and several key turning points in the adventure itself — was dedicated to explaining the consequences of having one of the PCs fall in love with a particular NPC.)
All true dangers, there.
I think Skill Challenges are a special case, though, in that they’re a “language mechanic” — a mechanism meant to be used for communication rather than strict adjudication. It’s meant to be riffed on. Which isn’t to say that Skill Challenges were properly defined in the original rules, but is to say that they are slippery, and thankfully so. The lifespan of D&D’s history with Skill Challenges — the layers of advice and variations, the demonstration of what’s possible rather than the prescription of what’s “legal” — turns out to be a great way to teach them. Skill Challenges aren’t a closed subject, and they should continue to alter and adapt through actual play and ongoing design. I still hold them to be one of the great strengths of 4th-Edition D&D.
But I’m wandering a bit afield now.