Friday, December 5, 2008

Testing in Layers

My old boss and mentor was chatting with me over Facebook recently about a concept that he termed, "testing in layers". It is the basic idea of slowly testing out a new release with people who are close to the code, then providing the release to people a little further away (friends/family/investors), then a little further (active members on your forums), while eventually moving out toward your normal everyday users. This idea struck me because it provided a simple mental model for actions that I think many in the testing community are already doing at a basic level, and provided much room for complexity if need be. It's nice to provide a concept around an existing process, so that it can be better codified and thought out.

In my world, I use this idea in the following context:
  • Test internally with engineering (QA)
  • Test internally with non-engineering (Marketing, etc)
  • Test by releasing to trusted users on our forums
  • Test by sending the release to particular people who email into Support, whose problems may be fixed with this release
  • Test by releasing to everyone on our forums
  • Test by releasing to a small number of people who download through the website
One of the issues with this idea is that a feedback loop needs to be in place at each layer. If we should release a build to the forums and don't listen to the complaints and issues that this layer is having, then this testing is pointless. However, the further out you get from the center, the harder it is to get solid feedback. When we release to a small number of people through the website, they probably don't know that they're getting a brand-spanking new release. In this case, how do we get their feedback? As of right now, we simply watch to see if they uninstall, and if the stars align, they might even leave us feedback. It's not perfect, and can use improvement, however so far it's a process that is working adequately to help us determine how the release is doing.

The idea of quality feedback loops for your product will be saved for another blog post. Lets just say that twitter has been proving quite useful in getting product feedback lately.