Devil's Advocate: Where Context Driven Testing fails?

TL/DR: The community of Context Driven Testing is fractured and unclear how to implement the activity of test.

Lack of Clarity

I am still debating the title of this article, and will likely do so even after it is published.  The problem isn't that Context Driven Testing, the problem is the implementations.  For a technical analogy, CDT is like an Interface, something that defines a general contract of how to do things, but doesn't provide any meat on the bones of how to implement the philosophy.  The problem is that in order to learn how to implement all the various pieces of CDT, you must go searching all over the Internet trying to find all the data.  When it comes down to it, is that a failing of the Context Driven Testing philosophy, or the Context Driven Testing community?  I don't really know what to blame, and as someone who identifies with being part of the QA community I feel a need to point it out.  If only we had a sort of JIRA system.

Don't get me wrong, I do think we have a lot of data to offer, I just think it is not in a simple pathway.  There is no college like course structure or even a choose your own adventure system.  However, you might not believe me, so lets start at the beginning.  While context driven testing does say a few things, but it could be summed up as two words: "It depends."  It clearly labels itself an approach:

Context-driven testing is an approach, not a technique.

With that in mind, it becomes all too clear that CDT is not meant as a specific set of methods or patterns but rather an idea that you're smart, you figure it out within your context.  It is a little like going to a councilor and saying "I'm angry all the time, what do I do?" and them only asking why you are angry, without any suggestions.  It might help, but it sure seems to be a round about method.

No Patterns?

I don't disagree with the idea that things depend on context, but I do sometimes wonder if there aren't some context-dependent patterns.  That is to say, I do wonder if there is a sort of mapping of context-to-effective-practices or if literally every situation requires different analysis.  In either case, the current answer context driven testing gives is to consider the situation and create tests based upon that.

This leaves us with a sort of mob-rules attempt at QA where you can do any activity and call that testing, as long as the product solves a problem.  Now I've peeled away my first layer of the onion, and I expect someone to immediately argue with me that my claim is untrue.  In fact, there are many, many source of data from James Bach to Cem Kaner to Michael Bolton to ad nasium.  And you would be right, these are all CDT advocates, some of whom founded the ideas.

I can respect that idea, but now we are applying different people's ideas to the problem space using rough analogies or giving examples by inserting their context and how they react.  That is fundamentally a scatter shot approach, with variable quality between different bloggers and different blog posts, not to mention the fuzzy aspects of trying to apply someone else's context to your own situation.  If there isn't even a mapping of contexts to effective practices, how can you ever be expected to know what is a good method of testing?  Worse yet, even if these can be used as a map of context to ideas, there is no central repository of knowledge.  You have to search for these ideas and hope your picks turned out well.

Frameworks

But wait, what of the various frameworks given?  These frameworks were developed around the idea of CDT, like Session Based Test Management, Thread Based Test Management, and Tours, which is meant for the more detailed work.  I'm not saying those don't exist, but I don't think there is a very good way of knowing the right choice.  There isn't a particularly strong heuristic map, which means that someone coming into the industry just has to keep trying these until they find out what works.

Tours is perhaps the most specific of the set, and is perhaps the exception to these as I have seen some built in recommendations in the naming and descriptor.  The Money Tour is meant to check the features that the customers bought the tool for.  However, once you think a little about it, it seems like that is the useful area to test, so why worry about most of the other tours?  Cem Kaner talks of how researching what tours are useful is a bad option from a context driven perspective.  He asserts that you need to know your developers in order to know what sorts of errors might matter along with liability.  You might also need to understand your objectives in order to consider which tour to do.

So let's consider the sorts of questions Kaner would want us to consider.  What does the company want?  Well in many cases, I think the company looks at QA to say what it wants, or it says what it wants, but it really gets it wrong.  In reality, QA often pushes the agenda, particularly when the company is young in QA.  From my experience, young companies also don't have much money, so it might be reasonable to expect they would hire inexperienced QA, making for a terrible combination.  What would the customer want?  Often these young companies are creating their product with the hope to get customers, so who knows?  Should I still use the Money Tour using what we expect our customers to care about?  Even if I did assume I understood my context, how do I evaluate my success of a given tour when I don't have much idea of what matters?

Lack of Specific Direction

Ultimately, I feel that Context Driven's layers of abstraction make it impossible to learn in any quick fashion without meandering the Internet, reading lots of different opinions and trying lots of ideas until maybe, just maybe, you find a method that works.  Next thing you know, you change jobs with your new experiences and you now have "Best Practices" which you might claim come from the Context Driven Testing philosophy.

It's like someone saying you should use pairwise testing, because that always helps pair down parameters.  Yet, what parameters you choose matters more than the strict number you have, be it a random sampling (which based upon my reading of Bach's paper, only takes roughly 2x of pairwise to reach equal coverage).  Sure, Context matters for the choice of the parameters, but can there not be some 'Good Practices' we can push, based upon some reasonably high level contexts?  Like, oh, I don't know, say in a application that deals with customer purchasing, customer purchasing patterns might be a decent area to choose for your parameters?  But wait, what if you context....  *le sigh*

Doctrine or Methodology?

As one final aside, I ponder if this might be not unlike a religious war based upon if you want your bed made or not.  Some people really want everything neat and tidy, with QA having a very specific role, a specific set of guidelines that are neat and orderly.  On the other side you have the people who say that our role is so impossibly different based upon organization that we can just barely define the activity at a high level, but can't define what activities should be done.  Or if you rather, you can think of it as Linux, with its diverse set of implementations compared to the more mono-culture of Windows or Apple.  Ultimately, I wonder if there is something in the middle, a sort of, useful set of norms in certain settings that can be broken, but only when you have good reason.   As an example of what I'm interested in, let me quote from Kaner tour post,

Rather than looking for a few “best” tours, I think it would be more interesting to develop a more diverse collection of tours that we can do, with more insight into what each can teach us.

NOTE: As an aside, I had my co-author, Isaac, read my post and he noted I cut out Rapid Software Testing, which was unintentional.  I simply haven't studied that form of CDT, which is a big part of the post, that there is no repository of knowledge for CDT testing.