Consultant's War

My co-contributor Isaac has been pondering the fact that we see more consultants blogging regularly, speaking their mind and controlling the things we talk about.  I don't care if it is  Dorathy "Dot" Graham, Markus Gärtner, Matt Heusser,  Michael Bolton, Michael Larsen (Correction: As Matt pointed out, I got my facts mixed up and Micheal is not a consultant), Karen Nicole Johnson,  James Bach or Rex Black.  There are dozens of consultants I could have listed (I mean, how could I have missed Doug Hoffman!), some of whom I have personally worked with.  So this is not meant to be a personal attack on any of them.  I realize I list more CDT folk than others, and while that is not intentional, I read less of the thought leaders from the other schools.  If you look, I bet most of the people you read either are or were consultants [In interest of full disclosure, I have written a hand full of articles and have been paid, but I am not nor have I ever been a consultant, and with you reading this, it is at least one counter-argument, but we'll get to that in a bit.].  If you made it here, there is little doubt you've read at least of those author's works at least once.  They are stars!  I mean that literally.  I found this image of James Bach just by searching his name:

From: http://qainsight.net/2008/04/08/james-bach-the-qa-hero/

I didn't even use image search to find that, it came up on the first main page of my Google search results!  I respect James, he writes well and has interesting ideas, but I have no opinion on his actual testing skills, as using his own measure, I have not seen him perform.  Keeping that in mind, I have not seen any of the above consultants do any extended performances in testing, even if I have listened to many of them talk and gained insights from them.  They know how to communicate and they are often awesome writers.  Much better than I am.  They all are senior in the sense that they have done testing for years.  Some of them disagree with others, making it often a judgment call of their written works on who to listen to.  Some of it is the author's voice.  In my opinion, Karen Johnson is a much softer and gentler voice than James Bach.  But some of it is factual.  I have documented several debates between Rex Black and various other people I have listed.  Rex has substantially different ideas on how the world should work regarding testing.

 ISO 29119

James Christie, another consultant brought up a standards in CAST 2014.  It was a good talk and clearly he had some valid concerns about the standards that have been created.  I have talked about that earlier (I am not going to include as many 'justifying' links as all my points and links are in the earlier article).  I am actually not terribly interested in the arguments and in reality, ISO 29119 is more of a placeholder than the actual point of interest.  I am more interested in discussing who is creating these debates.  In part we are in a war between consultants and in part we are in a war of 'bed making'. Clearly the loudest voices are from those who have free time and have money at stake. The pro-ISO side has take the stance of doing as little communication as possible, because it is hard to have a debate when one side won't talk. It is a tactical choice, and it is very difficult to compromise when one side doesn't speak. Particularly when the standard is already in place, the pro-standards side doesn't have much to gain from the messy-bed side who wants the standard withdrawn.

Both groups of consultants have money to gain from it.  A defacto-RST-standard would make Bach a richer man that he already is.  Matt Heusser's LST does give at least some competition of ideas, but still that isn't much more diversity. The standards would probably make the pro-standards consultants more money as they can say, "as the creator of the standard I know how to implement it."  Even if they made no more money in so far as making the standard, they will be better known for having made the standard. Even if I assume that no one was doing this out of self-interest, the people best represented are the consultants and to a lesser degree, the academics, not the practitioners who may feel the most impact.  Those leading the charge both for and against the standard are primarily consultants or recent ex-consultants. Clearly this is a war for which the people with the most time are waging, and those are the consultants. Ask yourself, of those you have heard debate the issue, what percentage are just working for a company?

Granted, I am not a consultant, but it takes a LOT of effort to write these posts.  It isn't marketing for me, except possibly for future job hunting, and the hope that I will help other people in the profession.  I know non-consultants are talking about it, but we don't have a lot of champions who aren't consultants.  Maybe most senior level testers become consultants, perhaps due to disillusionment of testing at their companies.  Maybe that is why consultants fight so bitterly hard for and against things like standards.  Perhaps my assertion that money at the table is a part of it is just idle speculation not really fit for print.  I can honestly believe it to be that is possibly the large majority of the consultants involved.

 Bed: Do you make yours?

Then what is it that causes these differences of view?  Well let me go back to the bed making.  To quote Steve Yegge:

I had a high school English teacher who, on the last day of the semester, much to everyone's surprise, claimed she should tell just by looking at each of us whether we're a slob or a neat-freak. She also claimed (and I think I agree with this) that the dividing line between slob and neat-freak, the sure-fire indicator, is whether you make your bed each morning. Then she pointed at each each of us in turn and announced "slob" or "neat-freak". Everyone agreed she had us pegged. (And most of us were slobs. Making beds is for chumps.)

That seems like a pretty good heuristic and I think it is also a good analogy.  Often those who want more documentation, with things well understood before starting the work and more feeling of being in control via documentation are those who feel standards are a good idea.  They want their metaphorical beds made.  They like having lots of details written down, they like check lists and would rather make the bed than have a ‘mess’ left all day. A nice neat made bed feels good to them. Then there are the people who see that neat bed and think it is a waste of time at best. At worst, they think that someone will start making them make their bed too. Personally, I think "Making beds is for chumps", just like Steve Yegge. Jeff Atwood would go even further and call it a religious viewpoint:

But software development is, and has always been, a religion. We band together into groups of people who believe the same things, with very little basis for proving any of those beliefs. Java versus .NET. Microsoft versus Google. Static languages versus Dynamic languages. We may kid ourselves into believing we're "computer scientists", but when was the last time you used a hypothesis and a control to prove anything? We're too busy solving customer problems in the chosen tool, unbeliever!

Clearly the consultants, many of whom I take valuable bits of data from care about what they do and they tried to lock down their empirical knowledge into demonstrable truths.   But that doesn't mean they have 'the truth'. I know as testers we try to have controls, but I am not so convinced we have testing down to a science.  It is why I feel we aren't ready for standards, but I also recognize the limits of my own knowledge.

For what it is worth, I tend to be against bed making, I think having all this formal work and making checklists is rather pointless unless they fit what I am doing. Having one checklist to rule them all with a disclaimer that YMMV and do the bits you want doesn’t sound much like a standard, but those whom like their bed nice and neat probably feel very happy when they come home at night. The ‘truth’ about the value of making your bed, the evidence that it is better is less than clear. Maybe one day we will have solid evidence in a particular area that a particular method is better than another, but we aren’t there yet in my opinion. But still I don’t make my bed.  In case you are wondering, I think this is probably one of the hardest questions we have in the industry:  What methods work best given a particular context and what parts of the context matter most?

In consideration of ISO 29119

I have been aware of this debate ever since attending CAST 2014, but I've not been quick to sign.  I wanted to investigate and see what other viewpoints there might be.  To quote Everett Hughes, a sociologist:

“In return for access to their extraordinary knowledge in matters of great human importance, society has granted them a mandate for social control in their fields of specialization, a high degree of autonomy in their practice, and a license to determine who shall assume the mantle of professional authority. But in the current climate of criticism, controversy, and dissatisfaction, the bargain is coming unstuck. When the professions’ claim to extraordinary knowledge is so much in question, why should we continue to grant them extraordinary rights and privileges?”

This is a serious question, one for which I appreciate that a standard might seem like a correct method for 'proof' of our professional authority.  However, even if the standard is in fact a professional guide, I don't think anyone but practitioners can judge its value.  As someone who has had a keen interest in trying to understand this standard, and not wanting to judge too quickly, I have tried to get a hold of a great deal of material about it.  I have engaged with some of those whom disagree with my point of view, such as professional tester.  In fact, they have published my response in there Oct. 2014 issue (albeit with some minor mistakes from my draft).  I looked into purchasing the standard.  I have looked for those who are pro-standard and what they've had to say about this movement as well as what they have said in the past.

That being said, I'm not convinced we actually do know what a standard should look like, much less if this is the standard we need.  Maybe it is what we need, but I strongly doubt it.  I think we are likely many years or decades away before we will even be able to claim we have true repeatable practices oriented towards different contexts, assuming it is possible.  I am unable to judge at present what the standard does say, as the standard and the standard body's work is less than transparent.  I've been forced to sign up with personal details in order to read documentation about the standard's creation, although that has been changed since (NOTE: The file name also changed, including a date, which makes it hard to know if anything has changed, as there was no change log as of 10/26/2014).  The standard requests I pay in a currency that I don't use.  I've signing the petition to withdraw the standard not because I know it is wrong, but because I can't tell, which makes it useless at best and dangerously wrong at worst.  What I can tell is that the standard's various author's other documents around the standard demonstrate what I consider to be confusing if not out right contradictory statements, making me doubt the actual standard.  That breaks the social bargain us professionals have that Everett Hughes so eloquently described.

Perhaps you could say that I should have been personally involved in the standard, and that is a valid complaint.  However, I'm not aware of anyone particularly reaching out to the AST, nor have I heard of it from any other group except for James Christie's talk in 2014.  I have attended both AST sponsored and non-AST sponsored conferences for years, so this isn't a case of willful ignorance.  This is my first chance to review the material and process, yet I have not found the process particularly open or transparent.  I see claims of no best practices and claims that the standard will create best practices.  I have found so many confusing statements by the standard's body that I must conclude that the standard should be withdrawn until it can be thoroughly reviewed and modified, if it can even be modified into something useful.

Even ignoring past statements, the recent defense of the standard creates questions.  One of the easiest and most obvious to consider is who wrote the standard.  Then there is the question of who pays for the creation of the standard?  Well clearly this was not just a labor of love, as Dr. Reid says the costs of development have to be passed on to the customer's of the standard.  I should note, my blog makes me no money and I am not a consultant so I have little incentive to make money speaking about this.  I made no money in writing my letter to the editor, and I certainly don't demand you pay to read my work.  I am not discounting the cost of writing the standard, just simply saying that if you plan on having expenses paid by publishing your work, you are not simply doing this out of the kindness of your heart.  There is lots of analysis that could be done just on the defense of the standards alone, but is outside the scope of this particular post.

One of the oddest and most compelling arguments for both sides is from Rex Black, in which he notes that about 98% of all testers won't care one way or another.  I think this is true, which makes the standards mostly not matter, but it also means that those 98% who are silent count on us to ensure these are the right standards, lest they become popular and that silent 98% ends up forced into using them.  I fear that this non-involvement is also further evidence that we testers as a group are not acting like a profession.  It isn't that we don't claim to have "extraordinary knowledge" and Dr. Reid at least seems to argue we mostly agree on this knowledge, but rather the majority of people don't feel any need to actively participate.  I realize this might be an argument for why we need a standard -- to show the disinterested the 'right' way to test, but to me it seems to indicate just how young our industry is and seems to me that shows why we aren't ready for a standard.

Even if the ISO body decided these documents ultimately should stand, the objections of the AST/Context Driven community need to be noted in such a standard.  Furthermore, making the document open will go a long way in allowing the community to discuss this document beyond the smaller standard's committee.  I recognize that ISO needs income to maintain itself and won't publish them for free for everyone, but certainly some sort of 'for individuals, not corporations' license could be used (and I don't mean the sort of non-sense Matt Heusser describes).  Finally, if this is an attempt to demonstrate our commitment to professional testing, then it needs to be accessible to our community.  The work needs to demonstrate it's value rather than being buried away inaccessible to those who would use it.