Tuesday, November 4, 2014

Consultant's War

My co-contributor Isaac has been pondering the fact that we see more consultants blogging regularly, speaking their mind and controlling the things we talk about.  I don't care if it is  Dorathy "Dot" Graham, Markus Gärtner, Matt Heusser,  Michael Bolton, Michael Larsen (Correction: As Matt pointed out, I got my facts mixed up and Micheal is not a consultant), Karen Nicole Johnson,  James Bach or Rex Black.  There are dozens of consultants I could have listed (I mean, how could I have missed Doug Hoffman!), some of whom I have personally worked with.  So this is not meant to be a personal attack on any of them.  I realize I list more CDT folk than others, and while that is not intentional, I read less of the thought leaders from the other schools.  If you look, I bet most of the people you read either are or were consultants [In interest of full disclosure, I have written a hand full of articles and have been paid, but I am not nor have I ever been a consultant, and with you reading this, it is at least one counter-argument, but we'll get to that in a bit.].  If you made it here, there is little doubt you've read at least of those author's works at least once.  They are stars!  I mean that literally.  I found this image of James Bach just by searching his name:
From: http://qainsight.net/2008/04/08/james-bach-the-qa-hero/
I didn't even use image search to find that, it came up on the first main page of my Google search results!  I respect James, he writes well and has interesting ideas, but I have no opinion on his actual testing skills, as using his own measure, I have not seen him perform.  Keeping that in mind, I have not seen any of the above consultants do any extended performances in testing, even if I have listened to many of them talk and gained insights from them.  They know how to communicate and they are often awesome writers.  Much better than I am.  They all are senior in the sense that they have done testing for years.  Some of them disagree with others, making it often a judgment call of their written works on who to listen to.  Some of it is the author's voice.  In my opinion, Karen Johnson is a much softer and gentler voice than James Bach.  But some of it is factual.  I have documented several debates between Rex Black and various other people I have listed.  Rex has substantially different ideas on how the world should work regarding testing.

 ISO 29119

James Christie, another consultant brought up a standards in CAST 2014.  It was a good talk and clearly he had some valid concerns about the standards that have been created.  I have talked about that earlier (I am not going to include as many 'justifying' links as all my points and links are in the earlier article).  I am actually not terribly interested in the arguments and in reality, ISO 29119 is more of a placeholder than the actual point of interest.  I am more interested in discussing who is creating these debates.  In part we are in a war between consultants and in part we are in a war of 'bed making'. Clearly the loudest voices are from those who have free time and have money at stake. The pro-ISO side has take the stance of doing as little communication as possible, because it is hard to have a debate when one side won't talk. It is a tactical choice, and it is very difficult to compromise when one side doesn't speak. Particularly when the standard is already in place, the pro-standards side doesn't have much to gain from the messy-bed side who wants the standard withdrawn.

Both groups of consultants have money to gain from it.  A defacto-RST-standard would make Bach a richer man that he already is.  Matt Heusser's LST does give at least some competition of ideas, but still that isn't much more diversity. The standards would probably make the pro-standards consultants more money as they can say, "as the creator of the standard I know how to implement it."  Even if they made no more money in so far as making the standard, they will be better known for having made the standard. Even if I assume that no one was doing this out of self-interest, the people best represented are the consultants and to a lesser degree, the academics, not the practitioners who may feel the most impact.  Those leading the charge both for and against the standard are primarily consultants or recent ex-consultants. Clearly this is a war for which the people with the most time are waging, and those are the consultants. Ask yourself, of those you have heard debate the issue, what percentage are just working for a company?

Granted, I am not a consultant, but it takes a LOT of effort to write these posts.  It isn't marketing for me, except possibly for future job hunting, and the hope that I will help other people in the profession.  I know non-consultants are talking about it, but we don't have a lot of champions who aren't consultants.  Maybe most senior level testers become consultants, perhaps due to disillusionment of testing at their companies.  Maybe that is why consultants fight so bitterly hard for and against things like standards.  Perhaps my assertion that money at the table is a part of it is just idle speculation not really fit for print.  I can honestly believe it to be that is possibly the large majority of the consultants involved.

 Bed: Do you make yours?

Then what is it that causes these differences of view?  Well let me go back to the bed making.  To quote Steve Yegge:
I had a high school English teacher who, on the last day of the semester, much to everyone's surprise, claimed she should tell just by looking at each of us whether we're a slob or a neat-freak. She also claimed (and I think I agree with this) that the dividing line between slob and neat-freak, the sure-fire indicator, is whether you make your bed each morning. Then she pointed at each each of us in turn and announced "slob" or "neat-freak". Everyone agreed she had us pegged. (And most of us were slobs. Making beds is for chumps.)
That seems like a pretty good heuristic and I think it is also a good analogy.  Often those who want more documentation, with things well understood before starting the work and more feeling of being in control via documentation are those who feel standards are a good idea.  They want their metaphorical beds made.  They like having lots of details written down, they like check lists and would rather make the bed than have a ‘mess’ left all day. A nice neat made bed feels good to them. Then there are the people who see that neat bed and think it is a waste of time at best. At worst, they think that someone will start making them make their bed too. Personally, I think "Making beds is for chumps", just like Steve Yegge. Jeff Atwood would go even further and call it a religious viewpoint:
But software development is, and has always been, a religion. We band together into groups of people who believe the same things, with very little basis for proving any of those beliefs. Java versus .NET. Microsoft versus Google. Static languages versus Dynamic languages. We may kid ourselves into believing we're "computer scientists", but when was the last time you used a hypothesis and a control to prove anything? We're too busy solving customer problems in the chosen tool, unbeliever!
Clearly the consultants, many of whom I take valuable bits of data from care about what they do and they tried to lock down their empirical knowledge into demonstrable truths.   But that doesn't mean they have 'the truth'. I know as testers we try to have controls, but I am not so convinced we have testing down to a science.  It is why I feel we aren't ready for standards, but I also recognize the limits of my own knowledge.

For what it is worth, I tend to be against bed making, I think having all this formal work and making checklists is rather pointless unless they fit what I am doing. Having one checklist to rule them all with a disclaimer that YMMV and do the bits you want doesn’t sound much like a standard, but those whom like their bed nice and neat probably feel very happy when they come home at night. The ‘truth’ about the value of making your bed, the evidence that it is better is less than clear. Maybe one day we will have solid evidence in a particular area that a particular method is better than another, but we aren’t there yet in my opinion. But still I don’t make my bed.  In case you are wondering, I think this is probably one of the hardest questions we have in the industry:  What methods work best given a particular context and what parts of the context matter most?


  1. I like the way your post ended, JCD. It's a good question, and I've spent a great deal of time thinking on it, and no, I do not have a comprehensive framework. :-)

    As for the comment about consultants, I have to say a little bit of ouch. First off, Michael Larsen is not a consultant. Never has been. He writes because it helps him learn, at least that is what he says, and I have no reason to doubt him. (Though I suspect he'd agree that building a reputation for potential future jobs is probably some small part of the equation.)

    If you look at my writing, I was speaking at two, slowly becoming four conferences as early as 2004, and blogging heavily 2006-2011. It was only in 2011 that I left the day job behind to consult, and, I have to admit, it doesn't feel great to hear that consultants may write because they are greedy and/or have too much free time. I was writing and speaking years before I became a consultant, which, I guess, means I either had too much free time or am /very/ ego-driven. :-)

    I suspect there are more charitable interpretations to that, but I suppose the job brings with it the accusation by it's nature.

    Anyway, I'm pleased to see LST listed as competition to RST, even if in a minor sense. You should take the course, you'll like it. :-)



    1. I think it is not just a good question but an important question for our future. Computer Science is starting to think about this type of question, as can be seen by the recent Slashdot article on computer languages: http://developers.slashdot.org/story/14/11/05/0530228/the-effect-of-programming-language-on-software-quality/ . One day we, the testing community, might be able to come up with some decent studies for this around problems such as what methods work better in particular contexts. I think it will take time and effort, not to mention a receptive community.

      Thanks, I have noted the correction! :) But that is part of the meta point I was making, that one has to be aware of who is making the arguments and what they might have to gain from them, not to mention how much you trust them.

      I appreciate that you actively manage your biases but I suspect that at least some readers don’t put much thought into the biases of a given author. To be clear, I don't know that any consultant is fighting for anything but what they believe is right. However, human psychology suggests there is a lot going on in the back of the brain that may or may not have an affect, even when one doesn't know it. Of course, as a non-consultant, maybe it is I who have a biased view!

      Even if there is no link between money and opinion, the activities a consultant does are not the same as someone who works for the same company for years. It can't be helped that viewpoints are going to be different. This isn't a bad thing. This just means more people who don't work in consulting need to work on giving to the community.

      Perhaps those who do contribute ultimately become consultants, and this is a cycle that will just exist. Maybe that is the fate of most prolific authors and thought leaders. Hard to say. That is an alternative interpretation I do give some strong credence to, including in my blog post. It doesn’t stop this fact that consultants often drive what is discussed. One blog I read that caused me to write this post suggested RST to replace the ISO standard. That worries me. The James Bach superhero picture also worries me. It suggests Bach might have accidentally become a sort of alternative to a standard. That in my head makes James Bach a standard, even if he would reject that label. Then I saw how many other consultants were arguing against the standard and I re-realized how many people I regularly read who are consultants. It worries me. Maybe it is nothing, maybe it doesn't matter, but I think people should be aware of it.

      Maybe I will some day. Maybe someday you’ll come out here.

      - JCD

  2. Great analogy JCD!

    I am a slob, I don't make my bed and I don't like standards and checklists. The main reason I don't like standards is that they are antithetic to continuous improvement other reasons are described here:


    On the other hand, i personally don't believe that consultant are rejecting the standard for financial reasons, or at least I am too naive to think it.

    1. Thanks, I think it often explains the actions of different view points. I am not saying one side might not be right, but without a lot of science, it seems from an outsider viewpoint to be a matter of taste. Perhaps that is naive and there is in fact evidence demonstrating one idea over another. But even finding such evidence is difficult: http://xkcd.com/1445/

      It isn't that I am saying consultants are intentionally trying to lie to us for money. Rather, it is in their interest to talk loudly just to be noticed. They may completely believe what they are saying is right. Even if they never intentionally use their platform for advertising (as Matt says above with some reasoned evidence), that does not mean we should at least keep it in mind. To ignore evidence because you don't want it to be true is a logical fallacy. http://en.wikipedia.org/wiki/List_of_fallacies . Recall how we argue that when developers say 'no one would do that' they mean 'no one I like would do that.' Keep that in mind.

      Even if we ignore the money aspect all together, how often have you asked why an author wrote what they wrote? A large part of why I (consciously) wrote this article is because I saw someone say that standards should be replaced with RST. That profoundly disturbed me and was a platform for me exploring the role of consultants on our thinking. It wasn't as though they said 'training with things like RST, BBST, LST, etc', they said something like 'replace the standard with RST'. Horrified with that, I explored the role of consultants, and to exclude or ignore money as a possible reason why they do what they do seems naive. I think it could and often is more that they have more years of experience and happen to have more passion than the average tester. That seems like the majority reason for most consultants I know.

      Then again, what if I asked the same question about Dr Stuart Reid, convener of the ISO 29119 Working Group? Would some people jump up and down claiming it was money, power or advertising related? It feels implied here (http://www.developsense.com/blog/2014/09/dramatis-personae/ ) that having consultants is some how different than working for say a charity. Quoting from the link above [with formatting changed]:

      "A charity. Technically true, but Ms. Veeraraghavan very recently resigned from a large testing consultancy; a lucky break for Dr. Reid in terms of the timing of his response to the petition."

      Why is it a lucky break? What makes it lucky? Is it lucky because Dr. Reid would have gotten his facts wrong otherwise? It's an interesting article analyzing who worked on the standards, but not why it matters other than the obvious years of experience and in what contexts. Well the article ends with a link to http://en.wikipedia.org/wiki/Cui_bono . I think my title and description are reasonable considering.