A Personal Failure Analysis: Why it appears CDT has failed

Talks & Failure Analysis

For the last few weeks I have been doing one presentation a week to various monthly groups and to a yearly event.  I have one more planned presentation in June, but this really isn't about my presentations, but rather the audience and fellow speakers.  

The first presentation was a meetup group Isaac and I helped form up in March.  Isaac and I gave the talk and it went really well.  Our audience was maybe 40% high-level QA with a few junior and the rest mid levels.  The talk was about testing an object in black box style.  We gave this talk 3 times total, and each time it seemed to go a little worse.  

When we did it with a group of developers, we moved to more white box/boundary testing style.  We had one person refuse to test out of principle.  In part I think us changing the style was at fault, but two sharper developers really got it, so I don't think it was totally our fault.  Two people really didn't get it until I changed boundary testing into a UI exercise.  This tells me even (junior) developers have a hard time visualizing what code does in their head.  

The final audience was a mixture of testers and developers.  The developers were often dis-interested in our talk and several of them left to go see a different track.  Either they felt applying black box techniques was not part of their domain or that they already knew this and wanted something new.  The irony of that is that 60% of the sessions were marked beginner, so learning something 'new' for a mid/senior developer is challenging at best.

I had a second talk which was a shorter version of my intended lecture I will be giving out of the country.  It was around reflections, something I think I have some level of mastery around, so I felt very prepared for my talk.  I will admit my cadence was a bit off, I probably rushed a little too much and perhaps didn't give enough audience breaks, but it was a code oriented talk.  Given the complexity, that might have been a small part of the problem but in reality, most of the audience were either junior developers or students who in their 2nd-4th year of college.  I had marked my talk as advanced but I couldn't control the people who attended.  This caused me to wonder something.  Why is it we are still talking about TDD, Intro to jquery, etc.?  I don't mean to say we shouldn't teach new people, but why are we mostly limited to beginners?

Why do we retrain people all the time?

In talking with 'experts' in the business, it becomes clear that my question of why do we repeat ourselves is because in some communities, such as testing, people don't stick with it in the long run.  Roughly 5 years appears to be the mean time between starting a career in testing and going to something else.  The second issue is that many people are not particularly interested in expanding their knowledge.  They just want to go to work, do their job and go home.  Don't get me wrong, I don't mean to say you should work 90 hour weeks, but many people have little passion for their work or improving their craft.  

Sometimes managers make these engineers look into 'new' development techniques, which create such a long tail into training that something like TDD continues to need talks 10 years later.  

With that analysis, I want to talk about it with an 'expert', and so I found the keynote speaker whom I spoke to at length.  I don't want to cite his name as I have not asked his permission nor do I have an easily accessible method to communicate with him (I don't have his email address and don't use twitter).  

He is a well-known developer who has a pod cast program with hundreds of different interviews with various people within the software industry.  In our discussion, I asked him if he knew Matt Heusser and he said he didn't.  I asked if he knew of James Bach, Cem Kaner, Rex Black...  He knew of no one I asked of, although having reviewed his interviews, the only tester I recognized that he interviewed was James Whittaker, which was many years ago.  I had not asked if he knew Whittaker.  I did ask if he knew any US experts/thinkers in test, but he declined to answer my question, saying he didn't think that way.  Perhaps my question was somehow confusing.

In speaking with this gentleman, a man who clearly had a wide variety of experience, and seemed to subscribe to the Analytical School of Testing, it became clear to me.  One of the reasons we end up not keeping testers around of long is because we as a group are not well known.  James Bach, in spite or perhaps because of his controversy, has one of the biggest names in testing.  Sadly, managers aren't demanding people start doing 'Bachian testing' (even though I suspect Bach himself would object to that name and concept).  We don't have a glamorous single one size fits all solution, like Scrum or TDD.  CDT is so vague no one but its practitioners knows exactly what it is.  It isn't easily measurable like Scrum or TDD which I can say if we are doing it or not (Are there sprints?  Are there unit tests?).  It makes me wonder if CDT has lost its war when managers can't easily measure if there CDT testers are even doing CDT testing, much less if CDT is working for them.  The metrics (be it sprints or number of unit tests) might ultimately not matter, but managers still at least one security blanket before they are likely to invest in structural change.

Solutions?

I'm not sure I have any useful solutions other than to say this appears to be a community issue.  I have tried doing mentor-ships with mild success.  It however doesn't scale beyond 1-2 people at a time.  I have tried doing lectures, but as soon as I stray past beginner style thinking or just talking about the concepts without talking about implementation, it seems to be clunky at best.  I have started a group with Isaac to see if a more personal monthly audience will help.  Maybe there we can make headway, but I don't know.  One nice thing with it being monthly and having multiple speakers, I might find out if the issue is in fact somehow with me and how I am presenting.  Even outside of presenting, maybe the way I communicate poorly and that just needs improvement?  For that matter, maybe I'm using the wrong language (English) to reach my audience of highly-motivate testers and software developers?  Perhaps there are more motivated testers outside of the US and my issues only revolve around where I live?

Perhaps I need to accept that I can only speak to a handful of experts, but how do I find and group them together?  How can the experts then package this stuff up together in such a way that they might learn from my knowledge?

Outside of myself, I know Kaner is moving to a more teaching oriented approach generating classes in which the students come to him.  Having taken a BBST course and talking to others who have, I know only about 1 in 3 (or less) of those students seem to both try and have the skills to succeed.  Some choose only to pass by the skin of their teeth and others simply don't get it.  Maybe some of it is because the course is in English, but it certainly isn't the only reason.  However, 1 in 3 might be as good as it ever gets, perhaps my expectations are just too high.  Kaner appears to have given up on a education only system by adding in certifications.  It appears to me the ultimate answer maybe that one has to market one's ideas more in order to convince people to change.  Perhaps the reason Kaner is talking about moving to such a path is because he failed to market his solution well enough.  Going back the keynote speaker I spoke to after my talk, he suggested that if he didn't know about the thinkers/experts of test or the ideas of CDT, they must not be that important.  Because if you don't hear about it, it can't matter.

What is a comment? A miserable little pile of words!

A few weeks ago I tried an experiment.  I tried to comment on at least one test-related article or comment every day of the week.  I tried this for about a week and a half.  It was tiring, but interesting, with a result that made me sad.  I didn't do wimpy non-comments like "Great job".  I did in depth, well considered comments, that often took me an hour to develop.  And I got nothing back.  It gives me a new respect for people like James Bach who seem to be all over the Internet commenting all the time and doesn't seem to feel defeated.  As a consultant and trainer, that is a method of advertisement, but I do believe James Bach genuinely cares too.  However, my concerns with comments were developed much earlier...

I am a big fan of  Code Horror by Jeff Atwood who influenced my choice to do a blog in the first place.  While I don't always agree with him, I love his views on commenting.  In fact, he and James had a 'chat' about comments some years ago.  To summarize, James Bach had noticed a major flaw in Jeff Atwood's blog post and wrote his own post on it.  Jeff wanted to continue the conversation but couldn't because James didn't allow for comments.  Now that was years ago and once again, to Bach's credit, comments are enabled.  Victory.  We are all done.  Except...

Cranky Old Man

I still have complaints about how our community uses comments.  Certainly we have useless 'great job' comments without any critical thinking involved.  You can see them in even some of the bigger venues.  Even if you think the blog post is great, and you're their new biggest fan, that sort of comment helps no one.  I won't think about you, my biggest fan.  If you hate my work and yell at me without any particular logic, you're no better than the old man to the left.  However, as a blog writer, am I just a cranky old man yelling at human nature?

I don't think so.  Often Slashdot's comments are valuable and well thought out.  Bach's comments section is curated, often insightful and from some reports perhaps even censoring, but that is a different issue.  I think there are a few issues with our community.  The first is that we write too much.  You heard me.  An author says we write too much.  Ha!  I can read about 5x more than I can write.  When Markus Gartner on a nearly daily basis writes a 1-2 page blog, I think it is nearly impossible to have a conversation about any topic he brings up.  In two days Markus is off into the next subject.  Being 12 hours away doesn't help the conversation, but in the world of twitter, where detail doesn't matter but lip service is paid, you can't develop consensus or new ideas.

Why can't we slow down and as a community genuinely talk through issues?  I am amazed at those whom write daily, and I get that people like Matt Heusser who suggest writing at least 2000 words twice a week.  It isn't an invalid idea for improving writing, but that doesn't mean everything you write is worth posting.  I currently have some 2120 [this article is now published!] articles in draft format, including this one.  Not all of them make it out to you, the public.  I want to give you quality work, and hope to get quality comments back.  I don't write too much, in the hopes that I can learn from the community as I write articles.

But Wait...

What about all those spammers....?

Well to be honest, I rarely see them here.  I think Jeff Atwood might have a solution for bigger blogs, but I am reserving judgment for now.  Truthfully, that would be a good problem to have, as that would mean there are a lot more people caring about QA. 

What if I don't have anything to say?

Really?  You read a 1-2 page article, and found nothing of value to say? Nothing was added to your mental model?  You didn't see any mistakes, any logical leaps too far nor anything understated?  The blog post was so dull it would have been better off being ignored?  Which leads me to...

What if what I have to say is too mean?

Well, for this blogger, I give you permission to write mean things about my, JCD's blog posts.  In fact I WANT you to.  Please, say you hate it and then justify that opinion with facts.

...and if you do write a comment, with insight, which pushes me to write a response, I will try very hard to reply if not write an entire additional blog post on the subject.  So please do!

* Title with apologies to André Malraux.

Look, I wrote something.

Well I haven't been writing much as of late.
However I haven't been slacking...

http://www.stickyminds.com/article/helpful-tips-hiring-better-testers

I've just been focusing on a different venue for my ramblings.

Being a Fraud and a Failure

Often I feel like a fraud.  This comes in multiple sizes and shapes and I think it is often ignored in creative industries such as software because we liken ourselves more to scientists rather than artists.  Artists already know the feeling I am thinking off.  Art & Copy had some of the best Marketer's saying things like:

The frightening and most difficult thing about being what somebody calls a creative person is that you have absolutely no idea where any of your thoughts come from, really. And especially, you don't have any idea about where they're going to come from tomorrow. - Hal Riney

and

I think most of the creative people are so damn insecure that they want to think they know everything, but they know deep in their hearts they're just in deep trouble from the minute they get up in the morning. So if you can tell them "that's what you're supposed to be", that's kind of liberating. - Dan Wieden

When I go to test something, I start out with panic for a second or two, wondering what the hell I am going to do.  Then I draw from my experience and start developing tests.  But that panic for someone with less experience might last a good deal longer.  In CDT, there is a sort of freedom given, because 'it depends' on what your context requires.  I sometimes wonder if it isn't that there is a science, a way of telling what is best to do, not because context gets in the way but because what testing requires is as much art as science.  I may not be a pretend expert, but I know much of my creativity in testing comes from some place that is not just from my smarts.  Sure I have science to back up some of my testing, but I also have 'hunches.'

Hunches are not always easy to swallow when I know they could be wrong.  If I only have time for one feature but two features need testing, I could test feature X on a hunch and fail the entire company when feature Y is broken in a way I had not predicted.  Elizabeth Gilbert suggests we simply shift blame from the personal 'I failed' to a third party entity such as a muse, in which a muse hinted to me, giving me a hunch.  I don't find this a particularly satisfactory way of dealing with my feelings of being a fraud, failing at my job.  I don't like putting credit and blame onto a external entity, so I need another way of dealing with my feelings.

Alternatively, I can read testing blogs where we all collectively struggle to find better ways to improve our chances for success.  This can go all the way into the more scientific side to testing.  If only we had enough data, we could kill off the hunch completely.  The Factory School seems to see it as a problem of Taylorism, if only we could just lock down all the changes.  Document everything, know everything, then hunches aren't needed.

The more I research, the more I think our industry is full of fraudulent feelings with different people attempting to resolve them by adopting 'proven' systems and techniques.  They aren't the fraud, considering their sincere attempts to learn, but they have to find a way to limit failing.   So they turn to these 'experts' to solve your problems.  They have the answers, and you should just trust them.  They provide systems and will teach them to you, often for a fee.  However, if you fail, you aren't doing their system right, not that their system is broken.  You are forced to feel like you failed to heed the words taught, and only if you had worked harder, followed the rules closer, etc. you'd have succeeded.  These tools make us feel the failure and not that the tool failed.  Who is to blame, particularly if the tool often succeeds?

Then we move to team oriented solutions such as Agile.  Agile says it is a team effort, so if a bug is found in production by customers, the team is to blame.  Moving from an "I" to a "We", much like the muse.  This is a paradigm shift, even if not a perfect one.  You now can know if you are a fraud or not based upon the feedback from the team.  The problem is you cannot know if the team is filled with frauds, if the team is just being polite or if they are simply more experienced then you are.  To assume the group will be wiser than yourself, and able to judge your success has its limits based upon who they are.  To assume you are able to judge the team also maybe flawed.  To quote Robert Heinlein,

Democracy is based on the assumption that a million men are wiser than one man. How's that again? I missed something. Autocracy is based on the assumption that one man is wiser than a million men. Let's play that over again, too. Who decides?

Every method is flawed.  I personally like the method Art & Copy suggests.  Rather than blaming a third party or putting your faith in someone else's system, I think it is healthy knowing that sometimes you will fail.  That the feelings of being a fraud comes, but with as many wins, as many bugs as you've found, you aren't just pretending to test, you aren't a fraud.  You really are helping the customers, finding bugs early on, helping resolve issues.  Maybe you do fail, but pick yourself back up and do it again.  However, that is personal preference.  I don't think there is an answer.  Some will like science.  Some will like the sales pitches.  Some will like the art.  Some will like the team.  Some like having no right answer.  They are all fine and all flawed in their own way.

Book Consideration: Rethinking Systems Analysis & Design

To be clear, this is my second book by Gerald M. Weinberg, and I’m not reading the books he wrote in the particular order he published them in.  I like the author’s style of thinking, but in the Introduction to Systems Thinking book, he was very general in his descriptions, hitting a great deal of subjects, with his thoughts.  This of course is a big part of system’s thinking – the idea that you can apply what you know in one field against many different fields using a form of logical thinking and general rules.  While some of the design of the book could have used some refactoring, it was a good start.

In Rethinking Systems, the book focuses mostly on the development of software systems and how this thinking can be directly applied to software.  While some of the ideas are fairly standard (and the book was published more than 10 years ago), it provides some interesting insights.  He starts out by considering analysis vs slogans.  He talks of how we sometimes we over use history as a method for predicting the future.  While history is valuable, analysis will provide data not found in history.  That is to say if X is good, X+1 might not be better, depending upon the attributes that we want out of X+1.  However, we might have a slogan “Bigger is better [for X]”, which might be true up to a given point, where the system then fails to scale for some reason.  He warns of how software developers are particularly vulnerable to certain methodologies, because [in my reading of his opinion] we as an industry don’t have a long history, and even a 5% increase in productivity is great if we consider the number of failed projects in the past.  Unlike the physical world where you have to build a new bridge every time, in spite of it being very similar design, where as we in software always build something new, else we could just copy the previous build for “free”.

Chapter two talks of what actually makes up a [software] system, with a fairly reasonable picture, including the external world, the organization, training materials, training done on the job, wetware, existing files, test data, job control, (and all of this before), the program and hardware (Pg 34).  I thought it was a good point that part of the investment in software is the people and the wetware (the knowing what code is where because you built it which is faster than reading code or documentation).  He also talks about how we need to use our history more wisely, as we don’t often understand why something is the way it is.  While not an example given, I would say the Unicode set(s), is a good example, as they are design for different forms of optimization, but are fairly ironic since they were in fact an attempt to solve a very real world issue ASCII had, as computers were originally designed for English.  I also think of Joel’s amazing article on Martian Headsets (It is worth reading.  Go now, I’ll wait).

Chapter 3 is about how the observer fails often to observe anything of value (including thinking they saw X but saw Y), or at least the wrongs things (like how magicians have you look the wrong way).  In QA this is an important point, and one which I have suffered through more than once.  I actually talked on this a little in a presentation I gave about changing one’s perspective, a valuable thing to do.  He talks about studying the existing system to understand rather than criticize.  This is an important lesson, I hope to take more to heart, as a QA engineer, I have to be careful not to be hurtful to those who created the code, but more of an impartial observer, stating what I saw and why it seems wrong, not just to me, but from a more broad perspective.  In some ways this reminds me of a fair witness.

Chapter 4 talks of self-validating questions, that is to say, questions that require a response that will in fact validate an understanding of the question itself.  He talks of the question “Does that contain special characters?” which gets the answer “no”.  He assumes special characters means alpha + numeric, but nothing else.  In this day and age, it would mean something else, but by asking a yes/no question, without a follow up can get you into trouble.  He speaks of the problem that programmers tend to be defensive in what they do and how they act because users accuse them of deliberately causing trouble if the program says “don’t do X” but the user demands it anyway.  On the other hand, if they just ignore the user and do what they want, they can stubbornly miss important details.  This is a trick problem I also talked about in my talk with dealing with users.  I felt he had good insights and this chapter alone was worth the effort.

The last few chapters are on design of software, including the philosophy, trade offs and the mind of a designer.  I really don’t have much to say about these chapters.  It is not that they are bad, but it was not the most exciting to me.  He talks of being aware of your designing, not to over design nor to ignore the reality of the world.  I find these not exactly contradictory, but certainly a narrow path to follow.  To me, he could have done an entire book on design, but I suspect his strong point is system analysis, not design.  He suggests designing for understanding (know your audience), strike balance between variation and selection (don’t be too original or ground breaking for your audience), etc.  He speaks of trade offs and how to represent them as curves, which is a somewhat novel way of displaying the information, but fairly obvious to anyone who knows that famous triangle with fast, good, cost (pick two).

There are three other things which I would like to mention, but which I have not found (in my consideration of the book) what chapter they were in:

1. He speaks of spell check which is ‘obviously good’, but is it?  It seems to him that the cost of the grievous errors might be greater than the minor typos.  He noted examples where this caused confusion as he had intentional mistakes that were fixed by editors.  He actually complains about one editor using glue and razors to fix an intentional bug in a different book, which I found pretty funny.
2. He speaks of students in his class using systems analysis to gain ‘masters level’ knowledge in a subject and that on average some subjects took just 3 months, but others took longer (he didn’t say how long).  He specifically said English was a subject that took longer and computer science was one of the shorter subjects time wise.  I have my doubts, but interesting none the less, particularly when I recall an (apocryphal?) story of a man who said he could pass any test for his degree.  He was studying something like psychology, but said he could pass any test and thus was given a test in dentistry, which he got a B in, getting his degree.
3. I found it interesting that in the author’s epilogue, he states: “basic human needs… air, water, food, sex… the need to judge other people.”  What I find interesting about it is I have had several talks about this subject with various people, one of whom claims not to judge, when judging is defined as “A person able or qualified to give an opinion on something.”  The author continues, “To write a book… you have to have an uncontrollable urge to snoop and pass judgment… to study what people do and tell them how to redesign their activities.”  It seems to me that judgment is of utmost importance in both the high and the low.  In the high level, we build something we judge to be useful to others, in spite of never having perfect information (making one not qualified?).  This egomaniacal belief that we are good enough to play a sort of demigod, handing design from on high, with the assumption that what we do will be good, or at least better than having not.  We design not truly ever knowing what our users really want and at best we can develop heuristics on this, but not hard and fast rules.  On the low, we design the system’s structures with limitations based upon our best understanding of what will affect it, judging how the system will be used (E.G. Computers will only need English characters) and who will use them, not really knowing what the future will bring.  The trick in my opinion is to be wise enough, not to intend harm with our judgments, nor be too harsh; for often times we end up blinded by the judgment, unable to let go based upon our assumptions.  Judgments, it seems to me (based upon the author’s words) are just another heuristic, not a truth.  In my own view, judgments are a non-binding, unenforceable guesses (with a probability weight behind it) as to how something works, based upon previously observed factors.

In my opinion, I would read the systems thinking book first and then read this book as Systems thinking gives you a broader base of theory, but this book is more practical and does have value.

Why can't anyone talk about frameworks?

In writing for WHOSE, I was dismayed at the total lack of valuable information regarding automation frameworks and developing them.  I could find some work on the frameworks with names (data driven, model driven and keyword driven), but almost nothing on how to design a framework.  I get that few people can claim to have written 5-10 frameworks like I have, but why is it we are stuck with only these 3 types of frameworks?

Let me define my terms a little (I feel like a word of the week might show up sometime soon for this).  An architecture is a concept, the boxes you write on a board that are connected by lines, the UML diagram or the concepts locked in someone's head.  Architecture never exists outside of the stuff of designs and isn't tied to anything, like a particular tool.  Frameworks on the other hand have real stuff behind them.  They have code, they do things.  They still aren't the tests, but they are the pieces that assist the test and are called by the test.  A test results datastore is framework, a file reading utility is framework, but the test along with its steps is not part of the framework.

Now let me talk about a few framework ideas I have had for the past 10 years.  Some of them are old and some are relatively recent.  I am going to pull from some of my presentations of old, but the ideas have at least been useful for one framework of mine, if not more.

Magic-Words

I'm sure I'm not the first one to come to this realization, but I have found no records of other automation engineers speaking of this before me.  I have heard the term DSL (Domain Specific Language) which I think is generally too tied to Keyword-driven testing, but a close and reasonable label.  The concept is to use the compiler and auto complete to assist in your writing of the framework.  Some people like the keyword driven frameworks, but in my past experience, they don't give compile time checking nor do they help you via auto complete.  So I write code using a few magic words.  Example: Test.Steps.*, UI.Page.*, DBTest.Data, etc.  These few words are all organizational and allow for a new user to 'discover' the functionality of the automation.  It also forces your automation to separate out the testing from the framework.  A simple example of that can be given:

@Test()
public void aTestOfGoogleSearch() {
 Test.Browser.OpenBrowser("www.google.com");
 Test.Steps.GoogleHome.Search("test");
 Test.Steps.GoogleSearch.VerifySearch("test");
}

//Example of how Test might work in C#, in Java it would have to be a method.
public class TestBase { //All tests inherit this
  private TestFramework test = new TestFramework();
  public TestFramework Test { get { return test; } }
}

Clearly the steps are somewhere else while the test is local to what you can see.  The "Test.*" provides access to all the functionality and is the key to discoverability.

Reflection-Oriented Data Generation

I have spoken of reflections a lot, and I think reflections are a wonderful tool for solving data-generation style problems.  Using annotations/attributes to tell each piece of data how to generate, what sorts of expectations there are (success, failure with exception x, etc.), filter the values you allow to generate and then picking a value and testing with it is great.  I have a talk later this year where I will go in depth on the subject and I hope to have a solid code example to show.  I will certainly post that up when I have it, but for now I will hold off on that.

...

Okay, fine, I'll give you a little preview of what it would look like (using Java):

public class Address {

 @FieldData(classes=NameGenerator.class)
 private String Name;
 @FieldData(classes=StateGenerator.class)
 private String State;
 //...

}
public class NameGenerator {

  public List<Data> Generate() {
   List<Data> d = new ArrayList<Data>();
   d.add(new Data("Joe", TestDetails.Positive);
   d.add(new Data(RandomString.Unicode(10),  {TestDetails.Unicode, TestDetails.Negative));//Assume we don't support Unicode, shame on us.
   //TODO More test data to be added
   return d;
  }

}

Details

Why is it that we as engineers who love the details fail to talk about them?  I get that we have time limits and I don't want to write a book for every blog post, but rarely do I see anyone outside of James McCaffrey and sometimes Doug Hoffman talk on the details.  Even if you don't have a framework, or a huge set of code, why can't you talk about your minor innovations?  I come up with new and awesome ideas once in a while, but I come up with lots of little innovations all the time.

Let me give one example and maybe that will get your brain thinking.  Maybe you'll write a little blog on the idea and even link to it in the comments.  I once helped write a framework piece with my awesome co-author, Jeremy Reeder, to figure out the most likely reason a test would fail.  How?

Well we took all the attributes we knew, mostly via reflections of the test and put them into a big bag.  We knew all the words used in the test name, all the parameters passed in, the failures in the test, etc.  We would look at all the failing tests and see which ones had similar attributes.  Then we looked at the passing tests and looked to see which pieces of evidence could 'disprove' the likeliness of a cause.

For example, say 10 tests failed.  All 10 involving a Brazilian page. 7 of those touched checkout and 5 of those ordered an item.  We would assume that the Brazilian language is the flaw if all tests failed, as that might be the most common issue.  However, if we had passing tests involving Brazilian, then that seems less likely, so we would see if we could at least establish if all checkout failures had no passing tests involving checkout.  If none had, we would say there was a good chance that checkout was broken and notify manual testers to investigate that part of the system first.  It worked really well and solved a lot of bugs quickly.

I do admit I am skipping some of the details in this example, like we did consider variables in concert, like Brazilian tests that involved checkout might be considered together rather than just as separate variables, but I hope this is enough that if you wanted to you could build your own solution.

Now your turn.  Talk about your framework triumphs.  Blog about them and if you want to put a link in the comments.

Heroes and Villains

Heroes

I have a host of people that I read and read and read.  These are my literary Heroes.  They constantly give me new gems and bright insights.  I pale in my work before these folks.  The sad news is that my heroes have slowly slipped away and all I have is their tremendous bodies of work.  In the blog-arena, I really appreciate Jeff Atwood, Joel Spolsky and Steve Yegge (who has multiple blogs).  Just as an example of my reading, I am all the back to 2005 in Jeff's blog, reading it backwards entry by entry.

Now two things you'll note from that.  One, I really enjoy what developers have to say and I really don't have a lot of test-related bloggers I hit on a regular basis.  Even if I start widening my knowledge net, Martin Fowler, Scott Hanselman and those Dot Net Rocks podcast guys probably has beaten out other test-related heroes I have read.  Now I do tend to focus on what they have to say on test when they have something on test, but I spend way more time on the development process than on capital-T Test.  These guys are brilliant and have wonderful things to say.  They come up with all sorts of clever coding patterns and practices.  They consider the process as a whole and care about the craft.  They are Heroes.

Then I should consider my list of more generic Heroes.  Heroes of science like Richard Feynman, Heroes of science fiction like Robert Heinlein.  Heroes of thought like Rene Des Cartes, Heroes of humanity like George Bernard Shaw.  These are all men I have read and found to have enlightening things to say, although some are rather obscure.

Right now, this is how I feel about test and "Heroes".  There are no Heroes.  There are a few knowledgeable people, but the fractured nature makes it hard to pin down.  Most people who come into test come into it by 'accident'.  We are still forging paths and hitting dead ends.  Yet we do have another aspect.

Villains

Perhaps our lack of Heroes is the nature of our business, so at best we get Villains.  Well, Villains are cool, right?  Who doesn't like a good Super Villain?  The Riddler or Magneto come to mind.  They have super powers, they do cool things and in the end, they often feel justified in their deeds.  I hesitate to call anyone a "Villain" in a community I work in.  Worse yet, CDT might be so accepting that Villains are considered good guys in their own way.  Now if I was James Bach, I would have "“depraved” enemies".  Now does that make someone like Rex Black a Villain in Bach's eyes?  Is the visa-versa true for Black?  Am I a sort of Villain of the CDT community for asking questions?  In talking with one senior-level tester, I could be seen as a Villain.  The all-mighty dollar of consultancy demands for true purity of our testing Scotsmen and my questions for CDT is dangerous for those dollars. I won't answer these questions regarding villainy for you, but I do want to be clear, I am not calling any of these gentlemen Villains.

The downfall of our method is that we don't have any golden boys of testing.  Instead, we challenge each other to get better but don't have an easy way of showing off our skills.  I can't even personally say if Kaner, Bach or Black are good testers.  I can look at some of Atwood, Spolsky and Yegges code and judge them, as that is often visible.  Testing is often an invisible task, not one with an output easily examined.  I think some of the information of the test community can be useful, but I don't use all of any practitioner's 'methods'.

I am left with two things.  Are there any real Villains of testing and can we have Heroes in testing?  How can I evaluate a given person's skill in "Test" compare to writing or management or bug writing or all the other skills that make up "Testing"?  Even with testing an application, it often requires parameters that are hard to control, like the versions of software, builds, time, memory, threading and all the other 'impossibility of complete testing' pieces.  So even with bug reports, test cases, etc. it might be impossible for me to truly evaluate the output.  If I can't know (there are plenty of heuristics, but actually knowing is much harder) someone is a good tester, then I guess I'm just left with is a Villain just a perspective or are their absolutes?

I don't know that I can answer that, but maybe I'll look into getting an outfit, just in case.

Exploratory Software Testing: Tips, Tricks, Tours, and Techniques to Guide Test Design

Exploratory testing, according to James A. Whittaker, is “When the scripts are removed entirely…, the process is called exploratory testing.”  Whittaker’s statement taken at face value assumes that you cannot perform exploratory testing with a script, although he does later revise this to include what I might call “partial scripting” (running a test up to a point and then exploring from there) and Whittaker calls exploratory testing in the small.  He also looks at exploratory testing in the large, which is what I typically consider exploratory testing (which I will cover later on).  In my opinion, the problem with his initial statement and exploratory testing in the small is he is actually describing testing various states just without written steps.  Really, you could easily create a simple table of states and outcomes and it would be just scripted testing.  This to me is not exploratory testing.  To be fair, he later on acknowledges that you can in fact combined scripting and exploratory testing, but that seems to not be covered in his definition of exploratory testing (which I find ironic since it is his primary purpose for the book and as a tester he should have noticed this incongruity).

Exploratory testing in the large, as I described earlier is the act of testing one or more areas by using non-scripted patterns.  Whittaker describes a system of patterns, which he calls Tours.  Tours are, according to Whittaker, a “…mix of structure and freedom.…”  His Tours are designed around either attempts to change your point of view, the methods of testing or the scope of the testing.  Personally, I find the names of the Tours as very poor representatives of the concepts he is trying to put across.  For example, the Money Tour could have easily been called Customer Demo Testing or the FedEx Tour (one of the better name Tours) could be called Data Flow Testing.  I suspect that the reason for the funky naming convention was either so he could use the word Tour (and thus the analogies) or the so that he could make a book rather than a long blog.  While I might object to his implementation, I can’t object to his goal of making better testers.

My other objection comes from the limits of what he ultimately came up with.  He describes to a very limited degree the attempt to see the application from other user’s prospective.  He tells stories of tourists, but I didn’t notice him explicitly link the tourists to users, what the users do and what they expect.  He uses these tourists to ask questions, but not to change one’s bias from one type of imagined user to another.  For me, when testing, I make a great deal of effort to simulate various types of users and to intuit what a given user would do.  If I see a button, I might see 6 different styles of users.  The elderly person who doesn’t get it is a button because it doesn’t look like a button, the expert who wants to middle click the button to open in a new window, etc.  Even then I sometimes go further and imagine not only the users but what their reactions might be to a given system.  What it is that they would want, where they would get confused, where they want the system simpler rather than more complex, etc.?  The book does not seem to address this type of cross-user non-functional style thinking (exploratory testing?) or what type of outcomes might be generated from it.


Outside of the actual techniques addressed in this book, another question left unaddressed is how one manages the various forms of testing in harmony.  I realize this is a complex question, but that is why one pays good money for the book.  To be clear, I don’t mean, what Tours to choose for a given project, which he does address through examples (ironically not written by the author; how many pages did he actually write?).  I do mean, when Tours, when scripting, when automating, or when doing whatever other types of testing the author might have cared to mention.  


NOTE: In this post, I use the term tester as a generic term including Test Engineers, SDETs, Developers writing unit tests, hackers attempting to break in or even customers on a beta product.  In all cases, the person is testing the system.  Only customers on production systems should not be considered testers, since at that point the product should have a fairly low number of bugs, and the customer is not looking for defects.

Unstructured Additional Notes [I wanted to include some unstructured misc thoughts on little bits and bobs that I wanted to make some minor comments.]:

• On page 120, Memorylessness is described, which I think is a huge problem with testing outside of monolithic organizations, but not exactly in the way described.  The problem he describes includes testers forgetting what they tested (a reasonable problem) and testers forgetting what testing techniques find large sets of bugs (a mostly unreasonable problem IMHO and too easily abused by management).
• He should also mention how it is hard to “remember” what is automated already, as that too seems to be a serious issue.  I suppose perhaps his “Testipdedia” is the closest thing he had to a solution, but that seems like a very manual effort or so generic to be of little value.
• The WMP and VSTS (Page 97-111) testing comments were of some value.
• The idea of tracking bugs by % of effort (time wise) to % of bugs found seemed like a good idea (Page 104).  Also the visual test tool falls under a similar area, where testers are able to see where areas have the most functionality, testing/automated focus, bugs, etc.
• I think his commandments where worth reading, even if only 5 pages of the book.
• I found his comments on measuring a tester by the improvements to the development team interesting, but rather hard to measure.  Do you look at bug trends, points completed by the team or what?  To add to that, I think that quality of the design (both interface and code) can be part of a QA’s responsibility, but how do you grade those things?  Even worse, a manager of a large group of developers can’t be sure which tester caused which improvements or if it was indeed testers who did it.
• Whittaker asserts automation needs to be backed up by manual testing, and I agree that automation has a great deal of limitations, and probably needs another 30 years (if ever) to mature before it really provides what has been sold to upper management.
• The comments on correcting Microsoft’s quality problems were some of the better insights, yet the “how are we going to test this thing” with regard to design, doesn’t get  addressed often enough in the book.

WHOSE up for a little skill?

A Brief Summary:

I went to WHOSE, a workshop with a mandate to create a list of skills and am now back.  I want to briefly summarize my experiences.  The idea was generated by Matt Heusser to attempt to have a list of skills that could be learned by anyone interested in a particular skill in testing.  The list of skills was to be put in a modern format and presented to the AST board.  In part I did it because I felt like the CDT community had too little organized detail on how to implement the activity of test.

WHOSE worked on this thing?

I was not present for the first few hours, so I missed out on giving a talk.  When I got there I met my fellow WHOSE'rs:

Jess Lancaster, Jon Hagar, JCD (me), Nick Stefanski, Pete Walen, Rob Sabourin, David Hoppe, Chris George, Alessandra Moreria, Justin Rohrman, Matt Heusser (facilitator), Simon Peter Schrijver (facilitator), Erik Davis (facilitator).

It was a blur of new faces and people I had read from but had not gotten to meet before.  Later, Doug Hoffman showed up.  I was made late when my plane did not actually make it to the airport, having been rerouted.

WHOSE line is it anyway? (Day 1)

This being my first conference, my initial reaction was to keep silent and just observe.  The group had about 200 cards on a table with a bunch of skills semi-bucketed (skills that 'felt' similar).  The definition of a skill was unknown to me still, in spite of the fact that I had researched and considered this problem for hours.  I had also looked into how to model the information, I had considered the Dreyfus model and how it might be used. 

Many of the blog posts I have written were in fact considerations of skills, such as my reflections post, to help prepare me.  I had debated what a skill is with my fellow testers, and even created a presentation, and now I was standing facing what felt like 200 or so skills.  How do you organize them?  Outside of that, what questions do I ask and who do I ask?  Sometimes when a tester doesn't have a lot of answers and no one obvious to ask, who has the time, you just poke around and that is what I did.  I created a few skills I saw were missing and possibly a few that might not be considered skills, or at least not as written.  For example, I wrote out Reflections, Tool Development both which I thought were reasonable and XML which I thought was questionable.  For some reason, as a young tester I found XML to be scary because I didn't understand the formatting and so XML seemed to belong, yet did it?  Eventually the task moved to grouping which seemed to happen while I was still behind a little.  Clearly my impromptu skills were a bit lacking.

WHOSE cards are these?

I wanted to take on the technical piece since I feel like that is the part I can provide the most feedback.  I had written in the airplane a hierarchy of technical skills in the hope that they could use, but feared might be too technical, too 'computer science'.  Having mentioned this, no one seemed enthused to combined the two lists, which I'm not sure if that is for better or for worse.  Having Object-Oriented Programming (OOP) without going into the depth of typing, generics, etc. and how that is useful in testing much less automation or tool smithing seems incomplete.  Coding as the only technical skill involving programming is clearly too small.  Is OOP a good stopping point?  I would have chosen more depth, but I also know there is a limit to what 15 or so people can get done in a few days.

We started through the cards and skipping my list, I acted as a sort of wiki gate keeper (a role I didn't much like) while other people did research on the definition of the skill as well as resources for the skill.  Some people seemed interested in very formal definitions and resources while others liked informal 'working' definitions and web blogs.  I mean no criticism on any particular view point, but I tended towards the informal side.  We ended with a show and tell of our work which seemed interesting.  One group had a lovely format.  Another group had extensive research and a third group had lots of definitions completed.  I noted that if we moved each definition onto its own page no gate keeper would be required.  We closed up feeling a little bit dazed by the amount of work left.

WHOSE on first? (Day 2)

Rob S. had emailed Matt H. over the night suggesting we change the format a little.  Why not make these into a more CDT style.  Instead of having very formal works move to a context-based definition set of skills.  That is to say, skills based upon stories of how we used a skill.  While Matt generated a proof of concept around this, we observed the formatting and tool usage.  Once it was understood, we started writing up skills based upon our interests.  We wrote and we wrote and we wrote.  The question of what skills belong where and what is a skill was pushed aside for a more formal editing process.  XML as as skill was removed even though the original list of 'skills' was saved in a deprecated area.

I wrote somewhere between 10-15 skills over the course of a day.  I know my skills as a writer were stretched that day.  I heard warnings about the flame wars and anger we might see from this venture.  I expect that, people in testing have a habit of finding bugs.  I still have lots of open questions on where this goes next.  I wonder how we will edit the skills.  I wonder a lot about how this process will be managed.  I wonder where this will be posted and how it will be viewed.  Those are still open questions.  Questions I hope to be resolved at some point.  After writing until our fingers bled, we finally went to dinner.  Much thanks to AST for bribing...er... treating us to dinner. :)

WHOSE going to finish this? (Day 3)

We as a group attempt to finish off with figuring out who will finish which skills.  I have a lot of skills still needing finishing.  I know others do too.  I signed up to help deal with the format shifting question so when this comes out it'll be readable in useful formats.  I appreciated Matt's openness in considering methodologies and talking through 'what is next'.  I maybe slower to blog for a while as I work through my WHOSE backlog.  Truthfully, I was not as 'on top of it' the last day as I was the previous two days.  I think exhaustion had finally hit, so I'm glad it was just a half day.

WHOSE missing from this narrative?

This was a rather interesting experience.  I have never been to a workshop before.  I never saw any 'LAWST-Style' workshops before so I didn't have that to compare to.  I have worked with a bunch of bright people before, but not some of the 'experts' of the world (even if they would reject that label...).  That is a little humbling.  Seeing Rob write is amazing.  Watching the speed Matt can break out an article is...well... something to be seen.  In fact, in the spirit of that, I have written this entire article straight and will attempt to limit my editing.  Sorry, poor readers. :)  

The group as a whole had some nice philosophical discussions, but no one got angry, and overall I think that helped make the content better.  Is the content useful?  I honestly don't know, I'm not an oracle, but I hope so.  Is the content done?  No, I hope this to be a living document and that others will get a chance to help contribute to this and grow.  I hope they too can understand that their context and usage of a skill will be considered just as valuable as our context and usage of a skill.

I would like to make a special thanks to Matt, Eric and Simon for setting up this conference.  Also thanks to the AST and Doug Hoffman for feeding us.  Thanks to Hyland for hosting the conference.  For other perspectives, see the following blogs: here, here, here, and here.  One last piece I would be remiss to neglect to mention.  At the airport afterwards I got to have a long chat with both Rob and Jon.  That was a great conversation which I really enjoyed.  I'm still considering some the questions Rob posed to me.  Expect some future blog posts!

UPDATED 12/27: Added more blog post links.

Book Consideration: An Introduction to General System’s Thinking

I think I need to first start this review with a little digression, which I think will be justified, and is of importance.  I personally have a strong interest in the study of how we work, be that sociology, physiology, psychology, philosophy, law, biology, etc.  I have had less "interest" (i.e. time spent studying) in the just as interesting (i.e. fascinating) hard sciences with the exception of computer science, which I have had some time invested.  That being said, I learned long ago that much of what applied in one subject could be applied to another.  I recall a story (urban legend?) of man who told the dean of his college he could pass any written test.  He was given a test for dentistry even though he had studied philosophy (or something like that) and got a B on the test.

How does that have anything to do with General Systems Thinking (GST)?  Well, everything in a way.  The idea of GST is that you can create heuristically rules around the concepts that apply to more than one system of thinking.  It could perhaps be summed up by xkcd.  One field is generally an "applied" version of another field.  Now if the book only talked about this, it would not have been a very impressive book indeed. 

Frankly this is the hardest review I’ve had to write for a book in a long time.  I really would like to gush over it, to say it changed my life, because I could see it doing so.  On the other hand, it really didn’t teach me that much.  The concepts are hard work, but thinking at a high, pure, abstract and with a great deal of rigour is not easy.  I genuinely enjoyed the book, and frankly I felt the author did a great deal of multiple-faceted examples, to ensure the ideas were communicated well.  The questions at the end of the chapter were thought provoking; although sometimes they lacked enough context and expected a teacher to have the materials somewhere nearby.

I loved his willingness to note the problems with science and even with the field under study.  It makes the book feel much more like an honest discussion with a friend than some jerk trying to persuade my view by forcing his on me*.  I would like to take a quote of the author’s out of context, to note one of the places where he didn’t do this, but which I think was a innocent mistake:  "Try to cope with unfamiliar, complex phenomena, we try to…complete view… minimal view… independent view…."  The author gives an enumerated list of possible views a person might take in order to analyze something unfamiliar, yet he seems to be missing some possibilities.  For example, what about the diverse view, where you sample at random a complex data set in order to understand it?  I feel this is an honest mistake as he makes an honest effort to build together a persuasive case, including the acknowledging the flaws.  There is also an ethereal quality that some people have that causes one to trust a person, which for me this author connects with that.

* To be fair, I’ve often been persuaded by said jerks for a short time, until I’ve had some time to clear my head and see just how manipulative their words are.

A few more side notes before I hit my last point.  It is really annoying that the law’s he proposes aren’t recorded anywhere in the book as a whole.  Those laws as a list would have been useful to me, particularly with a little context.  Instead I have to keep referring back to find the exact definition of a given law.  I also found it funny how I kept reading "Brilligance" as "Brilliance", which is in a sense the author’s point (read the book and you'll get this)!  I have to say I found his "Cousins"/"Friends" argument (Pg 156) weak, as I can completely imagine grouping cousins together via relations from a individual basis.  That is to say, A has cousins B and C; B has cousins A and C; and C has cousins A and B.  Thus you can group those people together as cousins.  He never states why he feels that a person must be a cousin of themselves to make it valid.  Maybe he has a point, but it is unclear to me.

As anyone who talks with me often enough will know, I love quotes.  I find the small points of wisdom in them, sometimes with great value.  That being said, I loved this book’s use of quotes and felt it could in fact be highly quotable.  So for the rest of this review I’m going to try to cite quotations and novel ideas worth looking at:

• "In short, we can learn about ourselves, which is really why all of us our playing this incredible game, call it poetry, beads, or, if you will, science" – Pg 143
• "It will be objected…misrepresentations depending on over-simplifications… dilemma of the teacher: the teaching of facts and figures or the teaching of truth.  To convey a model, the teacher must reify and diagram and declare what cannot be seen at all.  The student… approximation of the truth, an approximation [t]he[y] will continue to revise all his [sic] life long. " - Pg 38, Karl Menninger
• Pg 27, Robinson Jeffers, "The Answer" was amazing!
• "If we want to learn anything, we mustn’t try to learn everything." - Pg 105, "The Lump Law"
• (Boy am I guilty of this!)
• "Things we see frequently are more frequent: 1. because there is some physical reason to favor certain states… . 2. because there is some mental reason… ." - Pg 100-101
• "Laws should not depend on a particular choice of notation." – Pg 72
• "…Grown-ups love numbers. When you tell them about a new friend, they never ask questions about what really matters. They never ask: 'What does his voice sound like?' 'What games does he like best?' 'Does he collect butterflies?' They ask: 'How old is he?' 'How many brothers does he have?' 'How much does he weigh?' 'How much money does his father make?' Only then do they think they know him." -Antoine de Saint Exupery, Pg 67
• "Heuristic devices don’t tell you when to stop." – Pg 55
• "A general law must have at least two specific applications… If you never say anything wrong, you never say anything." – Pg 42
• "Method for detecting if something is inside or outside of a given line by counting the number of line in two direction (up/left, down/right) and seeing if the count is even or odd" – Pg 145
• While not specifically talked about, why is it that we can’t pass through glass (it is solid) but light can, yet light does not pass through everything, else we would never had shadows?
• The author believes his system will fail to become popular, or become diluted with new views, which eventually will reverse the creators...  Then why do people create?  Why not just give up?  Does he only fear an evolution or would a revolutionary new view also be just as bad.  He claims people can only adapt (roughly) once in a given system before they become too attached to their system.  Why?  Can it be avoided or would people who avoid it not have enough passion to get the first system in place?  Is this a case of people dying too soon to adapt again?
• "With respect to a given transformation, there are those properties that are preserved by it and those that are not." – Pg 154
• What of money to an item?  What of the happiness in having money?  Compare that to the happiness of spending it?  On others?  On yourself?  Are those things equal even though the property might remain?  What if the other converts items back to money (pays you back)?  Is the transformation of the first dollar equal to the last dollar in your pocket?
• "A system is a collection of parts, no one of which can be changed." – Pg 162
• "We cannot with certainty attribute observed constraint either to system or environment." – Pg 214
• "The number of untested assumptions in science is staggering.  On any say we can open any one of dozens of newly arrived journals and find reports of "discoveries" that were made simply by relaxing a constraint…" – Pg 215
• I find it interesting that the author assumes that the untested constraints are simply made by science because of ignorance of the constraint rather than by necessity of time or other possible variable(s).
• "1. Why do I see what I see? 2. Why do things stay the same? 3. Why do things change?" – Systems Triumvirate, Pg 228
• "...Science comprehends the thought of reality, but not reality itself; thought of life, not life itself.  That is its limit, its only really insuperable limit, because it is found on the very nature of thought, which is the only organ of science." – Bakunin, Pg 229