Code Camp Presentations

I present two hours at code camp earlier this month on the topics of blackbox and whitebox software testing.  The two topic were broken out in a way that anyone could attend one session or the other and get value out of them, but both was better.  Blackbox Testing was the first hour and Whitebox Testing was the second hour.  For those who did not attend my presentation, I should note you maybe missing some context that I gave during my talk but was not recorded in the slides.

One important aspect not in my slides but I told the audience was my blackbox talk was heavily influenced by the BBST courses developed by Cem Kaner and the book Lessons Learned in Software Testing.  So if you have questions, please feel free to bring them up in the comments below.

Blackbox Testing Abstract

Blackbox Testing involves testing software without knowing the internals of the code. This is a survey presentation, covering a broad set of topics meant to expand your interests and provide self-study opportunities. We will cover: 

* Schools of Thought: Where we have been, why testing has changed

* Testing Missions: How to know what you should test 

* Testing Strategies: How to make your testing organized 

* Testing Tactics: How to make your testing better

This will include practical examples as well as theory. You do not need to attend the whitebox session to gain value from this talk. However, these presentations are meant as a pair and will not cover the same material.

Whitebox Testing Abstract

Whitebox Testing involves testing software with deep knowledge of the internals of the code. This is a survey presentation, covering a broad set of topics meant to expand your interests and provide self-study opportunities. We will cover: 

* Schools of Thought: Where we have been, why testing has changed. 

* Limits: Why you need both the light and dark sides. 

* Techniques: Static Analysis, Security Analysis, Unit, Integration and System Testing 

* Tooling to make that blackbox system more white. 

* Automation techniques, including both white and blackbox techniques.

Almost none of the content here will be a repeat of the Blackbox Testing presentation. While it is not required, it is recommended that you attend the Blackbox Testing presentation.

Having too much to do

"Hello my name is Isaac, and I'm a task addict."

Do you thrive in an environment of chaos?
Do you have a list of ‘would like to learns’ that is at least a years worth of work for an unemployed person?
Everytime you talk with someone do you write down at least one new thing that you would like to learn / research / reaffirm your understanding of?

That is a world I constantly live in. Every project requires me to learn new things, new tech, new ways to interact with people and new ways to do all of those things.

My current list is stored in 5 different places, that's how much I have, I have an iPhone version, a moleskine notebook, Evernote, Todoist and finally my work backlog. (The work backlog is slowly transitioning to Evernote and Todoist)

I've tried several different versions of organization (Get Things Done, Kan-Ban, Backlog) but none of them really work for me. I haven't found the idea of a prioritized backlog all that useful.

Currently I'm trying to write a blog, start up a Tester group, help a couple of people with larger scope test community projects, volunteer to help with BBST, run an elementary school carnival, volunteer in my child's classroom, plus a reading list of 20-30 blogs, 8-12 articles, my stack of books is overwhelming (literally, it fell over as I tried to count them) and then my list of word definitions, small learnings and topics to research is 100+. Garage full of projects. 101 things to do this year. Building a fort. Expand chicken coop. And none of this really even just deals with normal life maintenance; mow lawn, do dishes, feed offspring, clean house, entertain the children or pay attention to the wife.

I have found that when my brain says “oh, what about…”, and I have the time, space and/or resources to dive myself into that topic immediately. I can focus and delve deeply into a subject, and stay there for hours on end sometimes bordering on mania. (What Chimanski calls Flow). If I am unable to dive into my minds chosen topic immediately my mind wanders for a time. Eventually, it settles on something else it finds interesting, and the cycle repeats, until I am capable of doing something that my mind wants to do.

I have been attempting to find ways to narrow down or show my mind the path that I think would most benefit me at the moment, and sometimes I can grind my mind into something. Usually my mind wins, and I wander.

Also, when I don’t have this massive list of things to do I find myself swimming in a sea of boredom, never able to accomplish the next thing, cause my list of potentially useful things is too small to find something that strikes my mind as interesting right now. As I tend to task change every 15-90 minutes unless I get into a flow.

People are always saying "you have too much, you'll burn out". But that's not quite right, I have found that when I don't move at light speed for 8-10 months...I just don't move. But then there are those periods where nothing, just nothing seems to interest me and I am required to recharge, vegetating on kung-fu, zombie flicks and some random luddite project (woodworking, blacksmithing, calligraphy… you know the normal stuff ;).

Strengths finder tells me it's Intellection. Briggs tells me it's my Intuition and Thinking pieces. Both pieces to me imply that after I've spent a fair amount of time doing massive amounts of thinking / doing, I need time off to ponder and study, connecting things internally before I can speak them outloud to other people.

Or maybe I just need to recharge mentally…


As an aside to prove my point, this blog post is from 2013, yet I couldn’t find the words to do the topic justice till just now, cause my brain would look at the article and go, meh not now.

Expectations...again...

This was written a long time ago...but finally edited to readability.

So I just finished the AST BBST Instructors course. In it, we started by introducing ourselves. I of course summarized my introduction from this blog cause I was being lazy, and it said what I thought I wanted to say.

In response to that I was asked:

"It sounds like you have the confidence and experience to be a manager, but I wonder if this might make it hard for you to teach? What do you think?" - Another Student

I responded with:

"Yes I find that my mental model has a hard time bonding with people who aren't also self-starters and/or newbies. However I've been specifically working on how do we 'as a testing community' get the first level of people into testing. I still have a problem dealing with people who just want to be spoon-feed information.
My experience is mainly in taking people with 3-5 years of experience and tuning (yes tuning) them into the next skill they need to be the best at their current position. However so far this has been limited to where I work, as I usually understand the context enough to say "if you could do this" it's the next skill you should learn.
So part of my goal is to be able to teach less experienced people. But I still struggle with if I should TRY to deal with non-self-starters…"

I’d like to take a more detailed look at my personal thoughts on the matter. Specifically around, “Does your own attitude and personality cause you difficulty in teaching people?” I’m going to try and detail an internal dialogue I’ve been having about this lately. I first tried to sum up some feelings in an older blog about Expectations.

This is what I mean when I say, "not-self starters". There have always been people at places I've worked that are comfortable doing just the basics, show up at work, do their job, go home. These are the people that when asked in an interview "What do you do to improve your skills", have an answer similar to "On the job training". These are the people that don't seem to want to take time outside of work to improve their skills (which to me translates to "I don't consider this job a career"). Some of these people seem bright, others are merely content at getting along and that's enough for them.

Then there are the self-starter people who will go out of their way to learn new things. These people ask you questions, and when you say, "tell me what you know about it so far", they blow your mind with the research they have done. Or alter your understanding of the subject with some new pieces that you haven't heard or seen of yet. Sometimes at a minimum they have clearly researched and/or understood the material, but don't change your level of knowledge.

I know mentors / leaders want to use terms like taught, educated or instructed. But when you have self-starters, that isn't the appropriate wording. The closest I've come is tuning them. When I chat with a self-starter cause they are asking "What should I learn next?" (and them asking is one of the key points). There are a couple of ways to encourage them. And generally that is all you can do. Point in a direction, say 'that way', and then get out of their way.

• Have them learn a new skill. Sometimes these are easy for beginning self-starters. Don't know SQL, yep I have yet to meet any tester that doesn't benefit from knowing it. As the self-starters get more and more experience, this can get harder and harder to find for each individual. Currently I've had serious success with just allowing them the freedom to find new things. This is the real power of the self-starter, they aren't okay with sitting idly by and surfing reddit, they WANT to provide value, they WANT to solve problems.
• Have them level up a skill they are already strong in. This helps those who have just finished something rigorous, demanding or seriously mentally intensive. (This can be work related like finishing a project or a mentally challenging class, or a non work related life changing event.) It allows them to learn something, but with the permission to not be as intensive with it. An example would be having someone learn the singleton pattern, instead of a whole new programming paradigm like LISP.
• Have them up a skill they are weak in. I don't normally recommend this for very many people. I tend to follow the Good to Great idea, that weakness isn't inherently bad. But there are genuinely some weaknesses you need to either compensate for, or bring up to a minimum bar.

Sometimes even that isn't enough. I recall one situation where I was attempting to tune someone I classify as a full-blown self-starter. We were going over the OSI model, and I was attempting to explain how each layer could be an attack point for testing. Unfortunately we were not speaking the same language, after about 30 minutes, they were frustrated and on the verge of crying. I was frustrated and not understanding why they couldn't get the 'simple concepts'. We parted that day, and I don’t recall us ever trying to train together again. I took this as a personal failure…how could I not explain such simple concepts (which OSI is and is not) to someone who tries really hard when they self-educate and generally succeeds? It took me a long time to understand that I had a base of knowledge about computers that they had never had exposure to. Bad on me for assuming what they knew. The problem was that I know stuff they didn't. They also know stuff I don't. It takes time and effort to truly get to know people. I hadn't taken the time or maybe didn't have the perception to understand that they had no idea what I was talking about. They didn't have the trust to tell me 'What the hell are you talking about'.

S'long as you learn to temper what you tell people you want, with realistic acceptance of what people can really accomplish. It all comes down to do you trust your people to work hard, do they trust you to not hose them with unreasonable expectations. It takes some serious time to build that trust. It's why you see people follow a strong leader around to companies.

If you don't fall into the category of self-starter as I've laid out here. There is still hope, you can become one. Start today, motivation breeds motivation, find something interesting, and learn it. When you're done learning that, find another. Then repeat ad nauseum.

I've going to wrap up and caveat this entire article with a thinking exercise for you.
Can your expectations be too high? Is this a bad thing?
How else would we ever achieve something new?

Leveling Up Your Testing Skills

Recently I gave a talk intended to be a quick survey of some of the core areas of testing.  I certainly didn't hit everything, but meant to give people tools to go exploring on their own.  I used a text editor for my presentation, because not everything needs power point.  I thought it might be useful to others so I am recreating it here.  I have done some minor modifications, removing things that only made sense in context of a performance.  I also added a few things based upon the discussion during the presentation.

 ----- Page 1

Survey:

Who has heard about AST?

How many of you have read or even skimmed a testing book in the last year?

How many of you have read a white paper?

How many of you have written ANY bugs outside of professional obligations (your job, school, etc.)?

Has anyone tried to formally QA an article?

How many of you have heard about the “No Tester” movement?

How many of you have heard about ISO 29119?

 ----- Page 2

Purpose:

My purpose is not to teach you facts about testing, even if some of that happens, but rather to teach you what questions you didn’t know that you should even ask and show you places you didn’t know to look.

NOTE: I know I cite myself several times, and while there are other sources, few are as narrowly focused to the strategies I am going to talk about..

 ----- Page 3

Do you know the different ‘schools’ of testing?

Four Schools of Testing*: www.testingeducation.org/conference/wtst_pettichord_FSofST2.pdf

Analytical School - Code Coverage, Unit Testing

Factory School - Metrics, Traceability

Context-Driven School: Exploratory, Multidisciplinary based upon context

Quality Assurance School: Protect user, Gatekeeper, Process-Oriented

* [Edit: Since this presentation, I have discovered an updated model including a 5th school, Agile.  See https://www.prismnet.com/~wazmo/papers/four_schools.pdf)

 ----- Page 4

I’m mostly going to speak from the CDT point of view…

(Aside: While I appreciate CDT’s philosophy, be aware that if you approach the community online, some people participating in CDT’s development tend to be passionate/challenging/aggressive and some are very interested in defining words.  Even if you are not interested in that, CDT is still worth looking into, just figure out who you find value from and whom you don't get value from.)

  

 ----- Page 5

Two of the largest “CDT“ classes:

BBST: http://bbst.info/ ; http://www.testingeducation.org/BBST/ ; http://www.associationforsoftwaretesting.org/training/courses/ ; http://altom.training/bbst-foundations/

RST: http://www.satisfice.com/rst.pdf

(Both have free portions, BBST includes all the lectures for free)

  

 ----- Page 6-8

Models and Heuristics:

What are they? Read my work: Words of the Week: Heuristic [& Algorithm]: http://about98percentdone.blogspot.com/2013/10/words-of-week-heuristic-algorithm.html

“A Heuristic is an attempt to create a reasonable solution in a reasonable amount of time.  Heuristics are always Algorithmic in that they have a set of steps, even if those steps are not formal.” 

List of Heuristic Mnemonics: http://www.qualityperspectives.ca/resources_mnemonics.html

SFDIPOT - Structure, Function, Data, Integrations, Platform, Operations, Time

HICCUPPSF - History, Image, Comparable Product, Claims, User Expectations, Product, Purpose, Standards and Statutes, Familiar Problems

Heuristic Test Strategy Model: http://www.satisfice.com/tools/htsm.pdf

How do you know when you are right? http://about98percentdone.blogspot.com/2013/11/how-do-you-know-when-you-are-right.html

Models:

Session-Based Test Management: http://www.satisfice.com/sbtm/index.shtml

Thread-Based Test Management: http://www.satisfice.com/blog/archives/503

Tours: http://www.developsense.com/blog/2009/04/of-testing-tours-and-dashboards

   

 ----- Page 9

Books (just a few):

An Introduction to General Systems Thinking: http://www.amazon.com/exec/obidos/ASIN/0932633498

Exploratory Software Testing: Tips, Tricks, Tours, and Techniques to Guide Test Design: http://www.amazon.com/Exploratory-Software-Testing-Tricks-Techniques/dp/0321636414

Lessons Learned in Software Testing: A Context-Driven Approach: http://www.amazon.com/Lessons-Learned-Software-Testing-Context-Driven/dp/0471081124/

Agile Testing: A Practical Guide for Testers and Agile Teams: http://www.amazon.com/Agile-Testing-Practical-Guide-Testers/dp/0321534468/

 

 ----- Page 10

Self-based learning

Go find a testing job for the weekends like:

http://www.utest.com/

http://sourceforge.net/p/forge/helpwanted/testers/

http://www.guru.com/d/jobs/

http://weekendtesting.com/america

   

 ----- Page 11

Where are you going?

http://blog.red-gate.com/wp-content/uploads/2014/02/Test-Engineering-Skills-v3.pdf

(Go look at this guy, it is an interesting list of skills)

   

 ----- Page 12 & 13 

Experts and active members of the community:

There are different people who have different expertise’s that are worth reading.  Not everyone does much automated testing, while other people are more interested in coaching or test design.  Look at where your strengths are and find people who will be able to supplement them.  Some of these people may in fact oppose my own views, but that too can be valuable.  Debate helps you understand your own point of view.  Here is an example or two….

James Bach* – Testing is an Intellectual Inquiry

James Christie* — Consultant, Professional Auditor and Tester, Started entire ISO 29119 discussion

Jon Bach* — Session-Based Test Management (with his brother)

Michael Bolton — Testing vs Checking

Huib Schoots — Dutch Testing Coach

Scott Barber — Load Test Guru

Rex Black — Proponent of Factory School of Testing***

Harry Robinson — Model Based Testing

Justin Rohrman* — Active in community, board member of AST

Robert Sabourin** — Author of I Am Bug (pictures by his 12 year old daughter)

Jerry Weinberg  — Systems thinking, consulting, author

Karen N. Johnson* – Professional Tester’s Manifesto

Fiona Charles* — Author, Speaker

Matt Heusser* — Founder of Miagi-Do

Cem Kaner — Originator of CDT, BBST, Legal concerns around testing

Alan Page — (Former?) Microsoft Test Architect

* I met these folks at CAST, and you too can meet these sorts of folks at CAST.

** I met Rob at WHOSE, I enjoyed his writing at WHOSE but he tends to be more academic in my experience.
*** [edit:  Since publication my understand of Rex Black's ideas has evolved. Rex specifically rejects the concept of Schools and, as best I understand would prefer I say he's a Proponent of the Strategy of Factory Style Testing in some contexts.  I leave the original work as stands for historical reasons.]

(Some of these people are accessible to talk with, some only speak through their work, some through conferences…)

    

 ----- Page 14

An Exemplar (with more details): 

Doug Hoffman

http://www.softwarequalitymethods.com/h-papers.html

(pay attention to the dates of the papers, some have updated versions)

He has written exhaustively around automation and testing.  While that isn’t the only work he’s done, you will get a good idea of the depths you can go in with automation.  On the other hand, if you want to learn to code, Doug is probably not the man to visit.  He’s interested in theory behind automation.  He was one of the minds behind the ideas High Volume Test Automation and Exploratory Automation.

I wouldn’t go to Doug to find out the latest idea behind Selenium.  I would go look at Doug’s work to understand what sort of automation techniques might make sense to apply to a problem.

    

 ----- Page 15

Questions you need to ask (that I can’t answer):

How much time am I willing to dedicate to learning?

What sort of things do I want to learn?

How varied do I like my studying?

Is this a job or a profession?

Do you want to do something that matters?

Are you okay with having 10 years of experience that really only add up to 1 year of valuable experience?

 ----- Page 15-17

Questions you need to ask (that I can give hints on):

Where can I learn?

 - Blogs

 - BBST/RST and other classes (LST is another CDT class, RBCS is from the factory school)

 - Books

 - Co-workers (particularly challenges)

 - Conferences (CAST - https://www.youtube.com/playlist?list=PLQB4l9iafcelXpJnK6IyDsoFeEb1icqrl , Code Camp, Star West, Star East, Agile testing days, GTAC - https://developers.google.com/google-test-automation-conference/2014/stream , etc. ; Often recorded on Youtube)

 - White papers: http://about98percentdone.blogspot.com/2015/01/white-papers-and-who-owns-your-education.html

What should I learn?

 - Start by glancing at the foundations class.  See how much of it you know already.

 - Read blogs about what interests you.

        - Look up the people I talked about (I suggest Googling "<Name> Testing”.  Almost all of them have blogs).

        - Look at the top blogs and see what they have to say: http://www.testbuffet.com/lists/top_114_software_testing_blogs_2014/page/1/

        - If those methods don’t work, try looking at the fire hose and see if there is an interesting topic: http://www.testingreferences.com/testingnews.php

 - Start your own blog and learn to write; here is mine: http://about98percentdone.blogspot.com

       - Feeling shy?  Why not start by writing comments.

 - Pick up one book.  Read it.  Don’t like the ones I listed?  Check out this extensive list: http://www.testingreferences.com/software_testing_bookstore.php

   - Write a review on your blog

 - QA a news article.  Write up bugs on failed assumptions.
      - http://about98percentdone.blogspot.com/2013/09/testing-babies-for-learning.html

Learn about your interests AND figure out how much you are gaining.  If your energy wanes, move to a different topic or take a break.

What else can I do?

- Figure out what matters and work on that: http://about98percentdone.blogspot.com/2014/10/noise-to-signal-attempt-to.html

- Don’t keep working in the same position for too long.  Changing jobs is healthy.

- If you aren’t much of a reader, but rather audio or video, try some of these: https://flowoftesting.wordpress.com/2014/09/02/podcasts-and-videos-for-testers/

- Learn from the history of testing: http://www.testingreferences.com/testinghistory.php

- Find some heroes and some villains and learn from them (villains will help you clarify your own views): http://about98percentdone.blogspot.com/2014/01/heroes-and-villains.html

Have fun.  If this doesn’t sound fun at all, go find something that does, even if it is a different career.

Consultant's War

My co-contributor Isaac has been pondering the fact that we see more consultants blogging regularly, speaking their mind and controlling the things we talk about.  I don't care if it is  Dorathy "Dot" Graham, Markus Gärtner, Matt Heusser,  Michael Bolton, Michael Larsen (Correction: As Matt pointed out, I got my facts mixed up and Micheal is not a consultant), Karen Nicole Johnson,  James Bach or Rex Black.  There are dozens of consultants I could have listed (I mean, how could I have missed Doug Hoffman!), some of whom I have personally worked with.  So this is not meant to be a personal attack on any of them.  I realize I list more CDT folk than others, and while that is not intentional, I read less of the thought leaders from the other schools.  If you look, I bet most of the people you read either are or were consultants [In interest of full disclosure, I have written a hand full of articles and have been paid, but I am not nor have I ever been a consultant, and with you reading this, it is at least one counter-argument, but we'll get to that in a bit.].  If you made it here, there is little doubt you've read at least of those author's works at least once.  They are stars!  I mean that literally.  I found this image of James Bach just by searching his name:

From: http://qainsight.net/2008/04/08/james-bach-the-qa-hero/

I didn't even use image search to find that, it came up on the first main page of my Google search results!  I respect James, he writes well and has interesting ideas, but I have no opinion on his actual testing skills, as using his own measure, I have not seen him perform.  Keeping that in mind, I have not seen any of the above consultants do any extended performances in testing, even if I have listened to many of them talk and gained insights from them.  They know how to communicate and they are often awesome writers.  Much better than I am.  They all are senior in the sense that they have done testing for years.  Some of them disagree with others, making it often a judgment call of their written works on who to listen to.  Some of it is the author's voice.  In my opinion, Karen Johnson is a much softer and gentler voice than James Bach.  But some of it is factual.  I have documented several debates between Rex Black and various other people I have listed.  Rex has substantially different ideas on how the world should work regarding testing.

 ISO 29119

James Christie, another consultant brought up a standards in CAST 2014.  It was a good talk and clearly he had some valid concerns about the standards that have been created.  I have talked about that earlier (I am not going to include as many 'justifying' links as all my points and links are in the earlier article).  I am actually not terribly interested in the arguments and in reality, ISO 29119 is more of a placeholder than the actual point of interest.  I am more interested in discussing who is creating these debates.  In part we are in a war between consultants and in part we are in a war of 'bed making'. Clearly the loudest voices are from those who have free time and have money at stake. The pro-ISO side has take the stance of doing as little communication as possible, because it is hard to have a debate when one side won't talk. It is a tactical choice, and it is very difficult to compromise when one side doesn't speak. Particularly when the standard is already in place, the pro-standards side doesn't have much to gain from the messy-bed side who wants the standard withdrawn.

Both groups of consultants have money to gain from it.  A defacto-RST-standard would make Bach a richer man that he already is.  Matt Heusser's LST does give at least some competition of ideas, but still that isn't much more diversity. The standards would probably make the pro-standards consultants more money as they can say, "as the creator of the standard I know how to implement it."  Even if they made no more money in so far as making the standard, they will be better known for having made the standard. Even if I assume that no one was doing this out of self-interest, the people best represented are the consultants and to a lesser degree, the academics, not the practitioners who may feel the most impact.  Those leading the charge both for and against the standard are primarily consultants or recent ex-consultants. Clearly this is a war for which the people with the most time are waging, and those are the consultants. Ask yourself, of those you have heard debate the issue, what percentage are just working for a company?

Granted, I am not a consultant, but it takes a LOT of effort to write these posts.  It isn't marketing for me, except possibly for future job hunting, and the hope that I will help other people in the profession.  I know non-consultants are talking about it, but we don't have a lot of champions who aren't consultants.  Maybe most senior level testers become consultants, perhaps due to disillusionment of testing at their companies.  Maybe that is why consultants fight so bitterly hard for and against things like standards.  Perhaps my assertion that money at the table is a part of it is just idle speculation not really fit for print.  I can honestly believe it to be that is possibly the large majority of the consultants involved.

 Bed: Do you make yours?

Then what is it that causes these differences of view?  Well let me go back to the bed making.  To quote Steve Yegge:

I had a high school English teacher who, on the last day of the semester, much to everyone's surprise, claimed she should tell just by looking at each of us whether we're a slob or a neat-freak. She also claimed (and I think I agree with this) that the dividing line between slob and neat-freak, the sure-fire indicator, is whether you make your bed each morning. Then she pointed at each each of us in turn and announced "slob" or "neat-freak". Everyone agreed she had us pegged. (And most of us were slobs. Making beds is for chumps.)

That seems like a pretty good heuristic and I think it is also a good analogy.  Often those who want more documentation, with things well understood before starting the work and more feeling of being in control via documentation are those who feel standards are a good idea.  They want their metaphorical beds made.  They like having lots of details written down, they like check lists and would rather make the bed than have a ‘mess’ left all day. A nice neat made bed feels good to them. Then there are the people who see that neat bed and think it is a waste of time at best. At worst, they think that someone will start making them make their bed too. Personally, I think "Making beds is for chumps", just like Steve Yegge. Jeff Atwood would go even further and call it a religious viewpoint:

But software development is, and has always been, a religion. We band together into groups of people who believe the same things, with very little basis for proving any of those beliefs. Java versus .NET. Microsoft versus Google. Static languages versus Dynamic languages. We may kid ourselves into believing we're "computer scientists", but when was the last time you used a hypothesis and a control to prove anything? We're too busy solving customer problems in the chosen tool, unbeliever!

Clearly the consultants, many of whom I take valuable bits of data from care about what they do and they tried to lock down their empirical knowledge into demonstrable truths.   But that doesn't mean they have 'the truth'. I know as testers we try to have controls, but I am not so convinced we have testing down to a science.  It is why I feel we aren't ready for standards, but I also recognize the limits of my own knowledge.

For what it is worth, I tend to be against bed making, I think having all this formal work and making checklists is rather pointless unless they fit what I am doing. Having one checklist to rule them all with a disclaimer that YMMV and do the bits you want doesn’t sound much like a standard, but those whom like their bed nice and neat probably feel very happy when they come home at night. The ‘truth’ about the value of making your bed, the evidence that it is better is less than clear. Maybe one day we will have solid evidence in a particular area that a particular method is better than another, but we aren’t there yet in my opinion. But still I don’t make my bed.  In case you are wondering, I think this is probably one of the hardest questions we have in the industry:  What methods work best given a particular context and what parts of the context matter most?

In consideration of ISO 29119

I have been aware of this debate ever since attending CAST 2014, but I've not been quick to sign.  I wanted to investigate and see what other viewpoints there might be.  To quote Everett Hughes, a sociologist:

“In return for access to their extraordinary knowledge in matters of great human importance, society has granted them a mandate for social control in their fields of specialization, a high degree of autonomy in their practice, and a license to determine who shall assume the mantle of professional authority. But in the current climate of criticism, controversy, and dissatisfaction, the bargain is coming unstuck. When the professions’ claim to extraordinary knowledge is so much in question, why should we continue to grant them extraordinary rights and privileges?”

This is a serious question, one for which I appreciate that a standard might seem like a correct method for 'proof' of our professional authority.  However, even if the standard is in fact a professional guide, I don't think anyone but practitioners can judge its value.  As someone who has had a keen interest in trying to understand this standard, and not wanting to judge too quickly, I have tried to get a hold of a great deal of material about it.  I have engaged with some of those whom disagree with my point of view, such as professional tester.  In fact, they have published my response in there Oct. 2014 issue (albeit with some minor mistakes from my draft).  I looked into purchasing the standard.  I have looked for those who are pro-standard and what they've had to say about this movement as well as what they have said in the past.

That being said, I'm not convinced we actually do know what a standard should look like, much less if this is the standard we need.  Maybe it is what we need, but I strongly doubt it.  I think we are likely many years or decades away before we will even be able to claim we have true repeatable practices oriented towards different contexts, assuming it is possible.  I am unable to judge at present what the standard does say, as the standard and the standard body's work is less than transparent.  I've been forced to sign up with personal details in order to read documentation about the standard's creation, although that has been changed since (NOTE: The file name also changed, including a date, which makes it hard to know if anything has changed, as there was no change log as of 10/26/2014).  The standard requests I pay in a currency that I don't use.  I've signing the petition to withdraw the standard not because I know it is wrong, but because I can't tell, which makes it useless at best and dangerously wrong at worst.  What I can tell is that the standard's various author's other documents around the standard demonstrate what I consider to be confusing if not out right contradictory statements, making me doubt the actual standard.  That breaks the social bargain us professionals have that Everett Hughes so eloquently described.

Perhaps you could say that I should have been personally involved in the standard, and that is a valid complaint.  However, I'm not aware of anyone particularly reaching out to the AST, nor have I heard of it from any other group except for James Christie's talk in 2014.  I have attended both AST sponsored and non-AST sponsored conferences for years, so this isn't a case of willful ignorance.  This is my first chance to review the material and process, yet I have not found the process particularly open or transparent.  I see claims of no best practices and claims that the standard will create best practices.  I have found so many confusing statements by the standard's body that I must conclude that the standard should be withdrawn until it can be thoroughly reviewed and modified, if it can even be modified into something useful.

Even ignoring past statements, the recent defense of the standard creates questions.  One of the easiest and most obvious to consider is who wrote the standard.  Then there is the question of who pays for the creation of the standard?  Well clearly this was not just a labor of love, as Dr. Reid says the costs of development have to be passed on to the customer's of the standard.  I should note, my blog makes me no money and I am not a consultant so I have little incentive to make money speaking about this.  I made no money in writing my letter to the editor, and I certainly don't demand you pay to read my work.  I am not discounting the cost of writing the standard, just simply saying that if you plan on having expenses paid by publishing your work, you are not simply doing this out of the kindness of your heart.  There is lots of analysis that could be done just on the defense of the standards alone, but is outside the scope of this particular post.

One of the oddest and most compelling arguments for both sides is from Rex Black, in which he notes that about 98% of all testers won't care one way or another.  I think this is true, which makes the standards mostly not matter, but it also means that those 98% who are silent count on us to ensure these are the right standards, lest they become popular and that silent 98% ends up forced into using them.  I fear that this non-involvement is also further evidence that we testers as a group are not acting like a profession.  It isn't that we don't claim to have "extraordinary knowledge" and Dr. Reid at least seems to argue we mostly agree on this knowledge, but rather the majority of people don't feel any need to actively participate.  I realize this might be an argument for why we need a standard -- to show the disinterested the 'right' way to test, but to me it seems to indicate just how young our industry is and seems to me that shows why we aren't ready for a standard.

Even if the ISO body decided these documents ultimately should stand, the objections of the AST/Context Driven community need to be noted in such a standard.  Furthermore, making the document open will go a long way in allowing the community to discuss this document beyond the smaller standard's committee.  I recognize that ISO needs income to maintain itself and won't publish them for free for everyone, but certainly some sort of 'for individuals, not corporations' license could be used (and I don't mean the sort of non-sense Matt Heusser describes).  Finally, if this is an attempt to demonstrate our commitment to professional testing, then it needs to be accessible to our community.  The work needs to demonstrate it's value rather than being buried away inaccessible to those who would use it.

WHOSE - Part 1

I'm keeping this brief and unedited, as I don't have much time.  As many people, I have been getting busier for the holidays, but for me part of it is going to WHOSE.  I will perhaps be delayed by a few weeks as I gather together material and prepare for some presentations for next year.  That being said, I hope to have some good material from WHOSE and I will make sure to publish it as quickly as I can.  I do have a few older review I wrote for some classic books.  I may post them to keep you all entertained.  Be back soon.