Leveling Up Your Testing Skills

Recently I gave a talk intended to be a quick survey of some of the core areas of testing.  I certainly didn't hit everything, but meant to give people tools to go exploring on their own.  I used a text editor for my presentation, because not everything needs power point.  I thought it might be useful to others so I am recreating it here.  I have done some minor modifications, removing things that only made sense in context of a performance.  I also added a few things based upon the discussion during the presentation.

 ----- Page 1

Survey:

Who has heard about AST?

How many of you have read or even skimmed a testing book in the last year?

How many of you have read a white paper?

How many of you have written ANY bugs outside of professional obligations (your job, school, etc.)?

Has anyone tried to formally QA an article?

How many of you have heard about the “No Tester” movement?

How many of you have heard about ISO 29119?

 ----- Page 2

Purpose:

My purpose is not to teach you facts about testing, even if some of that happens, but rather to teach you what questions you didn’t know that you should even ask and show you places you didn’t know to look.

NOTE: I know I cite myself several times, and while there are other sources, few are as narrowly focused to the strategies I am going to talk about..

 ----- Page 3

Do you know the different ‘schools’ of testing?

Four Schools of Testing*: www.testingeducation.org/conference/wtst_pettichord_FSofST2.pdf

Analytical School - Code Coverage, Unit Testing

Factory School - Metrics, Traceability

Context-Driven School: Exploratory, Multidisciplinary based upon context

Quality Assurance School: Protect user, Gatekeeper, Process-Oriented

* [Edit: Since this presentation, I have discovered an updated model including a 5th school, Agile.  See https://www.prismnet.com/~wazmo/papers/four_schools.pdf)

 ----- Page 4

I’m mostly going to speak from the CDT point of view…

(Aside: While I appreciate CDT’s philosophy, be aware that if you approach the community online, some people participating in CDT’s development tend to be passionate/challenging/aggressive and some are very interested in defining words.  Even if you are not interested in that, CDT is still worth looking into, just figure out who you find value from and whom you don't get value from.)

  

 ----- Page 5

Two of the largest “CDT“ classes:

BBST: http://bbst.info/ ; http://www.testingeducation.org/BBST/ ; http://www.associationforsoftwaretesting.org/training/courses/ ; http://altom.training/bbst-foundations/

RST: http://www.satisfice.com/rst.pdf

(Both have free portions, BBST includes all the lectures for free)

  

 ----- Page 6-8

Models and Heuristics:

What are they? Read my work: Words of the Week: Heuristic [& Algorithm]: http://about98percentdone.blogspot.com/2013/10/words-of-week-heuristic-algorithm.html

“A Heuristic is an attempt to create a reasonable solution in a reasonable amount of time.  Heuristics are always Algorithmic in that they have a set of steps, even if those steps are not formal.” 

List of Heuristic Mnemonics: http://www.qualityperspectives.ca/resources_mnemonics.html

SFDIPOT - Structure, Function, Data, Integrations, Platform, Operations, Time

HICCUPPSF - History, Image, Comparable Product, Claims, User Expectations, Product, Purpose, Standards and Statutes, Familiar Problems

Heuristic Test Strategy Model: http://www.satisfice.com/tools/htsm.pdf

How do you know when you are right? http://about98percentdone.blogspot.com/2013/11/how-do-you-know-when-you-are-right.html

Models:

Session-Based Test Management: http://www.satisfice.com/sbtm/index.shtml

Thread-Based Test Management: http://www.satisfice.com/blog/archives/503

Tours: http://www.developsense.com/blog/2009/04/of-testing-tours-and-dashboards

   

 ----- Page 9

Books (just a few):

An Introduction to General Systems Thinking: http://www.amazon.com/exec/obidos/ASIN/0932633498

Exploratory Software Testing: Tips, Tricks, Tours, and Techniques to Guide Test Design: http://www.amazon.com/Exploratory-Software-Testing-Tricks-Techniques/dp/0321636414

Lessons Learned in Software Testing: A Context-Driven Approach: http://www.amazon.com/Lessons-Learned-Software-Testing-Context-Driven/dp/0471081124/

Agile Testing: A Practical Guide for Testers and Agile Teams: http://www.amazon.com/Agile-Testing-Practical-Guide-Testers/dp/0321534468/

 

 ----- Page 10

Self-based learning

Go find a testing job for the weekends like:

http://www.utest.com/

http://sourceforge.net/p/forge/helpwanted/testers/

http://www.guru.com/d/jobs/

http://weekendtesting.com/america

   

 ----- Page 11

Where are you going?

http://blog.red-gate.com/wp-content/uploads/2014/02/Test-Engineering-Skills-v3.pdf

(Go look at this guy, it is an interesting list of skills)

   

 ----- Page 12 & 13 

Experts and active members of the community:

There are different people who have different expertise’s that are worth reading.  Not everyone does much automated testing, while other people are more interested in coaching or test design.  Look at where your strengths are and find people who will be able to supplement them.  Some of these people may in fact oppose my own views, but that too can be valuable.  Debate helps you understand your own point of view.  Here is an example or two….

James Bach* – Testing is an Intellectual Inquiry

James Christie* — Consultant, Professional Auditor and Tester, Started entire ISO 29119 discussion

Jon Bach* — Session-Based Test Management (with his brother)

Michael Bolton — Testing vs Checking

Huib Schoots — Dutch Testing Coach

Scott Barber — Load Test Guru

Rex Black — Proponent of Factory School of Testing***

Harry Robinson — Model Based Testing

Justin Rohrman* — Active in community, board member of AST

Robert Sabourin** — Author of I Am Bug (pictures by his 12 year old daughter)

Jerry Weinberg  — Systems thinking, consulting, author

Karen N. Johnson* – Professional Tester’s Manifesto

Fiona Charles* — Author, Speaker

Matt Heusser* — Founder of Miagi-Do

Cem Kaner — Originator of CDT, BBST, Legal concerns around testing

Alan Page — (Former?) Microsoft Test Architect

* I met these folks at CAST, and you too can meet these sorts of folks at CAST.

** I met Rob at WHOSE, I enjoyed his writing at WHOSE but he tends to be more academic in my experience.
*** [edit:  Since publication my understand of Rex Black's ideas has evolved. Rex specifically rejects the concept of Schools and, as best I understand would prefer I say he's a Proponent of the Strategy of Factory Style Testing in some contexts.  I leave the original work as stands for historical reasons.]

(Some of these people are accessible to talk with, some only speak through their work, some through conferences…)

    

 ----- Page 14

An Exemplar (with more details): 

Doug Hoffman

http://www.softwarequalitymethods.com/h-papers.html

(pay attention to the dates of the papers, some have updated versions)

He has written exhaustively around automation and testing.  While that isn’t the only work he’s done, you will get a good idea of the depths you can go in with automation.  On the other hand, if you want to learn to code, Doug is probably not the man to visit.  He’s interested in theory behind automation.  He was one of the minds behind the ideas High Volume Test Automation and Exploratory Automation.

I wouldn’t go to Doug to find out the latest idea behind Selenium.  I would go look at Doug’s work to understand what sort of automation techniques might make sense to apply to a problem.

    

 ----- Page 15

Questions you need to ask (that I can’t answer):

How much time am I willing to dedicate to learning?

What sort of things do I want to learn?

How varied do I like my studying?

Is this a job or a profession?

Do you want to do something that matters?

Are you okay with having 10 years of experience that really only add up to 1 year of valuable experience?

 ----- Page 15-17

Questions you need to ask (that I can give hints on):

Where can I learn?

 - Blogs

 - BBST/RST and other classes (LST is another CDT class, RBCS is from the factory school)

 - Books

 - Co-workers (particularly challenges)

 - Conferences (CAST - https://www.youtube.com/playlist?list=PLQB4l9iafcelXpJnK6IyDsoFeEb1icqrl , Code Camp, Star West, Star East, Agile testing days, GTAC - https://developers.google.com/google-test-automation-conference/2014/stream , etc. ; Often recorded on Youtube)

 - White papers: http://about98percentdone.blogspot.com/2015/01/white-papers-and-who-owns-your-education.html

What should I learn?

 - Start by glancing at the foundations class.  See how much of it you know already.

 - Read blogs about what interests you.

        - Look up the people I talked about (I suggest Googling "<Name> Testing”.  Almost all of them have blogs).

        - Look at the top blogs and see what they have to say: http://www.testbuffet.com/lists/top_114_software_testing_blogs_2014/page/1/

        - If those methods don’t work, try looking at the fire hose and see if there is an interesting topic: http://www.testingreferences.com/testingnews.php

 - Start your own blog and learn to write; here is mine: http://about98percentdone.blogspot.com

       - Feeling shy?  Why not start by writing comments.

 - Pick up one book.  Read it.  Don’t like the ones I listed?  Check out this extensive list: http://www.testingreferences.com/software_testing_bookstore.php

   - Write a review on your blog

 - QA a news article.  Write up bugs on failed assumptions.
      - http://about98percentdone.blogspot.com/2013/09/testing-babies-for-learning.html

Learn about your interests AND figure out how much you are gaining.  If your energy wanes, move to a different topic or take a break.

What else can I do?

- Figure out what matters and work on that: http://about98percentdone.blogspot.com/2014/10/noise-to-signal-attempt-to.html

- Don’t keep working in the same position for too long.  Changing jobs is healthy.

- If you aren’t much of a reader, but rather audio or video, try some of these: https://flowoftesting.wordpress.com/2014/09/02/podcasts-and-videos-for-testers/

- Learn from the history of testing: http://www.testingreferences.com/testinghistory.php

- Find some heroes and some villains and learn from them (villains will help you clarify your own views): http://about98percentdone.blogspot.com/2014/01/heroes-and-villains.html

Have fun.  If this doesn’t sound fun at all, go find something that does, even if it is a different career.

White Papers and Who Owns Your Education?

Let me start out by point you to this rather complex white paper, A Large Scale Study of Programming Languages and Code Quality in Github.  It inspired me to write a little about reading white papers and why they matter.  In trying to read or perhaps decipher, it made me feel a little stupid.  What, never heard of the Chi-Square test?  How about Cramér's-V?  Does that make you feel like, "Then why are you reading this?  Don't you feel dumb?  You should go back to yahoo news or wherever you came from."  That is the voice in the back of my head, and yes, reading these sorts of papers certainly can make me feel dumb.  Fortunately, I know better.  Even if I am not a accomplished statistician, I'm sure there could be bits of data that are worth gathering.  I personally believe we should try to improve our development* in whatever ways possible, and this study is trying to do so by providing a very high level view of computer languages.  Even though it is not particularly about finding bugs or testing, it does give some insights into bugs and their relationship to languages.
* Pun intended

Let me start with a word of warning before we deep dive into the analysis of this whitepaper.

"One should not overestimate the impact of language on defects.  While these relationships [in the chart in the study] are statistically significant, the effects are quite small.... activity [commits] in a project accounts for the majority of explained deviance [difference in bug rates]... the next closest predictor which accounts for less than one percent of the total deviance, is language."

So while language is important, code churn is more important.  That however does not end the story.  There are still important pieces to pickup.  I am going to leave the definitions of what constitutes a particular category, such as a 'functional language' to the actual paper, which you can look up if it is of interest.  Perhaps you know the language you are working with, so considering a list of languages might be of interest.

"...a greater number of defect fixes... include C, C++, JavaScript, Objective-C, Php, Python. ... Closure, Haskell, Ruby, Scala, and TypeScript... are less likely than average to result in defect fixing commits."

For most companies this is bad news.  Few people write Scala or Haskell on a day to day basis.  Worse yet, I wonder if the experience level of those who write in such languages is greater than average, something the study could not control for.  From what I could tell, they didn't even consider looking at the length of time a user's name was associated to a repository as a proxy for experience.  Since experience was not considered, perhaps the broader categories such as 'functional languages' is more helpful.

"...strongly typed languages are less error prone than average while weakly typed languages are more error prone than average."

"This is strong evidence that functional static languages are less error prone than functional dynamic languages..."

"Functional languages have a smaller relationship to defects than either procedural or scripting languages."

So if you are using a strongly typed language you are better off and if you are using a functional language you are better off.  Does it matter what sort of application you are working on?

"...there is only a weak relationship between domain and language class."

Oh, well I guess then it doesn't really matter what you do, it is how you use the language.  Are there classes of defects that matter?

"Languages matters more for specific categories than it does for defects overall."

"In fact, we noticed 28.89% of all the memory errors in Java come from a memory leak.  In terms of effect size, language has a larger impact on memory defects than all other cause categories."

Interestingly, even though Java has managed memory, it seemed to suffer more from memory problems than most other memory managed languages.  This could be because of the applications considered, such as ellasticsearch, a Java based search engine compared to SparkleShare, a C# file sync tool.  I couldn't find anything language related that seemed obvious for Java's problems with memory compared to C#.  On the other hand Java is way better than any unmanaged memory languages.

"Go has a lot more concurrency bugs related to race conditions due to its race condition detect tool..."

"TypeScript is intended to be used as a static, strongly typed language... However in practice, we noticed developers often [50%]... use any type, a catch-all union type, thus makes TypeScript dynamic and weak."

In looking at what may be wrong with this study, I found it interesting that tool support maybe a reason for differences between languages.  Basically, because the language is only as good as the people who use it and the tools around it, it could be the study gives a false sense of knowledge.  It could be that the use cases are so complex, that this one study and method will not get us much in regards to languages and when to use them.  To be fair, the researchers noted other studies, with their own limitations and similarities.  Confirmatory studies are useful, but humanity is really complex and their study wasn't completely confirmatory.  This is a sort of social-science meets well recorded data.  Git has perfectly recorded data, but even with it, it is hard to know because there is so much data.  There was a 'study' done a few years ago regarding swearing and checkins in github.  While amusing, it also is one more piece of data.  Keeping in mind the warning I started with at the beginning of the article, this was the authors' conclusions

"The data indicates functional languages are better than procedural languages; it suggests that strong typing is better than weak typing; that static typing is better than dynamic; and that managed memory usage is better than unmanaged. Further, that the defect proneness of languages in general is not associated with software domains. Also, languages are more related to individual bug categories than bugs overall."

For the average tester, is there something to learn here?  I think the primary thing is that code churn has more to do with defects than anything else.  Fighting over the tool to use after a project is started is probably not worth it, unless you are rebuilding from scratch.  Keep in mind what else comes along with the language and what value that gives you.  If race conditions matter, then Go is likely a winner.  Just because you have managed memory does not mean you can't have memory issues.  Functional languages like SQL tend to have less defects, possibly because state is less of an issue.  Finally, no matter what language you are using and no matter what you are doing, you will have bugs.  What else did you pick up from the white paper?

Wait, wait... don't drop the curtains yet.

In violation of the end an article with a question to cause action, I want to give pause in considering that perhaps this subject is not to your liking.  Well why not read up on all the data browsers leak out.  It will give you a good feel for just how complicated the simple underlying HTTP connections are.  Maybe you want to learn how to model learning, in which case the Dreyfus Model of Skill Acquisition might be up your alley.  Or perhaps you want something more on testing, such as the pair wise testing white paper.  Or perhaps the famous session based test management paper.  Perhaps none of the ideas I've suggested have struck your fancy.  Cem Kaner developed hundreds of links in his citations for one of his classes, many of which are white papers.  And that is to say nothing of rfcs, a Request for Comments, which helps define standards we all use.  Even books like The Reflective Practitioner, which I am slowly reading through, but has been well analyzed in Wikipedia.

Which brings me to the problem with Wikipedia and blogs like mine that analyze content.  People like myself pull out the conclusions for you and so you skip reading the white paper.  Do you really want to leave your entire education up to those who form conclusions for you?  Let's me be honest, most of you won't click half the links I provided and just as many won't seriously read the cited white paper.  It is not unusual for a large percentage of people not even to read this far, complaining "TL/DR".  "So what?", you, the wise reader who has made it this far might ask.  In my book review of An Introduction to General System’s Thinking I tried to pull out some of my favorite little pieces, some flavour, which was not directly related to the review.  Some of Gerald Weinberg's wisdom will be lost if all you do is read a short summary of his ideas.  Now you say you don't have time, wiki is better than nothing.  Maybe that is true.  I won't argue that you shouldn't use wiki and speed through some of one's education.  Just make sure to occasionally spend time looking at a subject deeply to help deepen your thinking and understanding.  I am sure you might come up with more reasons why you shouldn't, and I won't try to out reason you in your quest to not read deeply.  It is your choice.  I will say it might be hard to do that sort of deep reading, but it will be worth it.

So once again, we are back to the cheesy, end with a question.  Well I won't burden you with that sort of guilt-driven work harder question at the end of this post.  Instead I will give you an offer and a statement.  If anyone leaves a comment with a white paper or rfc for me to read, I will read it and I might even report back on it.  Book suggestions are a dime a dozen and sadly even I have finite amounts of patience for books and a stack of books I want to read, but if you give a recommendation I will at least seriously consider it.  And now for the statement:  I want to gain a deeper understanding of the world, and I will share some of what I learn, but you must do your own investigating as well and I want to say...  Good luck with your own education!

More on No Testers

I sometimes look to see who is reading my articles, and if the incoming link looks interesting, I try to see if I can find the blog it came from, maybe even comment on it.  I particularly wanted to see if anyone had comments on my latest blog about No Testers.  Well I did that today.  I went to look at a Russian article (in Russian), and found a comment by Maxim Shulga saying (translated by Google):

Detailed and very adequate answer to the same question to my http: //about98percentdone.blog ... The only pity is that uncomfortable read: white on black. Burns my eyes :)

Now normally I would just reply to such a comment by filling out the form or using my gmail credentials and writing back.  I might say something silly about how I know not everyone likes the black on white style, as Jeff Atwood pointed out years ago.

While Google translate did translate the article, it didn't translate the comments nor the method to reply to a comment... using some developer tools I was able to establish this was from Disqus.  I am now going to chronicle my efforts to get an account and demonstrate how usability might have benefited from a tester, not to mention a bug tracking system.  Before I go on, I want to say I have no business relation with Disqus or any other discussion or forum/blog related software outside of this blog.  I am not trying to pick on them, it just happens I was trying to get something done and they are blocking me.  They also happen to have no testers, or at least no one with a label of "QA" or "Tester" or anything outside of 'engineering'.  This just happens to be an interesting example which relates to my previous post. I did not intentionally go looking for a company that has no testers and only discovered after I started writing this blog post that they have no testers.  I have contacted them about the issues I have noted here and none of the issues relate directly to security or should go unpublished for ethics reasons.

What I Found

Item 1:

Disqus did not give me English versions of their UI for replying even though my browser should be requesting English.  I tried several browsers, including IE which I checked to see it was set to en-US.  I am not sure if this could be detected via any sort of metrics to figure out they should be looking at the browser language rather than some user/blog setting.  To me this is the sort of choice that happens when you don't think long and hard about localization.  But perhaps they know and plan on changing it or maybe they think this is the right choice.

Item 2 & 3:

So I go to https://disqus.com/ and click sign up.  I am asked for a email AND a username.  I don't know what the username is used for in this context and there is no useful description or even an icon to click.  So I enter an email, username I enter JCD and a password and get told "Username already exists."  So I try JC-D.  Nope, "Letters and numbers only please."  Okay, what about JÇD.  It is all ASCII characters but I just get "Letters and numbers only please."  In fact, it thinks all sorts of things are not letters or numbers in their view but which I would consider letters.  I didn't bother with much Unicode but I imagine it would have the same sorts of issues.  Maybe this is intentional, but their error is not meaningful to me.  Worse yet, it excludes many people with names that are not just alpha characters.  My name has a hyphen in it, thus the JC-D.  The hyphen was excluded, excluding me using my real name.  Granted the math models would say I don't count as few people have hyphenated names.  Maybe I shouldn't care, maybe the name doesn't matter, but the username's usage is not clearly explained.  However, not including Markus Gärtner because his name has a non-English character seems really wrong.

Item 4:

I thought for a moment to use their gmail integration.  I have used it for Khan's Academy to log in and it worked fine.  So I tried but it looked like they wanted me to have a Google+ account, which I intentionally don't have.  I am a bit odd, wanting both privacy and a voice.  I don't like giving away personal details and being tracked, even if I have professional opinions that I wish to voice.  My professional and personal lives mix, but only a little.  So that was a no-go for me.  Worse yet, I get an warning saying that OpenID2, the method Disqus uses to sign up, will being no longer supported early next year (April I believe).  Clearly they have some updating to do.

Item 5:

I considered using a fake email address, but their terms of service were not on their sign up page.  In fact, most of the links on their page went away on the sign up page.  Maybe that is intentional.  Maybe it was A/B tested.  If so, awesome, but for me it was less than optimal.  I admit to not being the main use case, and perhaps that is one problem with testers.  We are not equal to users.

Item 6:

I go to report these issues and the best I can find is a contact support.  Not saying support is a bad place to start, but the form of input gives me about 3 sentences and a scroll bar.  If I wanted to tweet the error, that might work, but I had detailed points to give.  Only someone who is concerned with the customer would notice this, but I, a potential customer did.

Analysis of Why the Issues Exist

I suppose the question is, how do we capture this sort of data or if we care.  Maybe annoying your users is okay when you are a free product.  Maybe alienating users is fine when your metrics show few users try to use Unicode.  Perhaps that is what data scientists are useful for, deciding which problems matter?  Maybe having a functional tester would notice these issues and bring them up?  Having the customer deal with the problems until you figure out if it is a good idea is not an uncommon model, particularly if you own the market.  But keep in mind I never did get to make that comment.  Speaking in psychological terms, even with professional distancing, I will have a more negative view of their product and it will take effort on their part to turn me around.  Even if they magically changed it all tomorrow a potential customer like me might be long gone.  Perhaps with a billion users, it doesn't matter.

While I personally don't feel this way, but maybe the Disqus team is not the right team, which is the argument made for hiring 'the right team' that I have heard from the no tester camp.  If they were, I'd not be writing this post, with so many issues.  I think the language used by the no-tester side is unclear what the 'right team' is, and perhaps what they think testers do.  I am not sure that a mythical right team, with or without testers will ever produce bug-free code, but there are certainly good and bad team mixtures.  I feel that it is rather more difficult to evaluate their team dynamics having never met any of them.

Perhaps they are 'the right team', and I am just the wrong customer?  That is the other half of this particular no tester argument.  That testers are not like customers, so use customers.  As a customer who thinks like a tester, maybe I'm not representative?  Then again, if I pull out my heuristics, I can compare this to other products that don't require you to sign up at all to write a comment.  That may not be a complete defense in comparing myself to a real customer, but it certainly gives credence to these issues.

Finally, it could be they do have QA/Testers but they were renamed to some other title.  That just made it harder for me to figure out if they have testing and if what was built was what was intended.  These are design choices, but it is unclear if there was anyone questioning these design choices. Without someone in that role, the 'get it done' mentality comes into play, at least in some organizations.  Perhaps that happened here.

I am sure the reality of Disqus is way more complex than I have presented it, but I am an outsider.  I welcome any feedback from the company and will update this accordingly.  I also was not looking for this example case.  I was not planning on posting any more this year.  It just showed up and I thought it was interesting.  I'd love to hear from those who feel no testers is an appropriate choice and how they would interpret this.

To Maxim Shulga, I am sorry you don't like my black background with white text.  I will take your view under advisement if I ever try to re-theme this blog.  I hope at least the content is useful.  And next time, just leave a comment on my blog... this reply to your comment took way too long to write. :)

Thinking about No Testers

In attempting to hear out the various viewpoints of those whom subscribe to the idea that Agile should move towards removing the role of tester.  I have yet to see anyone who actually suggested eliminating testing completely, if that is even possible.  So let us unbox this idea and examine it, with as little bias towards keeping my own job as possible.  Here are the rough set of arguments I have seen/heard in TL/DR (Too long, didn't read) form:

• Limited deploys (to say .01% of the users and deploy to more over time) with data scientists backing it will allow us to get things done faster with more quality.
• Monitoring makes up for not having testers.
• Hiring the right team and best developers will make up for the lack of testers.
• Writing code that "just works" is possible without testing, given loose enough constraints (which leads to the next argument).
• Since complete testing is impossible, bench testing + unit testing + ... is enough.
• Quality cannot be tested in.  That is the design/code.
• Testers make developers lazy.
• It is just a rename to change the culture so that testers are treated as equals.
• Testers should be under a dev/team manager, not in a silo.
• It is a minor role change where testers now work in a more embedded fashion.
• We hire too many testers who add inefficiencies as we have to wait <N time> before deploying.
• We only hire testers to do a limit set of activities like security testing.
• Testers do so many things the name/profession does not mean much of anything.
• With the web we can deploy every <N time unit>, so if there are bugs we can patch them quickly.
• As a customer I see those testers as an expensive waste.

That is sure a lot of different reasons, and I'm sure it is not an exhaustive list.  Now I shall enumerate the rough responses I have seen:

• A little genuine anger and hurt that 'we' are unwanted.
• A little genuine disgust that 'we' are going through the 80's again.
• Denial ('this will never work')
• It can work for some, but probably not all.  (This follows a very CDT 'it depends' point of view)
• Not everything can easily be instrumented/limited deployed.  For example, I have heard Xbox 360 games cost ~10+k to do an update.
• In some software, having less bugs is more important than the cost of testing.
• This is why we need a standard (I never heard this one, but it is an obvious response)
• Concern about the question of testing as a role vs task.
• Different customers have different needs they want to fulfill.
• Good teams create defects too.
• We need science to back up these claims.
• Testing should move towards automation to help eliminate the bottlenecks that started this concern.
• It is very difficult to catch your own mistaken assumptions.
• If you need someone else to help catch mistakes, why not a specialist?

While I don't have a full set of links I have heard this back and forth viewpoints, but I think it is an interesting subject to consider.  I am going to go through my bulleted list top down, starting with the "no testing" side of the world.  I believe I have seriously considered the limited deployment and letting the tester be the customer in a previous post.  In that same post I noted that at least not all teams can make up for a lack of testers, so again, I think it has been addressed.

Complete Testing is Impossible & Quality Cannot be Tested in

The premise is true, complete testing is in fact impossible, I do agree with that.  The conclusion that hiring testers is not needed is the part I find interesting.  Quality of code and design cannot be tested in is also in interesting statement.  The idea that testers make developers lazy feels very unsubstantiated, with possible correlation-causation issues (and I have seen no studies on the subject), so I am going to leave that part alone for now.

I write software outside of work.  Mostly open source projects, of which I have 5 projects to my name.  Most of them were projects in which I was looking to solve a problem I had and wanted to give to the world.  In more than half of them I wrote no formal tests, no unit tests and to be honest the code quality in some of them is questionable.  There are bugs I know how to work around in some cases, or bugs that never matter to me and thus the quality for me is good enough.  I have had ~10k downloads from my various projects and very few bugs have ever been filed.  Either most people used it and then deleted it or the quality was good enough for them from a free app.  I hired no testers and as the sole dev I was the only one testing the app.  I think this is a +1 for the no tester group, if you work on an open source app for yourself, no tester (maybe) required.  CDT even would agree with this as it does depend.  Context matters.  I am sure other contexts exist, even if I have not seen them.  What I have not seen is specifics around contexts that might require no testers, with the exception of the Microsoft/Google sized companies.

Now can you test code quality in?  If you are about to design the wrong thing and a tester brings that up, did the tester 'test' code quality in?  Well he didn't test any software, the no tester side would say.  Sure, but he did test assumptions, which is often part of testing.  What about the feature the tester notes the competition has that you don't, which was discovered when comparing competitor protect quality?  What if I suppose the dev knows exact what to develop, can the tester add value?  Well I know I have made code suggestions that have gotten a developer thinking.  However, I could have just as easily been an XP developer sitting beside him as he coded rather than a tester.  Well what about TDD?  Is that testing quality in?  If I write tests first, I have to think about my design.  That is testing quality in, but again that is just the developer.  What if the tester and the developer co-design the TDD or the tester recommends additional tests added during development.  Again, this could be a developer, but a tester is often thinking about the problem differently.  So while I think the statement is false, if I look at it from the spirit, I think it might be true in some cases.  There are some really great developers who do think like testers and they could do it.  However, I think that is only some of the developers.  There will always be junior developers and there are always other mind set factors that play in.

 What is in a Name & Quick Deploys?

What's Montague? it is nor hand, nor foot,
Nor arm, nor face, nor any other part
Belonging to a man. O, be some other name!
What's in a name? that which we call a rose
By any other name would smell as sweet;
- Juliet, Romeo and Juliet

Does the fact that we are in a in-group called 'test' and some see developers as an outsider matter?  Visa-versa is probably true too.  In fact, the other day Isaac and I had a small debate about correcting a tester in some very minor minutia about deep mathematical and development theory.  To me, perhaps a developer who was neck deep in building new languages would find such a correction helpful, but a tester who was doing some exploration around the subject didn't need such a correction.  In fact such a correction might make things more confusing to anyone who didn't have a deep understanding of the subject.  I myself am guilty of the insider/outsider mentality.  We all categorize things and use those categories to make choices.  Perhaps renaming us all as human beings would make things better.  If an organization is particularly combative, this might be a useful change.  That being said, there is also a group identity for people who see themselves as testers or QA.  There is a specific field of study around it, and that also adds value.  Also, by having a sub-group, the tension, even if minor, might possibly help people grow.

This is a mushy, non-business oriented view of the world, and perhaps those who are just interested in the how fast things go today would prefer avoiding the conflict in order to get things done.  The fear of doing this from a QA point of view comes from the 1980s, where defects were a big deal.  Renaming people feels like going back to that era.  The argument back is that this isn't the 80s, we can ship software almost instantly.  The argument to this is that some software can't just be shipped with broken code.  Work on many different areas must be 'mostly right' the first time.  Also, this grand new world that people want to move into has not been vetted by time.  Wait a while and see if it is the panacea it is believed to be.  I recognize the value in renaming positions in order to improve team cohesion and believe it comes from a genuine desire to improve.  Just keep in mind what is lost and the risks in doing so.

I should note, I have seen some of this implemented before in organizations I have worked in.  Working under a single development manager, the development manager would often ignore the advice given by the "QA" person.  Perhaps a full rename would have fixed things, but often development wants to move forward with the new shiny thing.  Having no testers is perhaps a new shiny, and so development would be on board.  The people who are more aware of risk of change are the QA and operations people.  This is a generalization, unsubstantiated by anything but my observation, but it feels like QA is now calling this a risk and because it threatens our jobs we are being told we are biased, so we can be ignored.  Psychology and politics are complex things and anything I write will not adequately cover this subject.  I don't object to people trying this, maybe it will work.  I still see it as a risk.  I hope some people take it and report back their observations.

Perhaps the final argument about us is that we are many different roles tied with one name.  This is a complexity that came up at WHOSE.  We talked less about testing than you might expect.  Communication and Math and Project Management and ...  We had a list of about 200 skills a tester might need.  We are a swiss-army knife.  We are mortar for the projects developed, and because every project is different, we have to fill in different cracks for each project.  We happened to do that best in some ways because we happen to see many different areas of a system.  We have to worry about security, the customers who use it, the code behind it, how time works, the business, and lots more.  That is ignoring the testing pieces, like what to test, when to test and how to justify the bugs you find.  Maybe we could remove the justification piece, but sometimes the justification matters even to the developer, because you don't know if it is worth fixing or what fix to choose.  I think we can't help being many different things because what we do is complicated.

Customers Don't Want QA?

While that is a lie in regards to me personally (I get excited when I see QA in the credits of video games), perhaps it is different for others.  I have written elsewhere how users of mobile devices are accepting more defects.  Perhaps that is true in general.  We know there is such a thing as too much QA.  To go to the absurd, when QA bankrupts the company, it is clearly too much QA.  We also know there is too little QA, when the company goes bankrupt from quality issues.  Perhaps we could say when there are regulatory constraints that close the company down because of a lack of QA could also be there.  Those are the edge cases, and like a typical tester, I like starting at boundary cases.  Customers clearly want some form of quality, and as long as testers help enhance quality (E.G. Value to someone who matters), testers are part of that equation.  Are they the only way to achieve quality?  Maybe not, but that involves either re-labeling (see above) or giving the task to others, be they the developers or the customers or someone else.  Some customers like the cheaper products with less quality, and that does exist.  Having looked at the games that exist on mobile devices that are free, my view is the quality is not nearly as good.  Then again, I want testers.  Maybe I'm biased.  Even if I was a developer for a company, I would like to have testers.

Should the ideas of no testers be explored?  I have no objection, but I would say that it should be done with caution and be carefully examined.  Which makes the companies who try it...well, testers.

 Brief Comment on the Community

I have heard a lot of good responses, albeit limited.  I have also seen some anger and frustration.  Sometimes silly things are said in the heat of the moment, likely from both sides.  We also have some cultural elements whom are jagged edged.  These can create an atmosphere of conflict, which can be good, but often just causes shouting without much communication.  One of the problems is most people don't have the time or energy to write long winded detailed posts like this.  We instead use twitter and use small slogans like #NoTesting.  I personally think twitter as a method for communicating is bad for the community, but I acknowledge it exists.  How can we improve the venue of these sorts of debates?  How can we improve our communication?  I don't pretend to have an answer, but maybe the community does.

UPDATE: I wrote a second piece around this topic.

Shorts: Ideas described in less words; Code Coverage, Metrics, Reading and More

Most times I write more 'essay' style articles, but Isaac and I have sometimes had small ideas we wanted to discuss but didn't feel like they were big enough to post on their own.  So I'm trying this out, with a series of small short ideas that might be valuable but are not too detailed.  Please feel free to comment on any of these shorts or on the idea of these smaller, less essay-style posts.  If you are really excited about a topic and ask interesting questions, I might try to follow it up with another essay-style post.

Code Coverage

Starting with a quote:

Recently my employer Rapita Systems released a tool demo in the form of a modified game of Tetris. Unlike "normal" Tetris, the goal is not to get a high score by clearing blocks, but rather to get a high code coverage score. To get the perfect score, you have to cause every part of the game's source code to execute. When a statement or a function executes during a test, we say it is "covered" by that test. - http://blog.jwhitham.org/2014/10/its-hard-to-test-software-even-simple.html

The interesting thing here is the idea of having manual testing linked to code coverage.  While there are lots of different forms of coverage, and coverage has limits, I think it is an interesting way of looking at code coverage.  In particular, it seems like it might be interesting if it was ever integrated into a larger exploratory model.  Have I at least touched all the code changes since the last build?  Am I exploring the right areas?  Does my coverage line up with the unit test coverage and between the two what are we missing?  This sort of tool would be interesting for that sort of query.  Granted you wouldn't know if you covered all likely scenarios, much less do complete testing (which is impossible), but more knowledge in this case feels better than not knowing.  At the very least, this knowledge allows action, where as just plain code coverage from unit tests as a metric isn't often used in an actionable way.

I wonder if anyone has done this sort of testing?  Did it work?  If you've tried this, please post a comment!

 Mario’s Minus World

Do you recall playing Super Mario Brothers on the Nintendo Entertainment System ("NES")? Those of you who do will have more appreciation for this short but I will try to make it clear to all.  Super Mario Bros, which made Nintendo's well known series of Mario games famous.  In it you are a character who has to travel through a series of 2D planes with obstacles including bricks and turtles that bite, all to save a princess.  What is interesting is that fans have in fact cataloged a large list of bugs for a game that came out in 1985.  Even more interesting, I recall trying to recreate one of the bugs back in my childhood, which is perhaps the most famous bug in gaming history.  It's known as the minus world bug.  The funny thing is in some of these cases, if a tester had found these bugs and they had been fixed and the tester would have tested out value rather than adding value in, at least for most customers.  I am not saying that we as testers should ignore bugs, but rather one man's bug can in some cases be another man's feature.

How Little We Read

I try not to talk much about the blog in a blog post (as it is rather meta) or post stats, but I do actually find them interesting.  They to some degree give me insight into what other testers care about.  My most read blog post was about the Software Test World Cup, with second place going to my book review of Exploratory Software Testing.  The STWC got roughly 750 hits and EST got roughly 450 hits.  Almost every one has heard of Lessons Learned in Software Testing (sometimes called "the blue test book").  It is a masterpiece made by Cem Kaner, James Bach and Bret Pettichord back in 2002.  I just happened upon a stat that made me sad to see how little we actually read books.  According to Bret Pettichord, "Lessons Learned in Software Testing...has sold 28,000 copies (2/09)".  28,000 copies?!?  In 7 years?!?  While the following assertion is not fully accurate and perhaps not fair, that means there is roughly 30k actively involved testers who consider the context drive approach.  That means my blog post which had the most hits saw roughly 3% of those testers.  Yes, the years are off, yes I don't know if those purchased books were read or if they were bought by companies and read by multiple people.  Lots of unknowns.  Still, that surprised me.  So, to the few testers I do reach, when was the last time you read a test related book?  When are you going to go read another?  Are books dead?

 Metrics: People Are Complicated

Metrics are useful little buggers.  Humans like them.  I've been listening to Alan and Brent in the podcast AB Testing, and they have some firm opinions on how it is important to measure users who don't know they are (or how they are) being measured.  I also just read Jeff Atwood's post about how little we read when we do read (see my above short).  Part of that appears to be those who want to contribute are so excited to get involved (or in a pessimistic view, spew out our ideology) that they fail to actually read what was written.  In Jeff Atwood's article, he points to a page that only exists in the Internet Archives, but had a interesting little quote.  For some context, the post was about a site meant to create a community of forum-style users, using points to encourage those users to write about various topics.

Members without any pre-existing friends on the site had little chance to earn points unless they literally campaigned for them in the comments, encouraging point whoring. Members with lots of friends on the site sat in unimpeachable positions on the scoreboards, encouraging elitism. People became stressed out that they were not earning enough points, and became frustrated because they had no direct control over their scores.

How is it that a metric, even a metric as meaningless as score stressed someone out?  Alan and Brent also talked about gamers buying games just to get Xbox gamer points, spending real money to earn points that don't matter.  Can that happen with more 'invisible metrics' or 'opaque metrics'?  When I try to help my grandmother on the phone deal with Netflix, the fact that they are running 300 AB tests.  What she sees and what I see sometimes varies to my frustration.  Maybe it is the AB testing or maybe it is just a language and training barrier (E.G. Is that flat square with text in it a button or just style or a flyout or a drop down?).

Worse yet, for those who measure, these AB tests don't explain why one is preferred over another.  Instead, that is a story we have to develop afterwards to explain the numbers.  In fact, I just told you several stories about how metrics are misused, but those stories were at least in part told by numbers.  In more theoretical grounds, let us consider a scenario.  Is it only mobile users who are expert level liked the B scenario while users of tablets and computers prefer A?  Assuming you have enough data, you have to ask, 'does your data even show that'?  Knowing the device is easy in comparison, but which devices count as tablets?  How do you know someone is an expert?  Worse yet, what if two people share an account, which one is the expert?  Even if you provide sub-accounts (as Netflix does), not everyone uses one and not consistently.  I'm not saying to ignore the metrics, just know that statistics are at best a proxy for the user's experience.