Leveling Up Your Testing Skills

Recently I gave a talk intended to be a quick survey of some of the core areas of testing.  I certainly didn't hit everything, but meant to give people tools to go exploring on their own.  I used a text editor for my presentation, because not everything needs power point.  I thought it might be useful to others so I am recreating it here.  I have done some minor modifications, removing things that only made sense in context of a performance.  I also added a few things based upon the discussion during the presentation.

 ----- Page 1

Survey:

Who has heard about AST?

How many of you have read or even skimmed a testing book in the last year?

How many of you have read a white paper?

How many of you have written ANY bugs outside of professional obligations (your job, school, etc.)?

Has anyone tried to formally QA an article?

How many of you have heard about the “No Tester” movement?

How many of you have heard about ISO 29119?

 ----- Page 2

Purpose:

My purpose is not to teach you facts about testing, even if some of that happens, but rather to teach you what questions you didn’t know that you should even ask and show you places you didn’t know to look.

NOTE: I know I cite myself several times, and while there are other sources, few are as narrowly focused to the strategies I am going to talk about..

 ----- Page 3

Do you know the different ‘schools’ of testing?

Four Schools of Testing*: www.testingeducation.org/conference/wtst_pettichord_FSofST2.pdf

Analytical School - Code Coverage, Unit Testing

Factory School - Metrics, Traceability

Context-Driven School: Exploratory, Multidisciplinary based upon context

Quality Assurance School: Protect user, Gatekeeper, Process-Oriented

* [Edit: Since this presentation, I have discovered an updated model including a 5th school, Agile.  See https://www.prismnet.com/~wazmo/papers/four_schools.pdf)

 ----- Page 4

I’m mostly going to speak from the CDT point of view…

(Aside: While I appreciate CDT’s philosophy, be aware that if you approach the community online, some people participating in CDT’s development tend to be passionate/challenging/aggressive and some are very interested in defining words.  Even if you are not interested in that, CDT is still worth looking into, just figure out who you find value from and whom you don't get value from.)

  

 ----- Page 5

Two of the largest “CDT“ classes:

BBST: http://bbst.info/ ; http://www.testingeducation.org/BBST/ ; http://www.associationforsoftwaretesting.org/training/courses/ ; http://altom.training/bbst-foundations/

RST: http://www.satisfice.com/rst.pdf

(Both have free portions, BBST includes all the lectures for free)

  

 ----- Page 6-8

Models and Heuristics:

What are they? Read my work: Words of the Week: Heuristic [& Algorithm]: http://about98percentdone.blogspot.com/2013/10/words-of-week-heuristic-algorithm.html

“A Heuristic is an attempt to create a reasonable solution in a reasonable amount of time.  Heuristics are always Algorithmic in that they have a set of steps, even if those steps are not formal.” 

List of Heuristic Mnemonics: http://www.qualityperspectives.ca/resources_mnemonics.html

SFDIPOT - Structure, Function, Data, Integrations, Platform, Operations, Time

HICCUPPSF - History, Image, Comparable Product, Claims, User Expectations, Product, Purpose, Standards and Statutes, Familiar Problems

Heuristic Test Strategy Model: http://www.satisfice.com/tools/htsm.pdf

How do you know when you are right? http://about98percentdone.blogspot.com/2013/11/how-do-you-know-when-you-are-right.html

Models:

Session-Based Test Management: http://www.satisfice.com/sbtm/index.shtml

Thread-Based Test Management: http://www.satisfice.com/blog/archives/503

Tours: http://www.developsense.com/blog/2009/04/of-testing-tours-and-dashboards

   

 ----- Page 9

Books (just a few):

An Introduction to General Systems Thinking: http://www.amazon.com/exec/obidos/ASIN/0932633498

Exploratory Software Testing: Tips, Tricks, Tours, and Techniques to Guide Test Design: http://www.amazon.com/Exploratory-Software-Testing-Tricks-Techniques/dp/0321636414

Lessons Learned in Software Testing: A Context-Driven Approach: http://www.amazon.com/Lessons-Learned-Software-Testing-Context-Driven/dp/0471081124/

Agile Testing: A Practical Guide for Testers and Agile Teams: http://www.amazon.com/Agile-Testing-Practical-Guide-Testers/dp/0321534468/

 

 ----- Page 10

Self-based learning

Go find a testing job for the weekends like:

http://www.utest.com/

http://sourceforge.net/p/forge/helpwanted/testers/

http://www.guru.com/d/jobs/

http://weekendtesting.com/america

   

 ----- Page 11

Where are you going?

http://blog.red-gate.com/wp-content/uploads/2014/02/Test-Engineering-Skills-v3.pdf

(Go look at this guy, it is an interesting list of skills)

   

 ----- Page 12 & 13 

Experts and active members of the community:

There are different people who have different expertise’s that are worth reading.  Not everyone does much automated testing, while other people are more interested in coaching or test design.  Look at where your strengths are and find people who will be able to supplement them.  Some of these people may in fact oppose my own views, but that too can be valuable.  Debate helps you understand your own point of view.  Here is an example or two….

James Bach* – Testing is an Intellectual Inquiry

James Christie* — Consultant, Professional Auditor and Tester, Started entire ISO 29119 discussion

Jon Bach* — Session-Based Test Management (with his brother)

Michael Bolton — Testing vs Checking

Huib Schoots — Dutch Testing Coach

Scott Barber — Load Test Guru

Rex Black — Proponent of Factory School of Testing***

Harry Robinson — Model Based Testing

Justin Rohrman* — Active in community, board member of AST

Robert Sabourin** — Author of I Am Bug (pictures by his 12 year old daughter)

Jerry Weinberg  — Systems thinking, consulting, author

Karen N. Johnson* – Professional Tester’s Manifesto

Fiona Charles* — Author, Speaker

Matt Heusser* — Founder of Miagi-Do

Cem Kaner — Originator of CDT, BBST, Legal concerns around testing

Alan Page — (Former?) Microsoft Test Architect

* I met these folks at CAST, and you too can meet these sorts of folks at CAST.

** I met Rob at WHOSE, I enjoyed his writing at WHOSE but he tends to be more academic in my experience.
*** [edit:  Since publication my understand of Rex Black's ideas has evolved. Rex specifically rejects the concept of Schools and, as best I understand would prefer I say he's a Proponent of the Strategy of Factory Style Testing in some contexts.  I leave the original work as stands for historical reasons.]

(Some of these people are accessible to talk with, some only speak through their work, some through conferences…)

    

 ----- Page 14

An Exemplar (with more details): 

Doug Hoffman

http://www.softwarequalitymethods.com/h-papers.html

(pay attention to the dates of the papers, some have updated versions)

He has written exhaustively around automation and testing.  While that isn’t the only work he’s done, you will get a good idea of the depths you can go in with automation.  On the other hand, if you want to learn to code, Doug is probably not the man to visit.  He’s interested in theory behind automation.  He was one of the minds behind the ideas High Volume Test Automation and Exploratory Automation.

I wouldn’t go to Doug to find out the latest idea behind Selenium.  I would go look at Doug’s work to understand what sort of automation techniques might make sense to apply to a problem.

    

 ----- Page 15

Questions you need to ask (that I can’t answer):

How much time am I willing to dedicate to learning?

What sort of things do I want to learn?

How varied do I like my studying?

Is this a job or a profession?

Do you want to do something that matters?

Are you okay with having 10 years of experience that really only add up to 1 year of valuable experience?

 ----- Page 15-17

Questions you need to ask (that I can give hints on):

Where can I learn?

 - Blogs

 - BBST/RST and other classes (LST is another CDT class, RBCS is from the factory school)

 - Books

 - Co-workers (particularly challenges)

 - Conferences (CAST - https://www.youtube.com/playlist?list=PLQB4l9iafcelXpJnK6IyDsoFeEb1icqrl , Code Camp, Star West, Star East, Agile testing days, GTAC - https://developers.google.com/google-test-automation-conference/2014/stream , etc. ; Often recorded on Youtube)

 - White papers: http://about98percentdone.blogspot.com/2015/01/white-papers-and-who-owns-your-education.html

What should I learn?

 - Start by glancing at the foundations class.  See how much of it you know already.

 - Read blogs about what interests you.

        - Look up the people I talked about (I suggest Googling "<Name> Testing”.  Almost all of them have blogs).

        - Look at the top blogs and see what they have to say: http://www.testbuffet.com/lists/top_114_software_testing_blogs_2014/page/1/

        - If those methods don’t work, try looking at the fire hose and see if there is an interesting topic: http://www.testingreferences.com/testingnews.php

 - Start your own blog and learn to write; here is mine: http://about98percentdone.blogspot.com

       - Feeling shy?  Why not start by writing comments.

 - Pick up one book.  Read it.  Don’t like the ones I listed?  Check out this extensive list: http://www.testingreferences.com/software_testing_bookstore.php

   - Write a review on your blog

 - QA a news article.  Write up bugs on failed assumptions.
      - http://about98percentdone.blogspot.com/2013/09/testing-babies-for-learning.html

Learn about your interests AND figure out how much you are gaining.  If your energy wanes, move to a different topic or take a break.

What else can I do?

- Figure out what matters and work on that: http://about98percentdone.blogspot.com/2014/10/noise-to-signal-attempt-to.html

- Don’t keep working in the same position for too long.  Changing jobs is healthy.

- If you aren’t much of a reader, but rather audio or video, try some of these: https://flowoftesting.wordpress.com/2014/09/02/podcasts-and-videos-for-testers/

- Learn from the history of testing: http://www.testingreferences.com/testinghistory.php

- Find some heroes and some villains and learn from them (villains will help you clarify your own views): http://about98percentdone.blogspot.com/2014/01/heroes-and-villains.html

Have fun.  If this doesn’t sound fun at all, go find something that does, even if it is a different career.

White Papers and Who Owns Your Education?

Let me start out by point you to this rather complex white paper, A Large Scale Study of Programming Languages and Code Quality in Github.  It inspired me to write a little about reading white papers and why they matter.  In trying to read or perhaps decipher, it made me feel a little stupid.  What, never heard of the Chi-Square test?  How about Cramér's-V?  Does that make you feel like, "Then why are you reading this?  Don't you feel dumb?  You should go back to yahoo news or wherever you came from."  That is the voice in the back of my head, and yes, reading these sorts of papers certainly can make me feel dumb.  Fortunately, I know better.  Even if I am not a accomplished statistician, I'm sure there could be bits of data that are worth gathering.  I personally believe we should try to improve our development* in whatever ways possible, and this study is trying to do so by providing a very high level view of computer languages.  Even though it is not particularly about finding bugs or testing, it does give some insights into bugs and their relationship to languages.
* Pun intended

Let me start with a word of warning before we deep dive into the analysis of this whitepaper.

"One should not overestimate the impact of language on defects.  While these relationships [in the chart in the study] are statistically significant, the effects are quite small.... activity [commits] in a project accounts for the majority of explained deviance [difference in bug rates]... the next closest predictor which accounts for less than one percent of the total deviance, is language."

So while language is important, code churn is more important.  That however does not end the story.  There are still important pieces to pickup.  I am going to leave the definitions of what constitutes a particular category, such as a 'functional language' to the actual paper, which you can look up if it is of interest.  Perhaps you know the language you are working with, so considering a list of languages might be of interest.

"...a greater number of defect fixes... include C, C++, JavaScript, Objective-C, Php, Python. ... Closure, Haskell, Ruby, Scala, and TypeScript... are less likely than average to result in defect fixing commits."

For most companies this is bad news.  Few people write Scala or Haskell on a day to day basis.  Worse yet, I wonder if the experience level of those who write in such languages is greater than average, something the study could not control for.  From what I could tell, they didn't even consider looking at the length of time a user's name was associated to a repository as a proxy for experience.  Since experience was not considered, perhaps the broader categories such as 'functional languages' is more helpful.

"...strongly typed languages are less error prone than average while weakly typed languages are more error prone than average."

"This is strong evidence that functional static languages are less error prone than functional dynamic languages..."

"Functional languages have a smaller relationship to defects than either procedural or scripting languages."

So if you are using a strongly typed language you are better off and if you are using a functional language you are better off.  Does it matter what sort of application you are working on?

"...there is only a weak relationship between domain and language class."

Oh, well I guess then it doesn't really matter what you do, it is how you use the language.  Are there classes of defects that matter?

"Languages matters more for specific categories than it does for defects overall."

"In fact, we noticed 28.89% of all the memory errors in Java come from a memory leak.  In terms of effect size, language has a larger impact on memory defects than all other cause categories."

Interestingly, even though Java has managed memory, it seemed to suffer more from memory problems than most other memory managed languages.  This could be because of the applications considered, such as ellasticsearch, a Java based search engine compared to SparkleShare, a C# file sync tool.  I couldn't find anything language related that seemed obvious for Java's problems with memory compared to C#.  On the other hand Java is way better than any unmanaged memory languages.

"Go has a lot more concurrency bugs related to race conditions due to its race condition detect tool..."

"TypeScript is intended to be used as a static, strongly typed language... However in practice, we noticed developers often [50%]... use any type, a catch-all union type, thus makes TypeScript dynamic and weak."

In looking at what may be wrong with this study, I found it interesting that tool support maybe a reason for differences between languages.  Basically, because the language is only as good as the people who use it and the tools around it, it could be the study gives a false sense of knowledge.  It could be that the use cases are so complex, that this one study and method will not get us much in regards to languages and when to use them.  To be fair, the researchers noted other studies, with their own limitations and similarities.  Confirmatory studies are useful, but humanity is really complex and their study wasn't completely confirmatory.  This is a sort of social-science meets well recorded data.  Git has perfectly recorded data, but even with it, it is hard to know because there is so much data.  There was a 'study' done a few years ago regarding swearing and checkins in github.  While amusing, it also is one more piece of data.  Keeping in mind the warning I started with at the beginning of the article, this was the authors' conclusions

"The data indicates functional languages are better than procedural languages; it suggests that strong typing is better than weak typing; that static typing is better than dynamic; and that managed memory usage is better than unmanaged. Further, that the defect proneness of languages in general is not associated with software domains. Also, languages are more related to individual bug categories than bugs overall."

For the average tester, is there something to learn here?  I think the primary thing is that code churn has more to do with defects than anything else.  Fighting over the tool to use after a project is started is probably not worth it, unless you are rebuilding from scratch.  Keep in mind what else comes along with the language and what value that gives you.  If race conditions matter, then Go is likely a winner.  Just because you have managed memory does not mean you can't have memory issues.  Functional languages like SQL tend to have less defects, possibly because state is less of an issue.  Finally, no matter what language you are using and no matter what you are doing, you will have bugs.  What else did you pick up from the white paper?

Wait, wait... don't drop the curtains yet.

In violation of the end an article with a question to cause action, I want to give pause in considering that perhaps this subject is not to your liking.  Well why not read up on all the data browsers leak out.  It will give you a good feel for just how complicated the simple underlying HTTP connections are.  Maybe you want to learn how to model learning, in which case the Dreyfus Model of Skill Acquisition might be up your alley.  Or perhaps you want something more on testing, such as the pair wise testing white paper.  Or perhaps the famous session based test management paper.  Perhaps none of the ideas I've suggested have struck your fancy.  Cem Kaner developed hundreds of links in his citations for one of his classes, many of which are white papers.  And that is to say nothing of rfcs, a Request for Comments, which helps define standards we all use.  Even books like The Reflective Practitioner, which I am slowly reading through, but has been well analyzed in Wikipedia.

Which brings me to the problem with Wikipedia and blogs like mine that analyze content.  People like myself pull out the conclusions for you and so you skip reading the white paper.  Do you really want to leave your entire education up to those who form conclusions for you?  Let's me be honest, most of you won't click half the links I provided and just as many won't seriously read the cited white paper.  It is not unusual for a large percentage of people not even to read this far, complaining "TL/DR".  "So what?", you, the wise reader who has made it this far might ask.  In my book review of An Introduction to General System’s Thinking I tried to pull out some of my favorite little pieces, some flavour, which was not directly related to the review.  Some of Gerald Weinberg's wisdom will be lost if all you do is read a short summary of his ideas.  Now you say you don't have time, wiki is better than nothing.  Maybe that is true.  I won't argue that you shouldn't use wiki and speed through some of one's education.  Just make sure to occasionally spend time looking at a subject deeply to help deepen your thinking and understanding.  I am sure you might come up with more reasons why you shouldn't, and I won't try to out reason you in your quest to not read deeply.  It is your choice.  I will say it might be hard to do that sort of deep reading, but it will be worth it.

So once again, we are back to the cheesy, end with a question.  Well I won't burden you with that sort of guilt-driven work harder question at the end of this post.  Instead I will give you an offer and a statement.  If anyone leaves a comment with a white paper or rfc for me to read, I will read it and I might even report back on it.  Book suggestions are a dime a dozen and sadly even I have finite amounts of patience for books and a stack of books I want to read, but if you give a recommendation I will at least seriously consider it.  And now for the statement:  I want to gain a deeper understanding of the world, and I will share some of what I learn, but you must do your own investigating as well and I want to say...  Good luck with your own education!