Wednesday, October 29, 2014

In consideration of ISO 29119

I have been aware of this debate ever since attending CAST 2014, but I've not been quick to sign.  I wanted to investigate and see what other viewpoints there might be.  To quote Everett Hughes, a sociologist:
“In return for access to their extraordinary knowledge in matters of great human importance, society has granted them a mandate for social control in their fields of specialization, a high degree of autonomy in their practice, and a license to determine who shall assume the mantle of professional authority. But in the current climate of criticism, controversy, and dissatisfaction, the bargain is coming unstuck. When the professions’ claim to extraordinary knowledge is so much in question, why should we continue to grant them extraordinary rights and privileges?”
This is a serious question, one for which I appreciate that a standard might seem like a correct method for 'proof' of our professional authority.  However, even if the standard is in fact a professional guide, I don't think anyone but practitioners can judge its value.  As someone who has had a keen interest in trying to understand this standard, and not wanting to judge too quickly, I have tried to get a hold of a great deal of material about it.  I have engaged with some of those whom disagree with my point of view, such as professional tester.  In fact, they have published my response in there Oct. 2014 issue (albeit with some minor mistakes from my draft).  I looked into purchasing the standard.  I have looked for those who are pro-standard and what they've had to say about this movement as well as what they have said in the past.

That being said, I'm not convinced we actually do know what a standard should look like, much less if this is the standard we need.  Maybe it is what we need, but I strongly doubt it.  I think we are likely many years or decades away before we will even be able to claim we have true repeatable practices oriented towards different contexts, assuming it is possible.  I am unable to judge at present what the standard does say, as the standard and the standard body's work is less than transparent.  I've been forced to sign up with personal details in order to read documentation about the standard's creation, although that has been changed since (NOTE: The file name also changed, including a date, which makes it hard to know if anything has changed, as there was no change log as of 10/26/2014).  The standard requests I pay in a currency that I don't use.  I've signing the petition to withdraw the standard not because I know it is wrong, but because I can't tell, which makes it useless at best and dangerously wrong at worst.  What I can tell is that the standard's various author's other documents around the standard demonstrate what I consider to be confusing if not out right contradictory statements, making me doubt the actual standard.  That breaks the social bargain us professionals have that Everett Hughes so eloquently described.

Perhaps you could say that I should have been personally involved in the standard, and that is a valid complaint.  However, I'm not aware of anyone particularly reaching out to the AST, nor have I heard of it from any other group except for James Christie's talk in 2014.  I have attended both AST sponsored and non-AST sponsored conferences for years, so this isn't a case of willful ignorance.  This is my first chance to review the material and process, yet I have not found the process particularly open or transparent.  I see claims of no best practices and claims that the standard will create best practices.  I have found so many confusing statements by the standard's body that I must conclude that the standard should be withdrawn until it can be thoroughly reviewed and modified, if it can even be modified into something useful.

Even ignoring past statements, the recent defense of the standard creates questions.  One of the easiest and most obvious to consider is who wrote the standard.  Then there is the question of who pays for the creation of the standard?  Well clearly this was not just a labor of love, as Dr. Reid says the costs of development have to be passed on to the customer's of the standard.  I should note, my blog makes me no money and I am not a consultant so I have little incentive to make money speaking about this.  I made no money in writing my letter to the editor, and I certainly don't demand you pay to read my work.  I am not discounting the cost of writing the standard, just simply saying that if you plan on having expenses paid by publishing your work, you are not simply doing this out of the kindness of your heart.  There is lots of analysis that could be done just on the defense of the standards alone, but is outside the scope of this particular post.

One of the oddest and most compelling arguments for both sides is from Rex Black, in which he notes that about 98% of all testers won't care one way or another.  I think this is true, which makes the standards mostly not matter, but it also means that those 98% who are silent count on us to ensure these are the right standards, lest they become popular and that silent 98% ends up forced into using them.  I fear that this non-involvement is also further evidence that we testers as a group are not acting like a profession.  It isn't that we don't claim to have "extraordinary knowledge" and Dr. Reid at least seems to argue we mostly agree on this knowledge, but rather the majority of people don't feel any need to actively participate.  I realize this might be an argument for why we need a standard -- to show the disinterested the 'right' way to test, but to me it seems to indicate just how young our industry is and seems to me that shows why we aren't ready for a standard.

Even if the ISO body decided these documents ultimately should stand, the objections of the AST/Context Driven community need to be noted in such a standard.  Furthermore, making the document open will go a long way in allowing the community to discuss this document beyond the smaller standard's committee.  I recognize that ISO needs income to maintain itself and won't publish them for free for everyone, but certainly some sort of 'for individuals, not corporations' license could be used (and I don't mean the sort of non-sense Matt Heusser describes).  Finally, if this is an attempt to demonstrate our commitment to professional testing, then it needs to be accessible to our community.  The work needs to demonstrate it's value rather than being buried away inaccessible to those who would use it.

Tuesday, October 14, 2014

Noise to Signal: An Attempt to Understanding What Smart People Have To Say

Where I've been

Frankly, I don't know how to start this thing.  Perhaps it belongs in my bedroom when I was in high school writing little programs.  Perhaps it belongs just a few years back.  I have and am constantly looking to, in some way, do better for whatever "better" is for me.  I don't pretend to have answers for everyone or even that there is a static answer for myself.

In my bedroom I had finally found an interesting productive task.  I was learning to program.  I would search for hours for that one little magic bean to get me moving along again.  To use a game metaphor, I was leveling up in my knowledge at a rapid pace.

At my first testing job, I did firmware and hardware testing.  I sat around, watching machines work and sometimes break down.  As there was a large hardware component, I found quickly there was little to learn in rapid fashion, so instead I leveled up my political and social abilities, talking with the leads, learning how the organization worked.  I became a lead, and saw that it too was a trap, with just more paper pushing, so I found a different area to do more software-oriented testing.  I learned about LDAP and Kerberos and, and and ....  I was on the path to leveling up.  I was still programming in my bedroom, still writing games, and later different applications I found a need for, or an interest in developing.

Skipping a little, I started developing my own automation.  I built framework after framework, learning what things I didn't like and what I did.  I started in on teaching people.  I have mentored several people, sometimes more successful than others.  I leveled up in different areas around testing and development.  I had built my own testing tool, which admittedly has been neglected for years at this point, but I gained a lot of insight into the underbelly of an automation tool.  I even did market research to see if it was worth starting my own company.  I didn't start my own company for reasons not within scope of this blog, but it appears to have been a wise choice.

All of these things I did on the basis of my judgment and learning.  Isaac might have been a something of a mentor, I did read a few books, but mostly I did it on my own.  The last few years I have been looking at where to go from 'here'.  I have done most of the interesting difficult forms of automation, I have even worked towards pioneering my own methods and frameworks.  So in looking for what's next, I've been trying to learn from smart people and what they have to say, and not just within testing.

What do smart people say?

If you Google about what smart people know or do, you are bound to find pithy pieces of generic advice.  Not to say there aren't perhaps hidden gems in this, but these gems are minor.  Frankly in the world of wisdom, anything involving a list seems like someone who doesn't know what they are talking about but hope there work gets hits.  Add exclamation marks for further effect, and it feels less like real data and more like an advertisement.  Let me give you some examples:
  1. Work Hard!
  2. Don't Quit, Even When It's Hard!
  3. Never Stop Trying!
  4. Ignore Everyone Else and Do What You Think Is Right!
  5. Don't Look Back!
These are all easily disqualifable, be it too generic to be useful or falsifiable (E.G. Ignoring your past makes you a future fool ; another pithy statement, but sometimes true).  So seriously guys, how do I get better?

Well let me give you something that looks like a list but is rather a set of sources I have considered.  These are not quotes, but roughly the idea I took from them.
  • Don't set goals, use systems. - Scott Adams
  • Find things that don't scale in a linear fashion. - Scott Adams, Randy Pausch
  • Change is inevitable. - Robert Heinlein
  • Progress comes from people trying to solve a problem in a more effective/lazy way - Robert Heinlein, Paul Graham, Dr. Richard Hamming
  • Luck matters, but only a little - Too many to list
  • Knowledge and productivity compounds, the earlier or more you learn, the more you'll know later. - Dr. Richard Hamming, Steve Yegge
  • I have managed doubts in what I 'know', and that is okay. - Dr. Richard Hamming, James Christie
  • People's requests and errands take time, up to all of it. - Heinlein, Paul Graham
For those interested, I will include some links at the end of this blog for where I got some of my ideas.  I will stop there, because this isn't meant to be a final list of smart people's knowledge. Instead, I'm going to talk about the challenge Heinlein, Dr. Hamming, Paul Graham and Steve Yegge have all presented in some way or fashion.

Do you know what problems matter?  Why aren't you working on them?

After thinking about this mind-blowing set of questions, it makes me ponder what answer can I give?  I could give answers for my own company, but that feels too small.  I could give answers to the world's problems, but that is too big.  So maybe I can give answers around my industry, with a focus on testing:
  • Why do testers leave at year 5?  
  • Why do automation efforts fail so often (and is that related to people only staying in testing for 5 years or less)?  
  • How should we be teaching the next generation of testers?  Do we have accessible training designed to get people to the 5 year mark?  What about beyond that?
  • Why is only a small percentile of humanity appear useful to me (and am I in that percentile)?  
    • For that matter is my definition of useful a good and valid definition and would investigating that provide me with an interesting problem?  
  • How do we extract the most useful data in the world and provide it to the people who care?
  • How can I get machines to solve most basic problems?  
    • What do we do with those who were working before these problems were automated?
  • How do we effectively train the next generation so they don't repeat the same mistakes as the last generation?
  • When are particular methods of testing more or most productive?
  • How can we significantly improve testing without additional resources (E.G. Tools)?
  • How do we improve or prevent the conditions for sweatshop-like development companies?
  • How do we improve security in our software development?
This list was developed over one night and is not complete, but the important question to me is am I working on them?  Any of them?  However, I didn't write this blog post for me.  The real question is...  Are you working on problems that matter?  And if not, why not?  If you are working on important problems, then let's talk!

[Some] Sources:
Edit, Additional Link:

Thursday, October 2, 2014

Societal Norms, Software Development and Culture


I don't think that everything we do and consider in regards to test should actually revolve around testing.  That is to say, while testing is a primary consideration, it is just one of many considerations.  For example, making our team as optimal as possible might be a consideration.  Say that testing a particular feature would set back the sales group who are trying to do a demo of the product in the environment you are testing on.  Now not testing might in fact be considered an important activity in your testing career.  This post is simply about factors that while are very important in software development, they are not limited to that subject.

I personally tend to read many different and varied development related blogs, and you will find I write broad posts, posts that are not directly tied with testing, but certainly can be applied to testing.  This is one of those; you have been warned.  I have held this article for many months while I have struggled with the words, which I am still not completely happy with, so I hope you forgive the age of the linked article.  In my reading of the post Games Girls Onions, a post specifically about game development, I found some interesting questions I have asked in my career.  I think I can perhaps boil it down to two questions:

1. How do I deal with individuals?

2. How do I deal with groups of individuals?

This particular person found some of the interactions of her male co-workers to be sub-optimal, but the question is, did they intentional ignore those questions, or were they trying to find the right balance between the group and the individual?  Furthermore, if that balance is wrong, is that a cultural thing or a issue with a particular person?  Last but certainly not least, if some particular external group dislikes what the culture does, is it the individual or the group that needs to change?  Lets try to dive into these questions.

In my personal dealings, I have found that each and every individual is different.  That is not news worthy of course, but it is worth thinking about.  I have dealt with thousands of different people, and each one of them had some subtle variation to them.  However, most high level personality traits could be categorized, and those categories can play a part in how you deal with someone in a new situation.


Consider this example, you know your over-zealous co-worker who cares a great deal about usability.  Even though you don't know for sure, you can guess that like other testers you've dealt with in the past, they too will enjoy working in an area they are passionate about.  Offering to trade your UI piece for their API piece might be a good idea if you don't have any particular passion for UI testing.  This of course assumes your boss is okay with it, which makes this at least a small group choice.  Since you know the last few times you have juggled work around, she was totally good with it, so you don't even bother to ask first and just do it.  Lets pause here and consider.  You choose to do the thing that made your co-worker happy while not bothering your boss, as they are likely busy.  From an individual basis, you made your co worker happy.

You didn't consider the team, as you don't 'report' to the team.  Now in a culture that appreciates employee happiness and efficiency, this might work out fine.  In a culture that rather have consistency and order, this choice may be a problem, even when no one is directly affected.  Your boss might respect your choice of taking initiative, but still might chew you out because the group or culture doesn't embrace that style of work.  Maybe the guy you swapped with is disliked by one of the developers on the UI team (as he is too passionate), so this makes things worse for the team.  But who decides these things?  Human activities are really complicated and to put blame on any individual is likely unfair.


Social norms start to show up in a particular culture to define what is or isn't acceptable so that way a group knows how to react and what is acceptable.  I realize saying this might offend someone, but when you try to break social norms, expect the society to attempt to break you.  In the culture of development, there is an assumption that it was a male dominated activity.  Like it or not, that is presently, in most countries, a social norm and those who break that norm are more likely to feel social pressure.  Is it natural?  Norms are natural in human society, this particular norm is just cultural.  I recently heard in a BBC radio interview that one country (no citation, as I can't find the details) has about a 1:1 ratio of women to men in software development.  To consider a different culture and norm, Victorians expected a particular style of dress to fit into a particular class.  If you didn't fit into that dress, you failed their normative test and you were looked down upon.  Now that norm is broken and so we can now wear Jeans acceptably.  On the other hand, in my culture, I can't wear slippers into work everyday without being looked at strangely (if not fired, depending on which job I look back at).

Back to considering the article, the developer talks about her nice window office -- a thing that is usually reserved for higher level developers.  Maybe that isn't the case at her office, but to me that is automatically a status symbol.  Now she notes that one time, a weird incident happen where a drunk co-worker showed up and talked to her in a way she didn't like for 20 minutes.  She talks about being treated differently, because she is being classified in what is defined as a social norm that she did not agree to.  The problem is, we as a society don't have sophisticated categories in our culture which model real human beings.  It is difficult to know exactly how to handle a female developer who accepts crude jokes, wants to be looked upon as socially confident, but then feels threatened by being alone with a drunken coworker.  It is not that I don't appreciate the dichotomy, how situations can be uncomfortable, nor how the details might be important in giving me a more solid opinion.  In culture, norms tend to be broad heuristics and in order to change the cultural and norms, it takes a long time and requires us to make mistakes along the way.

Being a Fraud

In fact, right now, we have a very cold, unfeeling society.  Consider Kevin Fishburne's comment in that post,
From the mortgage industry to pizza joints it's been my experience that regardless of how good or bad your relationships appear at work once you walk out that door they're over. I liken being at work to acting in a play; you're not being you, but what you think you need to be at the time.
 I'm sure there are plenty of exceptions where great lifelong friendships are formed on the job but it hasn't happened to me, sometimes much to my surprise. Not sure if this is just me or an actual phenomenon, but if the latter it may have some bearing on why people act differently at work then they would elsewhere....
The idea that we all follow social norms, faking our way through the work experience appears to not be particularly a-typical.  The fact that the author feels the need to complain about these issues shows she is fighting at least one social norm.  As a tester, it is not unreasonable for us too to want to test our current social situations and attempt to improve them.  One item that I agree with the author on, but am not sure is a 'bug' is the feeling that you are a fraud.  To take a few quotes from the movie Art & Copy,
The frightening and most difficult thing about being what somebody calls a creative person is that you have absolutely no idea where any of your thoughts come from, really. And especially, you don't have any idea about where they're going to come from tomorrow. - Hal Riney

I think most of the creative people are so damn insecure that they want to think they know everything, but they know deep in their hearts they're just in deep trouble from the minute they get up in the morning. So if you can tell them "that's what you're supposed to be", that's kind of liberating. - Dan Wieden

When we are often not even genuine with ourselves, how can we be honest with others?  Dunbar's number (and others) suggests we can't care about more than about 300 people, with the number more likely 150 people.  This isn't just other people, this is you too.  This isn't just you, this is other people as well.  Keep that in mind.  Since our capacity to care about humanity is so low, our ability to repair or change an entire culture is extremely limited.  Thus our social norms are not and can't be designed by any particular person for the masses.  On the other hand, I can only keep track of so many people's personal preferences.  When a culture says something like, "Dirty jokes are okay at work" and the broader culture says "Dirty jokes are never okay when a woman is around," what do you do?  Perhaps your mind checks to see if you know the woman and decide based upon that?  Perhaps you test them with a joke?  What happens if that one joke gets you called in front of HR? I recall one man being written up for calling a group of testers 'test monkeys' (he too was a 'test monkey' in his own opinion), but that insulted one man's sensibilities.  You can't please everyone.

We Are All Unique

The alternative is to say we are all unique.  This idea of course runs counter to Dunbar's number that I can't handle everyone and have to limit myself to a certain set of people whom I use categories for.  Even if we are all unique I can't know how to start with each unique personality.  I start to develop rules, like "this group seems to often act like X" and apply that, but it takes time.  I can't start fresh, not knowing anything.

For example: Does silence means yes or no?

If you answered the question you are wrong.  If you remained silent you might too have been wrong (unless silence means I don't know).  As an example of yes, consider the political process in the EU.  Wiki itself notes that silence doesn't always mean consent.  In one of my Psychology text books back in college, I recall read that in the middle east, a pilot attempted an unscheduled landing and asked for permission to land.  He got no response back and assumed that meant yes.  He got into some rather big trouble when he tried to land and they shot at him, because silence meant no.

If you attempt no assumptions ever, you have to relearn everything every time, and with reading other's writing, you can't even know if the person you are learning is the same each time, as a persona does not equal a person (as Ben Franklin demonstrated).

At least in one example in the article, the author mentions how swearing is fine by her.  One of the commenter noted that she did not feel the same way.  If you can have people upset both ways, you bet there is a third way.  Hell, even asking, which feels a little odd, might upset someone who assumes that the matter is already culturally settled.  Note how I wrote hell just now.  I wonder if I upset any readers or caused them to pause.

Impossibility to Adapt

If you can't adapt because everyone's situation is too dynamic and everything depends, what do you do?  Since we can't adapt and culture is impossible to change by one person, we must suffer, with those who fight to change the situation slowly, making sacrifices for what they believe in.  I don't mean to sound harsh, but the only ways to change a culture is to work really hard at it and often suffer slings and arrows.  Sometimes waiting for others to move on or die off helps, but that takes years.  Alternatively, one can leave or build their own culture.  Basically, no matter which option you take, expect that it will take a lot of work!