Belief: Absolute Conviction or Probability?

I have long been thinking about the nature of the world, the nature of belief, knowledge, faith, etc. I have come to a conclusion that is both obvious and perhaps to some, scary.  This is one of those personal adventures, and it will take a little bit to explain.  It's also rather abstractly connected to testing, so if you are looking for how-to articles on testing,  I'd go looking here or even here instead.  You've been warned.

By modifying the definition of belief ever so slightly, my language in fact sounds completely reasonable, and I suspect that my version of belief is much closer to the reality of what a belief really is in a psychological sense.  In particular, I can say I have observed it with people who do scientific-like work.  Let's first start with dictionary definition of belief.

According to Merriam-Webster, a belief is:

1. a state or habit of mind in which trust or confidence is placed in some person or thing

2. something believed; especially : a tenet or body of tenets held by a group

3. conviction of the truth of some statement or the reality of some being or phenomenon especially when based on examination of evidence

Knowledge in epistemology is well known to be a "True Justified Belief."  Effectively, the difference between belief and knowledge is that a belief may not be true and it might not be justified. You can only claim to know something when it is true and justified (although how much justification is still in question).  I have also done some informal consideration of justification, which can be valuable in justifying a bug.

In a World...

Imagine, if you will, a world where beliefs were held so tightly that they were in fact the way you ran your life with exact and precise calculations.  You might be like some of the Vulcans who dedicate themselves to pure logic, as described in Star Trek, except your dedicate would be to your own beliefs, not logic per-se.  If, for example, you believed in a set of development principles, you would never break any of these principles.  Assuming your principles were designed correctly to never have functional bugs, a functional bug would not exist in a code base you exclusively created because your beliefs would be personal law, with you never straying. The only possible remaining set of functional bugs are those which are not covered by your beliefs or areas where your personal mental model of your belief was not exactly in congruity with the actual statements of the guiding principle, which is just a form of misunderstanding.

For example, if you stated your belief was "No text box I create will have an injection attack possible.", but your mental model was only based upon SQL injection and did not consider command injections.  Then if you had a vulnerability to a command injection attack in your text box, is it the fault of your beliefs that no text box should be vulnerable to injection attacks?  I think not, because you either had a limit to your understanding of your philosophy (what you believe an injection attack to be) or the world (what injection attacks are known to the world).  However, a more subtle question and interesting question is, did you really hold the belief?  I think the answer is complicated.  Internally, the answer is yes, of course you believed.  However, from an external view did you hold the true belief?  Yes, you held the belief in your own way, however, it might not meet some external evaluator's view of how that belief should be held.  This is a communication problem people run into all the time when they define words differently.

Thus, your personal reality would appear to succumb to your beliefs, or your beliefs would say nothing about reality. That is to say you would ignore your own senses when you observed the natural world contradict your beliefs. This occasionally happens at a developer's desk when they say, "That isn't possible." as I demonstrate a bug to them.  Do they see the world shifting out from under them or are they ignoring it?  Do they see their belief ruined and their world is crashing down?  For some people, this Vulcan-like demand for the perfect belief, the idea of the world crashing down might come close to their view of life.  Then there are others who choose to doubt only the facts they don't like, so that their belief is maintained.  But do these groups go from the point where they say “I believe” and run their life exactly according to their proclaimed beliefs?

Perhaps a few do run their life exactly according to their beliefs, but I suspect that it is more likely most have a journey where you slowly reform both themselves and their beliefs.  Even with all the defenses around their beliefs, and all the denial for the facts contradicting, I think given time to consider, people can migrate beliefs.  Like the developer who claims the bug is not possible, they adjust their beliefs over time. So, either most of us are able to hold two opposing beliefs at the same time ("That isn't possible" and "That just happened") as we reform our beliefs, or are able to ignore those former beliefs somehow, thus truth in the mind and actions are not strongly related.  The latter option is sometimes referred to as rationalization.

For the sake of argument, I am going to assume that in fact people can hold two opposing beliefs at the same time. The reasoning I am assuming this is that if belief of a truth and how one acts have limited relations, then we are creatures whose entire rationale does not matter and thus the entire nature of belief doesn’t matter in this context. Another reason for my assumption is that in working with software testers, I have observed this ability to weight two opposing beliefs.  In my observations, it wasn't that they had ignored their belief and just done something else but rather taken context into consideration.

50% Chance of Opposite Day

If we can in fact hold two opposing “beliefs” then those are not beliefs in the traditional sense.  One is not in fact completely confident in their belief.  Instead, what they almost become is an internal struggle for which belief is more accurate in the modeling of the world.  Since different people model the world differently, a realization occurs that multiple models can both be correct.  In a real sense, the person with said beliefs is weighing non-mathematical probabilities.  Now, if that is what we are doing, trying to decide which version of belief is more likely, and we have the ability to replace one belief with another based upon this evaluation, beliefs are just things we attach a probability on.  Another way of saying this is that they are context driven.  The probability that a particular belief will satisfice is determined through life-experienced context and those probabilities are constantly being re-evaluated as more of life is experienced.

I try very hard to avoid politics in this blog, but I wanted to address a good example of differences in mental models where it is not clear what the meaning or intent is.  Recently, a woman claimed that she is black when her parents disagree with her and claim she is white.  The interesting question here is, what is it to be "black"?  Is it a life style?  Is it a social upbringing?  Is it a description of particular genes?  Is it a color?  At what time and under what conditions?  Is it an origin?  How many generations back?  Is it a description of being disadvantaged? Is it an artificial classification?  How many of these checkboxes do you have to have to be considered black and which oracle(s) do you listen to?  Is this another true Scotsman problem?

If beliefs are actually statements we find likely to be true, we are attaching what I will refer to as a probability. It may not be numerically calculated, but a consideration of one’s experiences, perceptions, internal mental structure, and numerous other factors. Ultimately we come to some sorts of conclusions. "I write great code", "I’m in love", "JCD writes good blog posts", ad nauseum. Can you in fact believe both “men are evil” and that “evil does not exist”? If those are just probabilities, then yes, those can in fact both be beliefs at the same time for one person. Keeping in mind that knowledge is a true justified belief; you can’t “know” both of these things at once, because only one can logically exist at once, but which one? Even with all the brains we have, this appears to be a Gordian knot of a problem. Cutting the knot using things like Occam’s razor only goes so far (It is only a razor!).  Worse yet, the question of context pops its head in here.  "In the past year JCD has written good blog posts" adds context that the first statement did not have.  A developer's defense of "No one would do that" is true in the context of "that I like and hasn't made a mistake and ...."  Perhaps with enough context one could claim knowledge, but you know the saying about building an idiot proof box... I suspect if you could build enough context, someone would just come up with new unconsidered situations.

Why not accept our nature and in fact embrace it? If someone says that “men are evil” and provides limited evidence, you can assign a probability to that (let us assign an arbitrary tag “somewhat likely” to this). Then someone else comes along with highly convincing evidence that in fact evil can’t exist. You weigh the factors you now have and change the scales, applying a “very likely” for "evil can’t exist" and downgrade the “men are evil” viewpoint to “fairly unlikely.” You need not deny either one of those items and yet you can still hold a legitimate opinion that "According to my present data, evil is unlikely to exist."  The tricky thing is, English does not make it easy to give these sorts of answers.

"Nonsense is so good only because common sense is so limited." - George Santayana

In a brief change of topic, I want to stop and address what some readers might be thinking at this point.  I am sure some people see this as non-sense.  So I want to take a brief aside to consider the possibility that this idea doesn't matter.

"In the West, we take the “law of the excluded middle” as self-evident  and  as fundamental  to  logic.  Under  this  rule,  a proposition can have two states: something “is X” or “is not X”. There is no middle ground. In contrast, several Indian logicians assumed four states, “is X”, “is not X”, “is both X and not X” and “is neither X nor not X”. I puzzled over this for many years. I am not confident that my understanding of how this could be meaningful is the same as their understanding. But in human affairs, in interpreting  people,  how  they  interact,  what  situations  they  are  in,  and  how  to negotiate  with  them,  I  came  to  view  the exclusion of the middle as a heuristic rather than a law or a rule. It is often useful to assume that “X or not X” are the only two possibilities, but when that is not productive, it is time to look for a third way." - Dr. Cem Kaner, Tea-time with Testers, June 2015, Pg 61

I could go on, as Dr. Kaner certainly has more around this.  The idea that perception is in fact reality is often skewed.  In the same magazine issue, Jerry Weinberg tells a story in which the three team members whom did all the talking were perceived as the leaders performing actions but the the one with an effective action was not the team member being defined as the leader.  The team member with the affective action was another employee whom only did one thing, silently solve the problem.  Perceiving motion for action is not an uncommon occurrence.

The point I'm trying to make is those whom see belief as a solid, absolute thing probably have a more difficult time seeing that 'reality' is more difficult to pin down.  It seems likely to me, that for most people, as they gain more experience, they too will reform their view of belief into something more generalized, such as a probability.

"The unexamined life is not worth living." - Socrates

Briefly, I want to address another sticky subject, faith.  Faith is a belief with only external sources for justification. You might be able to cite examples where the external source was previously correct, but the external source is still your only justification. Faith is often replaceable with “trust” or “hope”. You can say I have faith that login is broken because my manager is telling me it is so. The justification is based upon hope or trust in some external or internal force that your belief is true. If instead we consider faith a probability based upon your level of trust in a given oracle, you can in fact see faith is just a specialized description of a belief.

One problem with this is our language does not naturally imply that our statements are guess work, nor easily allow one to explain what the most likely evaluation you have thus far computed. People say “I know ...”, or “It must be ...”. We assert our knowledge because people are not actually comfortable with “I don’t know, but I think ...” It is in our nature to say we are right. We even prefer people in which we perceive confidence, even though it maybe a lie.  It is also convenient:

"A little inaccuracy sometimes saves a ton of explanation." - Saki

Compare "According to my present data, evil is unlikely to exist." and the shorten "evil does not exist."  In my view, these absolutes often don't represent what people really think, but I admit it's only a probability.

Another problem is that people will claim this is a flip-flop philosophy that does not in fact hold a person’s “feet to the fire”. While this is true to a certain extent, it is designed to actually be more in line with how people and our estimation of reality actually work. People are rarely able to be self-consistent in what they say, much less what they do. How often do people say “Do as I say, not as I do.”?

In recent years, people have been able to pull out long and specific quotes from others because of how our technology records everything.  We are very much able to examine other people's lives.  Yet, in looking for this consistency, we fail to examine our own lives and the changes we make.  If other people are often unable to be consistent, it's likely you too will likely fail at consistency (Consider: did you know every time you recall something, the memory is re-written and thus modified?).  Yet we like consistency.  There are still two obvious options.  Either we are really failures in consistency but pretend we are not or we are constantly evolving our understanding and are designed to evolve our thinking.  I'd give it about a 20% / 80% probability for anyone individual in circumstances similar to mine when I consider....Oh you get the idea.

Ultimately this is a small chunk of analysis attempting to describe my view of how people work.  It is hard, if not impossible, to write out a full view due to the fact no one else has experienced my life and the context that brings.  I also recognize other views exist and I think they have some probability of them being correct and am open to replacing my own view with a more probable one.  So please feel free to share any insights you have, even if they contradict my own view.  I encourage you to sit down and attempt to write out your views on how people work.  It can be a useful exercise, as it will give your opinions a more solid foundation.  But if you have done all that hard work, why not post it as a comment?

Autonomation: Old is New Again; Toyota and Lean

I came upon the word Autonomation recently and felt it was interesting enough to bring it up.  The concept comes from Toyota as they were developing Lean Manufacturing in the 1990s.  The primary goal for Lean Manufacturing is to eliminate waste and thus improve production and quality. Autonomation is also referred to as jidoka in Toyota's TPS process.  Autonomation or jidoka is one of the layers in Lean Manufacturing, meant to trap failures rather than produce results.  The earliest example Toyota notes is from 1924.

In 1896, Sakichi Toyoda invented Japan's first self-powered loom called the "Toyoda Power Loom." Subsequently, he incorporated numerous revolutionary inventions into his looms, including the weft-breakage automatic stopping device (which automatically stopped the loom when a thread breakage was detected), the warp supply device and the automatic shuttle changer. Then, in 1924, Sakichi invented the world's first automatic loom, called the "Type-G Toyoda Automatic Loom (with non-stop shuttle-change motion)" which could change shuttles without stopping operation. The Toyota term "jido" is applied to a machine with a built-in device for making judgments, whereas the regular Japanese term "jido" (automation) is simply applied to a machine that moves on its own. Jidoka refers to "automation with a human touch," as opposed to a machine that simply moves under the monitoring and supervision of an operator. Since the loom stopped when a problem arose, no defective products were produced. This meant that a single operator could be put in charge of numerous looms, resulting in a tremendous improvement in productivity. - http://www.toyota-global.com/company/vision_philosophy/toyota_production_system/jidoka.html

The term Autonomation feels similar to the term I coined perviously, Manumation.  In the wiki article on Autonomation, I found it interesting that Shigeo Shingo claimed there were 23 stages between manual and fully automated processes.  Unfortunately, there is no citation for where that claim was made and while I saw others repeat the claim, no one had any citation nor data on the stages that I could find.  In my mind, Autonomation is just one form of Manumation.  However, it is also an attitude.  You don't have to try to fully automate something on your first attempt to create automation.  The idea that you set your automation up knowing it will fail, that it will need humans but don't bother the humans until it fails and that the failure is easily traceable and fixable.  Also, it means attempting to fail quickly rather than generating a bunch of waste work.

Ultimately, automation of any sort is meant to help people.  If it helps you get work done, even if it requires a human touch, it is worth considering.  What sort of autonomation have you done?

Happiness: On the Job and in Life

In this post, Matt Heusser does a great job describing some of the problems around hiring older workers and some of the reasons older workers become less valued.  Obviously, this is a concern for all technology workers in the field, regardless if you are in test, operations or development.  I have not had much experience with it, but I do feel some of the call that those who are in the 40s and 50s start to feel.  I don't want to invest hours upon hours doing development outside of work while working 40 hours a week.  This blog, which I write for often, requires time which I could use to get money from somewhere else.  I don't get paid to do this.  Sometimes I clarify my own thoughts, or use this to prompt me to learn, but more often than not I am actually trying to teach others the hopefully valuable information I have.

However, there is a underlying subject I think is more important than just ageism in the work place.  I want to talk about happiness.

First, a few words, I am not a doctor of any sort, I am not giving advice for depression and if you are experiencing depression you should see a doctor.  One other thing, I'm going to ask you to resist clicking all the exciting links until after you've finished reading this article, as there is a metric ton of data on the subject, but interrupting the narrative will likely distract you.  Now on with the show...

Were We Happy In The Past?

In the past, almost everyone did farming.  There were a few other jobs, like tailor, doctor and merchant, but most people were farmers.  These jobs were fairly well known and a town depended on one (hopefully) competent doctor, one tailor, etc.  A large town might have a few of each, but only in the cities were there really any large quantities of any specialties.  These specialties were all relied upon and considered vital.  While there was anger around the usage of power and prestige these ranks sometimes bestowed, in general they were good jobs.  Life could be brutish and short.  However, the idea of manifest destiny  and excitement of the renaissance show that life in the past could be seen as meaningful.  Is being a farmer happy making?  Or being a doctor or ...?  Sadly, there is little documentation I could find about happiness in the distant past.  However, I did find lots on various problems we see an experience today.  It seems that the general fighting we have today was going on in the past.

It is strange that there should be so little reading in the world, and so much writing. People in general do not willingly read, if they can have any thing else to amuse them. -  Richard Burke to Samuel Johnson, 1783.

The number of technical students has declined constantly... IF THE SAME TEACHER CAN INSTRUCT 500,000 OR MORE [students] SIMULTANEOUSLY? - Radio Electronics, 1956, http://www.americanradiohistory.com/Archive-Radio-Electronics/50s/1956/Radio-Electronics-1956-05.pdf

So while I have been unable to find anything quantitatively correlated to happiness in the distant past (and only a few data point in the recent past), it does seem that the same challenges facing us have been unchanged in the past few hundred years.  We are constantly looking for productivity changes while desiring to keep our hard gotten gains.  Furthermore, while I have yet to read this tome, Steven Pinker suggests that we were actually more violent in the past.  So if violence decreases happiness, which I have found lots of evidence for, it would appear there is little basis for the assumption we were happier in the past.  In fact, it appears arguable that collectively we might be over all happier now then in the past!

Do Jobs Affect Happiness?

The next question is, do we know if farmers of today are happier compared to the general population?
In a word, no.  Happiness and job are nearly unrelated, however there are a few key factors that matter.  One is that you have enough money to survive and not worry about where your next meal will come from.  Another is having meaningful work and yet another is having autonomy.  There are also some negative correlations with jobs.  For example, not getting enough sleep or too much mental stress (some physical stress is good for you).  Lacking creativity can be an indirect form of mental stress as well.  Often this comes in a form known as boredom.  Basically, I think I can sum it all up in a single sentence:

You want a job that pays a living wage with difficult but surmountable and clear goals in which you have enough control to get it done and in which you will get back useful unambiguous feedback and time to recharge.

Great, we're done, right?  Well, I want to explore some more topics around this and how I think the reason our job does not seem highly connected to happiness.  The first is that we don't know how to practice being happy, so why would ones choice of job have an affect?  That is to say, if happiness is fairly mysterious then choice of job will appear to be a random or not related to happiness.  So how do you practice happiness?  For that matter, how can we discuss happiness when there are so many varying view on how to create it.  Is it something that comes without you noticing it or is it something you can actively seek?  If significant increases in wealth doesn't change happiness but for a small and short period of time, what does?

Rather than going on with so many links, I'm going to stop citing so many sources.  The purpose of this essay is not to do research for you but rather to explore the subject and not just trends I have seen in the research.  Furthermore, I wish to explore how to have a happy life, not just being happy on the job.

How We "Need" Society for Happiness

Helping others can, in and of itself hurt those we hope to help, without regards to the permission they give. You might have heard the phrase "What doesn't kill you makes you stronger." The correlation to that would be, "What makes life easier makes you weaker."  Obviously things like chronic pain and long term mental stress eventually do damage to a human being, but in general these two ideas have some truth to them.  Consider the case of the blind, a blind person might appreciate being given an 'extra hand' but at the same time, that might also make them less able to do things on their own when that helping hand isn't around.  Married couples are known to have slightly lower IQ than the unmarried because of dependencies on each other.

In a sense, as John Donne said, no man is an island. That does not equal a learned dependency in my book. Babies have real dependencies. We are dependent on doctors during surgery. We depend on lots of things. However, learned dependencies are not things we are incapable of doing. Maybe we don't want to do them or maybe it would cost too much of our time, but a learned dependency is when you quit trying because you're not expected to try. I suppose you could say I quit trying to change my oil because Jiffy Lube says I'm not expected to try. However, I would argue that my dependency on oil changes is more a matter of time, energy and my personal interest. That being said, it is a fine line.

In regards to happiness, is happiness truly just an internal thing or does it have external dependencies?  Can you live and enjoy pure silence, or do we really need music?  It is an interesting question, but even if you depend on external things, they may not really require other humans.  Enjoying waterfalls or existing books for example require no (new) human work.

Learned dependency, be it having someone else get you around, using a calculator to do math or depending on others for happiness is a problem because now you need something else in order to do the things that you want. I don't know how to give an algorithmic way of judging when it is enabling and undermining independence and when it is a genuine hand up.  Perhaps it is even in part personal taste, as the hermit choose no or few dependencies while the city dweller depends on many people for city services.  The hermit might even see those city services as enabling the city dweller, as the city dweller needs the bus in order to go see the opera they desire to see.  That is always a struggle, and has been one for ages.

If our dependencies on others needs to be managed, it seems only reasonable to ask about the other way around.  How do I prevent asserting myself into other people's need for genuine challenge? The Bible famously asks the question:

Am I my brother's keeper? - Genesis 4:9

In that case it was a way of ignoring responsibility for the murder of his brother, but the question of responsibility and social value for watching out for others has been around for centuries.  I don't pretend I will answer the question today, but I think it is a balancing act.  Rarely do we know what we really want. Our simulators for happiness are poor at best. Dan Gilbert talked about that in his TED talk.  So if we have a hard time knowing what will make us happy, pretending we know what other people want is silly.  This can become a wicked problem when what you want to do is help others but worry about foisting your culture, will or opinions on others without giving them room to grow.  Teachers suffer this question all the time.

 Can Happiness be Treated as a Duty?

The Stoic's claimed virtue is sufficient for happiness and that virtue is created by a will that is aligned with nature.  Kant's Categorical Imperative, in a very rough and simplified form, linked duty to what you imagine a reasonable person would do.  So unless you imagine a reasonable person would choose to be unhappy most of the time, it would appear happiness might in fact be a duty.  There are other philosophical theories like Hedonism which say that you should look to maximize long term happiness.  That is to say, have the day to day discipline to make your life better in the long term.  All of these various ideas about optimizing life with consideration for happiness seem to have something of a pattern to them.  They all push us to grow.  Be it via virtue, by considering how a reasonable person would live or look at the long term and attempting to minimize pain.

So how do you generate growth?  Well one aspect is to not be in fear.  Fear often is the opposite of growth.  You need to feel safe enough that you can work.  Even when in fear, such as when in a castle surrounded by an army, you have enough pause to feel safe enough to gather your wits and think.  Safety by itself isn't enough.  While safety is important, you have to be safe to try experiments, pushing yourself beyond what you have done before or in new ways allows for the growth, which leads towards happiness.  The feeling of growth is in my view, at least one of the causes of happiness.  However, if one only attempts to grow in one area, it can become an addiction.  Growth also has to relate to goals, and in particular unending goals.  Goals you can meet can later feel unsatisfying.  For example,  when you think of buying something, you feel more satisfaction then after you have purchased the item.  Since having meetable goals is less likely to create happiness, how do you create a unmeetable goal?  You could say, "I will weigh 150 lbs in 3 months," but this is an unsatisfying goal.  If you make it, maintaining it is much more difficult.  Instead, I will go to the gym 3 days a week remains satisfying because there is no 'end point'.

In addition to safety, you also need time to develop happiness.  You need to have time to work on your goals, or else you stagnate into surviving life.  Time can mean exact and dedicated time, such as a day of the week or hour in the day, or it can mean just having open variable spots.  Time is perhaps one of the trickier things to give detail on because different people like doing things in different ways.  Some people like things well planned out while others like to be more sporadic.  I have not seen evidence demonstrating that one is superior to the other, but personally I rather have some vague planning rather than chaos or an exact fixed schedule.

While the goals don't change often, they should be allowed some variety.  If you always go to the gym on Monday, Wednesday and Friday, you may end up burnt out.  Instead, you probably should have some time allocated to 'breaking' the goal.  Going to the library once a month instead of your Wednesday workout.  You should feel the pull to get back into the routine which helps make the routine more satisfying.  If you don't feel excited to go back into the routine, it demonstrates something is wrong with your goal.

Finally, you need to be able to see that these goals are working.  Without being able to check up on your goals, without being able to study the results and see if they match expectations you get little satisfaction from 'achieving' them.  While weighing 150 lbs might be a very watchable (E.G. an easy metric), you don't need that to see if going to the gym is helping you or not.  Since using easy metrics tend to make goals that end, those are likely not the first type of metric choices to make.  Saving enough money to buy a house is an admirable goal, but learning the habit of saving should be the real goal.  So using 'enough money to buy a house' as the measurement means you're probably watching the wrong thing.

Now if you have been reading this far, you'll probably notice something interesting.  I'm going to break a general rule I have of not copying and pasting and repeat myself from above.  Basically, I think I can sum it all up in a single sentence:

You want a job that pays a living wage with difficult but surmountable and clear goals in which you have enough control to get it done and in which you will get back useful unambiguous feedback and time to recharge.

If you noticed what I wrote in the job section and in the life section, you can see the life section is very close to a repeat.  In fact I would use the job section as a sort of check sum from what was discussed in the broader life category.

The idea that you can treat happiness as a duty is not exactly correct.  Instead, just like with getting a job that creates happiness, you need to do similar things in life.  It maybe formulaic sounding, but it is more complicated in one's actual life than any author can generalize.  A duty implies it is required, but this is more of a method, backed by out current science and history.  Can other methods work?  Maybe.  Is what I have written 100% correct for your life.  Very likely not.  All situations vary.  But the general ideas seem to remain consistent throughout time.

Disagree with me?  Well, my goal is to engage my readers, so please leave a comment.  I'll be happy to reply.  Oh yes, and now feel free to go click all those links and enjoy all that awesome data.

Book Consideration: Tools of Critical Thinking: Metathoughts for Psychology by David A Levy

“Cogito ero[sic] sum” – Rene Descartes.* Those were the first words I saw in the book that got me excited to read it, although disappointed at the typographical error. As I made it through Tools of Critical Thinking, I found much of what the book describes I already understood, and much of it came from a class I took years ago in philosophy and logic. That being said, it did give me a little bit of thought in regards to psychology. For example, the book talks about how words have implied value. Other concepts include how concepts are not things, there are different levels of an idea and naming something does not imply you understand it. These ideas are all things I understood reasonably well. Then the book seemed to have started pulling quotes from Slashdot, with the whole correlation is not causation. Literally five chapters were dedicated to this one concept, and I think that is perhaps not excessive, but for me, it felt like an “I already know that, teach me something new”.

As I continued my personal journey through the book, I found one very interesting new concept I had not heard of before. Fundamental Attribution Error. Sounds a little complex, but it really isn’t that hard to understand. Roughly, it means that we tend to overestimate a person’s personality and underestimate their situation. Strangely enough, the reverse seems to occur frequently when you are looking at yourself.

The author also talks of other subjects like extraordinary events and how given enough events, you would expect some to occur. One case I experienced involved me leaving a parking lot following a person and then, about an hour later, on the way back from lunch on that same day, being followed back to the same parking lot by that exact same person. I found it really funny, because of the impossibility to it, yet it happened. That being said, I don’t seriously take it as anything but a fun and funny event. At the time this type of event occurs, perhaps it is reasonable to have a little “brain breakage” and see it is something more, but one should understand that random event occur throughout our lives and within chaos patterns emerge (see fractals).

Mr. Levy continues on about deductive and inductive reasoning and what poor conclusions can be made from what appears to be reasonable reasoning, which again is covered by logic 101. My favorite flaw goes something like this: Spaghetti is a food therefore I am right. Basically, if you have a true premise (Spaghetti is a food) you can conclude anything and claim your logic is infallible. There is also inductive logic where you base your conclusion on a single specific type of evidence. This has its own set of flaws, like a low sample size creating false new knowledge.

Other topics of interest include the idea that an observer affects the observed (E.G. To Grok; See Stranger in a Strange Land). We apply schemas to all sorts of data, from gender roles to mental illness to height to various other items. Most people have a personal investment in their beliefs (strangely, belief isn’t defined). To know and label something is not to solve the problem (which should be obvious in the world of QA, but think about it with how you interact with people).

Now I want to circle back to my own job, testing. Is this a testing book, does this book have value in testing? I think the answer is yes, but only if you don’t already have these logical concepts already well understood. Not to say a review doesn’t help, but honestly I think one could get much out of reading a logic book. That being said, I suggest one read the chapter on schemas, confirmation bias, deductive logic, as they apply very nicely for much of what we do. Let me give an example of how this applies to my job using the chapter on representative bias. This is one chapter I have some issue with, not because it is wrong, but because I think it is not a complete picture. The book talks of assuming things based upon what you are observing (for example a person you are observing). As a real life example, if you have found one developer typically does good work, you might choose to do less QA for work they do, but is that because of personal feelings or hard data? I don’t have hard data, but I can say I never regretted choosing to do less QA on one particular former developer I did work with and instead spent more time focused on other developers. Is that in fact a confirmation bias on my part? I think not*.

One last subject I would like to talk of is schemas and how we develop them as people. I have had a long running set of conversations with a friend on the value/cost of applying a “Schema” (labels) to people, ideas, groups, etc. The obvious pro in my mind is that the schema provides a way of connecting us together. That is to say, we can learn new data by applying old schemas to new situations. For example, I could say that males tend to be more aggressive and more aggressive people tend to play more sports**. From this you might apply a schema pattern match on a male athlete and “guess” he is more likely to be aggressive. Now, you have no evidence for this particular case, which is where testing comes in (in my mind) to confirm or deny that in a particular case. The trick is to keep an open mind and understand that no matter what your judgment shows, it could be wrong and to accept it.  Even when you have a well developed schema, there is likely to be some outliers which you need to expect and be flexible enough to believe exist. If you don’t keep an open mind, you can end up dividing people into us vs. them, which does not provide real long term value to humanity.

The other approach is to say, “I simply don’t know without direct observation. Even then, my observations might be flawed, and without reproducibility, I know nothing.” This method (to me) is also reasonable, as it has no value-judgment and thus requires no schema. In other words, every situation is unique, so a schema is too rigid for the real world data. The problem I personally have with it is it also provides no structure, possibly no value, any way of generically apply new knowledge and no hope to really learn. In a very real sense, it is giving up on gaining structured knowledge, because your knowledge will always be limited, flawed and low resolution. To be clear, and fair, I think that it takes an amazing amount of will to refuse to judge and simply keep an open mind since anything is possible. Just the pure discipline might make this method worth committing to.

I could go on for pages on schemas, but I think that if you don’t understand what you are doing when you sort types of knowledge, you really need to read this book.  Also, if you exchange the word “schema” with “database design”, it also comes out as an interesting talk on SQL vs. No-SQL solutions.

In summing it up, I think it is a handy book for a QA/Test professional, handy enough I bought my own copy of the book.

* This reminds me of a joke I couldn’t resist not including. Rene Descartes walks into a bar and the bar tender asks if he would like a beer. Descartes says, “I think not!” and disappears in a puff of logic.

** This statement maybe untrue.  I ask you to apply your own reasoning and conclusions around the subject.

Tainter's Composite

Introduction

First let me start by defining my terms.  Dr. Joseph Tainter, is an anthropologist who looked into the question of why societies collapse.  A composite is combining multiple things into a single image.  I believe using Tainer's mode of thinking, one can create a model or system including for software and organizations.  This model, using a lot of systems thinking includes the work of the Dreyfus model, The Gervais Principle and general economics, but the framework all rests on is from Tainter.  I will explain Tainer's ideas, however, if you don't know about the Dreyfus model or Gervais Principle, you will need to read up on those before continuing further.  In fact, without those you will be lost.  I realize it is a huge amount of reading just to read this post, but I promise you it is worth your while if you want to understand the inner workings of business culture.

Tainter asserts that societies become more complex as the societies needs for solutions to problems increase.  That is to say, in order to solve a problem, the society adds complexity.  For instance, when a tax loophole is discovered, society might create a rule around that loophole to end it.  This complexity means that someone must create the law, the people who are affected must learn of it, there must be a means of enforcing the law, someone has to interpret the law and when violated someone must enforce the penalty.  Most laws might add minor complexity, but at some point the laws become uncountable, making it very difficult to follow the law.  Now this by itself might not be a problem, unless the value received from additional complexity declines compared to the cost.  When enough of these sorts of mounting complexities cost more to the society than the society produces, eventually the society will fall.

A Corporate Example

Let me consider a different form of this, one that is very easy to understand.  Let us suppose that you have a piece of software that requires 9 engineers but has an income stream sufficient to pay for 10 engineers.  Customers keep demanding new features, and the complexity of the system rises.  Each new features has both cross-cutting concerns and interlaces with existing features.  More bugs are found, but few new customers are added.  Eventually, those whom knew the project will leave, but the complexity of the project does not.  The company finds itself needing more engineers in order to support the product.  The new engineers conclude that the best thing to do is to rewrite the application.  Even if you ignore Joel's words (from the last link about refactoring vs rewrite), you know that then they had to hire even more engineers to support the old application while the rewrite was in progress.  The company had not been making that much money, didn't appear to be able to make much more and so the obvious thing to do is the close shop.  A sociopath from The Gervais Principle would clearly do so.  They would not feel bad for the engineers who lost their jobs, for the customers who lost the product or even the history of the company.

This scenario has not happened to me personally, but I know that at least twice my entire automation code set was abandoned because the person whom replaced me was either not an expert or was at least less experienced compared to me in their programming skills and the company had no one else beside myself who knew the automation code base.  Instead of trying to learn the code base, the less knowledgeable people threw it away and started over.  Why do that?  My guess is the person doesn't understand the complexity in the system and decide that the last guy or gal must have been an idiot.  I too have inherited code bases, and not once have I started over from scratch.  I have thrown away pieces and parts, but never the whole, so I can't directly answer it.  I suppose, I once discover there was former automation scripts about 6 months after I started a project written in a different programming language.  In that particular case, the code developed had already outstripped all the functionality from the existing discovered code.  Even in this case, I still reviewed the work to ensure we had capture the same set of solutions.  This leads into an interesting question of losing complexity by forgetting, however, I am not going to cover that topic.

Company Growth & Avoid Responsibility

This also applies to big companies and as I alluded to earlier, governments.  According to Tainer, complexity growth without equal or greater forms of production end up with an eventual collapse.  Consider HR policies of different organizations.  With self-employment, there is no HR, thus zero complexity.  With a company of just a few people there is likely still no HR, but rather just a group of people all working towards a unified goal.  Often these people are very close.  Eventually there is a need for someone to manage the complexity and you hire someone into HR.  As you keep growing, someone does a boneheaded thing, like not taking time off and not call in sick for a week and then show up expecting that they still have a job.  There is no policy saying they will be fired and so they demand that they either keep their job or get unemployment.  So a policy is added, no big deal...

Then you get to be a mega corp with 100,000 employees.  You have divisions bigger than most medium sized companies and you have a handbook big enough to use as a deadly weapon.  Big organizations are often not run nearly as kindly towards employees because (in part) no one is empowered enough to go do that or if they are, it is spread unevenly causing jealousy, which causes things to then be 'not allowed'.  Having a pizza party for success now makes 500 others unhappy as they smell your reward.  You made 50 people happy and 500 grumpy.  An expert would have seen the problem ahead of time and planned on making it outdoors or away from the other employees, but the person who had the party didn't know better. So the company decides to make a rule saying you can't do that.  Eventually all these rules add up.  They hurt morale because the Dreyfus expert knows its stupid but has to think about how to get around the rule, making their 'expert' skill less valuable (as the point of the expert is they don't use rules).  They, like most Gervais losers (economic losers) flee, making such companies lose their best employees. It also encourages people to not try to make work better as that would break norms and likely added more rules to the already overly large set of rules they have to deal with.  Finally it encourages people to quit thinking because there is a rule, making them even less effective.

There is yet another reason that rules are created.  As The Gervais Principle describes, the sociopaths are always interested in getting the clueless to accept as much responsibility as possible while giving limited credit or authority.  The clueless are a buffer for the sociopaths so that they get as much value as possible from the losers without having to deal with the losers.  This is a form of protection that a zealous clueless person is willing to do, however, the clueless often don't know how to actually run an organization.  The clueless are in a position in which they use baby talk to pretend they know what they are doing.  Since they don't actually know what they are doing, the sociopaths use rules to force the clueless to do what the sociopath thinks is right.  These types of rules create a different type of complexity.  This is a social complexity, where the sociopath must juggle personalities to get the most production.  The clueless is actually a cost, but a small cost considering they are a sort of hedge that if things go wrong, the clueless can be blamed.  In a larger sense, this means that the person with bad ideas but who is good at manipulation can maintain power for a long time in spite of making bad choices because those choices aren't attributed to that person.  This can eat away at an organization or society, building up costs from complex institutions with mild production value, further corrupting the society.

Cat & Mouse

Capitalism is an attempt to alleviate the ineffective manager who somehow remains employed via competition.  The problem with capitalism, is like any system, the complexity builds up to protect the system from being hacked.  This can be anything from monopoly laws to defenses from regulatory capture.  However, this sort of question is mostly capture by Tainter himself, so I won't dive into it.

Ultimately the Tainter Composite describes how complexity not only infects society but the institutions around society.  The final piece I want to describe is how this leads to a cat and mouse set of activities.  Often rules are made because there is a real and legitimate concern.  Some of my examples might seem silly but are actually true (I have experienced some of them).  However, even 'smart' rules can get you into trouble.

Consider black hat crackers and the white hat hackers.  Code is in fact just a form of rules, often very useful rules.  You are currently reading this using a system with more lines of rules than one person could memorize.  The trouble is, there is always a group who wants to enjoy finding ways around these rules for their own purposes, just like some of the troubles in capitalism.  It might be someone who just adds everyone to your friends list or it might be to take all the money out of your bank account.  As security gets better, more rules have to be applied to protect the system.  Will code ever get to the point where maintaining and adding rules are too expensive for one person ?  Will it ever get to the point where building up all the rules is more costly than accepting the attacks?  To some degree that has happened where most people have traded up freedom (complexity) for a walled garden (less value).  Once again, this gets into reducing complexity, which is a topic for another day.

Rather than telling you how this applies to testing, effectively creating rules for your brain, I am going to try not to add complexity and let you interpret it for yourself.  If you found this useful, please write a comment and I will write more on the topic, otherwise, I will leave this dense, difficult topic alone for a while.

My Current Test Framework: Testing large datasets

I recently wrote how I felt few people talk about their framework design in any detail.  I feel this is a shame, and should be corrected as soon as possible.  Unfortunately, most companies don't allow software, including testing automation to be released into the public.  So most of the code we see is from consultants with companies occasionally okaying something.  In my case, I did something like a clean room implementation of my code.  It is much simplified, does not demonstrate anything I did for my company nor does it directly reference them.  It is open source and free to use.  Without further delay, here is the link: https://github.com/jc-d/Demos.

It is in Java and was intended as part of a demo for a 2 hour presentation.  Because of the complexity of the system, I'll write some notes about it here.  I used IntelliJ to develop this code and recommend using it to view the code.  It does use Maven for the libraries, including TestNG which you can run from IntelliJ.  Many of the concepts could be translated to C# with little difficulty.

So what does it do?  It demonstrates a few different, simple examples of reflections and then a build up of methods for generating test data in reflective and possibly smarter fashion (depending on context).  I'm sure your sick of hearing about reflections from me, so I'll try to make this my last talk on them for a while, unless I come up with something new and clever.

As a brief aside, Isaac claims that while this is a valiant attempt to write out a code walkthrough, it really needs to be a video or audio recording.  Perhaps so, but I don't want to devote the time unless people want it or will find it useful.  As I have my doubts, I'm going to let the text stand and see if I get anyone requesting a video.  If I do maybe I'll put some time into it.  Maybe. :)

Now on to the code...

First the simple examples which I do use in my framework but not as simple as this.  The two simple examples are DebugData and ExampleOfList and both are under test/java/SimpleReflections.  DebugData shows how you can use reflections to print out a simple object one level. down.  It 'toStrings' each field in the object given.  Obviously if you wanted sub-fields that would take more complex code, but this is often useful.  ExampleOfList takes a list of string and runs a method on each item on the list and returns back the modified list.  Obviously this could be any command, but for simplicity of the demo I limited it to methods that did not take arguments.

Now all the rest of the code is around different methods for generating data.  I will briefly describe each of them and if you want to you can review the code. 

The HardcodedNaiveApproach is where all of the values are hard coded by using quoted strings.  E.G. x.setValue("Hard coded");  This is a good method for 1-3 tests but if you need more you probably don't want to copy and paste that data.  It is hard to maintain, so you might go to the HardcodedSmarterApproach.  This method uses functions to return objects with static data so you can follow the DRY principle.  However, all the data is the same each time.  So you add some random value, maybe append it to the end.  The problem is what are your equivalent class values?  For example, do you want to generate all Unicode characters?  What about the error and 'null' characters?  Are negative numbers equally valid to positive numbers.  If not, then your methods are less DRY than you might want as you will need different methods for each boundary, if that matters.  Also you are writing setters for each value, which might fail when a new property is added.  We haven't even talked about validation yet, which would require custom validators based upon the success/failure criteria generated by the functions you write.  That is to say if you generate a negative number and it should fail for that, not only does your generator have to handle that but your validator does as well.   What to do?

Perhaps reflections could help solve these problems?  The ReflectiveNaiveApproach instead uses the typing system to determine what to generate for each field in a given class.  An integer would generate a random integer and a string would generate a random string.  We know the field name and class type so we could add if statements for each field/type but that puts in the same maintenance of new properties we had with the hard coded approaches.  If we didn't do that we could still handle new properties assuming we knew how to set the type, but it might not fit the rules of the business logic and we have no way to know if it should work or not.  For fuzz testing this is alright, but not functional testing.  Is there any solutions?  Maybe.

The final answer I currently have is the ReflectiveSmarterApproach.  In effect when you need to generate lots of different data for lots of different fields, you need to have custom generators per class of fields.  What is needed is an annotation for each field telling it what needs to be generated.  An example of that can be found in the Address class.  Here is a partial example of this:

public class Address {
 @FieldData(dataGenerators = AverageSizedStringGenerator.class)
 private String name;
 @FieldData(dataGenerators = AddressGenerator.class)
 private String address1;
 //...
}

Now let us look at an example generator:

public class AddressGenerator extends GenericGenerator {
 @Override
 public List<dynamicdata> generateFields() {
  List<dynamicdata> fields = new ArrayList<dynamicdata>();
  fields.add(new DynamicData(RandomString.randomAddress1(), "Address", DynamicDataMetaData.PositiveTest));
  fields.add(new DynamicData("", "Empty",
   new DynamicDataMetaData[] {DynamicDataMetaData.NegativeTest, DynamicDataMetaData.EmptyValue}).
   setErrorClass(InvalidDataError.class));

  return fields;
 }
}

This generator generates a random address as well as an empty address.  It is clear one of these addresses is valid while the empty address appears to be a negative test.

Through the power of reflections you can do something like this:

List<DynamicDataMetaData> exclude = new ArrayList<DynamicDataMetaData>();
exclude.add(DynamicDataMetaData.NegativeTest);
ReflectiveData<Address> shippingAddress = new CreateInstanceOfData<Address>().setObject(new Address(), exclude);

The exclude piece is where you might filter out generating certain values.  Say you want to only do positive (as in expected to be successful) tests, you might filter out the negative tests (those that expect not to complete the task and possibly cause an error).  The third line generates you an object with all the properties that have the attached annotation and values.  Now this does not handle new fields automatically but it could certainly be designed to error out if it found any un-annotated fields (it is not at present designed to do this) and if you embed the code in your production code, it would be more obvious to the developer they need to add a generator. 

Now how do we pick which value to test when we could test with the empty address or a real address?  At present it picks it randomly because according to James Bach, random only takes roughly 2x to get equal coverage to pair wise testing.  Since we know all about the reason for generating a particular value (what error it would cause, etc.) we can at run time say how the validation should occur.  The example validation is probably a bit complex, but I was running out of time and got a bit slap-dash on that part.  One issue with this method is it is hard to know what your coverage is.  You can serialize the objects for later examination and even create statistical models around what you generated if needed.

Summary

Obviously this is a somewhat heavy framework for generating say 20 test data values.  But when you have a much larger search space that approaches infinite this is a really valuable tool.  I have generated as many as 50 properties/fields about 400,000 times in a 24 hour period.  That is to say, generating roughly 400,000 tests.  I have found bugs that even with our generator would only be seen 1 : 40,000 times and would probably have never been found in manual testing (but would likely be seen in production).  The version I use at work has more than a years worth of development and research, supporting a lot more complexity than exists in this example, but I also don't think it could be easily be adapted as it was built around our particular problems.

This simple version can be made to support other environments with relatively little code modification.  It took much of the research and ideas I had and implemented in a simpler fashion which is more flexible.  You should easily be able to hook up your own class, create annotations and generates and have tests being generated within a day (once you understand how).  On the other hand it might take a little longer to figure out how to do the validation as that can be tricky.

One problem I have with what I have generated is there is no word or phrase to describe it.  In some sense it is designed to create exploratory data.  In another sense it is a little like model driven testing in that it generates data, has an understanding of what state it should go to and a method to validate it went to the correct state.  However it doesn't traverse multiple states and isn't designed like a traditional MDT system.   Data Driven Testing describes a method for testing using static data from a source like a csv or database.  While similar, this creates dynamic tests that no tester may have imagined.  Like combinatorics, this creates combinations of values, but unlike pairwise testing, the goal isn't just generating the combinations (which can be impossible/unpractical to enumerate) but to generate almost innumerable values and pick a few to test with, while enforcing organization of your test data.  This method also encourages usage of ideas like random values while combinatorics is designed to have a more static set of values.  Yes you can make combinatorial ideas works with non-static sets, but it requires more abstraction (E.G. Create a combination of Alpha, Alpha-numeric, ... and this set of Payment method, now use the string type to choose what generate you use) and complexity.  Finally combinatoric methods can have difficulties when you have too many variables, depending on implementation.  This is a strange hybrid of multiple different techniques.  I suppose that means it is up to me to try to name it. Let's call it:  JCD's awesome code.  Reflective Test Data Model Generation

I would say don't expect any major changes/additions to the design unless I start hearing people using it and needing support.  That being said I love feedback, both positive and negative.

While researching for this article I came across this which is cool, but I found no good place to cite it.  So here is a random freebie: http://en.wikipedia.org/wiki/Curse_of_dimensionality

What is the Highest Level of Skill in Automation?

Thanks to Robert Sabourin for generating this topic.  Rob asked me roughly, 'What in your opinion is the highest level of skill in automation?'  He asked this to me in the airport after WHOSE had ended, while we waited for our planes.  It gave me pause in considering the skills I have learned and help generate this post.

Let me make clear a few possible issues and assumptions regarding what the highest level of skill is in automation.  First of all, I think that there is an assumption of pure hierarchy, which may not exist.  That is to say, there might not be a 'top' skill at all or the top skill might vary by context.  So I really am mostly speaking from a personal level and with my own personal set of automation problems I have faced.  When I answered Rob's question in person, I neglected to add that stipulation.  The other possible concern is that the answer I give is overloaded, and so I will have to work on describing the details after I give the short answer.  Without making you wait, here is my rough answer: Reflections.

What are reflections?

In speaking of reflections, you might assume I am speaking of the technology, and for good reason.  I have spoken on them many times in this blog.  However, that is just a technical trick, albeit a useful one. I am not talking about that trick, even if the comp-science term 'reflections' is part of the answer.  In speaking of reflections, I mean something much broader.

There is the famous "thinker" sitting on his rock just pondering is much closer to what I had in mind.  But you might say, "Wait, isn't that human thinking?  Isn't that critical thinking or introspection?"  Yes, yes it is.  What I mean by reflections is the art form of making a computer think.  While a computer's intelligence is not exactly human intelligence, the closer we approach that vast gulf, the closer we are to generating better automation.

Most people might start to argue that requires someone with in depth knowledge or artificial intelligence or at least a degree in computer science or someone with a development oriented background.  Perhaps that is the logical conclusion we will ultimately see in the automation field, but I don't think that either an in depth knowledge of development or AI is required for now.  I know that you need to go to that level to start understanding this concept.

Instead, I think you need to start thinking of the automation in the way you think about writing tests.  In some ways this relates to test design.  Why can't the automation ask what am I missing? Why can't my automation tell me what the most likely reason a failure occurred*?  Why can't the automation work around failures*?  Or at the very least, ignore some failures so it isn't blocked by the first issue it runs into*?

* I've done some work around these, so don't say they are impossible.

Now that I have walked around the definition, let me define the reflections in context of this article.

Reflections:  Developing new ideas based upon what is already know.

An example

A good example for the need for reflections is the brilliant talk given by Vishal Chowdhary, in which he notes that in translations (and searches, etc), you can't know what the correct answer is.  You have no Oracle to determine if the results are correct.  Many words could be chosen for a translation and it is hard to predict which ones are the 'best'.  Since computer language translations are adaptive, you can't just write "Assert.Equals(translation, expectedWord)" with hardcoded values.  Since these values are dynamic, the best you can do is to use a "degree of closeness".  You see, they couldn't predict how the translation service would work because it has dynamic data and the world changes quickly, including new words, proper titles, et cetera.

So how do you test with this?  Well you can look at the rate of change between translations.  You can translate a sentence, translate it back and record how close it was to the original sentence.  Now track how close it is over time, with different code and data changes.  You could take translation string lengths and see how they vary over time and note when large deviations occur. There are lots of methods to validate a translation, but most of them require the code to reflect on past results, known sentences and the likes.  The automation 'thinks' about its past, and on that basis judges the current results.

Not to say some automation shouldn't be reflective.  For example you could hard code a sentence with "Bill Clinton" in it and check to make sure that it didn't in fact translate his name.  You could translate a number and check to see it didn't change the value.  You might translate a web page and check something not related to the translation such as layout.

Not just the code

In reading my blog you might assume that because I specialize in automation I think reflections is a code-oriented activity.  I do think that, but I think it applies more broadly.  When I write a test, I should be reflecting on that activity.  That is to say I should be thinking "Is that really the best design?", "Should I be copying and pasting?", "Should I really be automating this?", etc.  In always having part of my brain reflecting on the code, I too am write better code.  Hopefully between my writing better code and my code trying to do better testing using reflections, we do better testing overall.  This also applies to testing in general, with considerations around things like "That doesn't look like the rest of the UI." or "I don't recall that button there in the last build."

I have only scratched the surface of this topic and made it more specifically apply to automation/testing, but I think this applies to life too.  For a more broad look at this topic I would highly recommend Steve Yegge's blog post Gödel-Escher-Blog.  It will make you smarter.  Then next time you go do some automation, reflect upon these ideas. :)  And if you are feeling really adventurous, please put a comment about your reflections on this article here.

Why can't anyone talk about frameworks?

In writing for WHOSE, I was dismayed at the total lack of valuable information regarding automation frameworks and developing them.  I could find some work on the frameworks with names (data driven, model driven and keyword driven), but almost nothing on how to design a framework.  I get that few people can claim to have written 5-10 frameworks like I have, but why is it we are stuck with only these 3 types of frameworks?

Let me define my terms a little (I feel like a word of the week might show up sometime soon for this).  An architecture is a concept, the boxes you write on a board that are connected by lines, the UML diagram or the concepts locked in someone's head.  Architecture never exists outside of the stuff of designs and isn't tied to anything, like a particular tool.  Frameworks on the other hand have real stuff behind them.  They have code, they do things.  They still aren't the tests, but they are the pieces that assist the test and are called by the test.  A test results datastore is framework, a file reading utility is framework, but the test along with its steps is not part of the framework.

Now let me talk about a few framework ideas I have had for the past 10 years.  Some of them are old and some are relatively recent.  I am going to pull from some of my presentations of old, but the ideas have at least been useful for one framework of mine, if not more.

Magic-Words

I'm sure I'm not the first one to come to this realization, but I have found no records of other automation engineers speaking of this before me.  I have heard the term DSL (Domain Specific Language) which I think is generally too tied to Keyword-driven testing, but a close and reasonable label.  The concept is to use the compiler and auto complete to assist in your writing of the framework.  Some people like the keyword driven frameworks, but in my past experience, they don't give compile time checking nor do they help you via auto complete.  So I write code using a few magic words.  Example: Test.Steps.*, UI.Page.*, DBTest.Data, etc.  These few words are all organizational and allow for a new user to 'discover' the functionality of the automation.  It also forces your automation to separate out the testing from the framework.  A simple example of that can be given:

@Test()
public void aTestOfGoogleSearch() {
 Test.Browser.OpenBrowser("www.google.com");
 Test.Steps.GoogleHome.Search("test");
 Test.Steps.GoogleSearch.VerifySearch("test");
}

//Example of how Test might work in C#, in Java it would have to be a method.
public class TestBase { //All tests inherit this
  private TestFramework test = new TestFramework();
  public TestFramework Test { get { return test; } }
}

Clearly the steps are somewhere else while the test is local to what you can see.  The "Test.*" provides access to all the functionality and is the key to discoverability.

Reflection-Oriented Data Generation

I have spoken of reflections a lot, and I think reflections are a wonderful tool for solving data-generation style problems.  Using annotations/attributes to tell each piece of data how to generate, what sorts of expectations there are (success, failure with exception x, etc.), filter the values you allow to generate and then picking a value and testing with it is great.  I have a talk later this year where I will go in depth on the subject and I hope to have a solid code example to show.  I will certainly post that up when I have it, but for now I will hold off on that.

...

Okay, fine, I'll give you a little preview of what it would look like (using Java):

public class Address {

 @FieldData(classes=NameGenerator.class)
 private String Name;
 @FieldData(classes=StateGenerator.class)
 private String State;
 //...

}
public class NameGenerator {

  public List<Data> Generate() {
   List<Data> d = new ArrayList<Data>();
   d.add(new Data("Joe", TestDetails.Positive);
   d.add(new Data(RandomString.Unicode(10),  {TestDetails.Unicode, TestDetails.Negative));//Assume we don't support Unicode, shame on us.
   //TODO More test data to be added
   return d;
  }

}

Details

Why is it that we as engineers who love the details fail to talk about them?  I get that we have time limits and I don't want to write a book for every blog post, but rarely do I see anyone outside of James McCaffrey and sometimes Doug Hoffman talk on the details.  Even if you don't have a framework, or a huge set of code, why can't you talk about your minor innovations?  I come up with new and awesome ideas once in a while, but I come up with lots of little innovations all the time.

Let me give one example and maybe that will get your brain thinking.  Maybe you'll write a little blog on the idea and even link to it in the comments.  I once helped write a framework piece with my awesome co-author, Jeremy Reeder, to figure out the most likely reason a test would fail.  How?

Well we took all the attributes we knew, mostly via reflections of the test and put them into a big bag.  We knew all the words used in the test name, all the parameters passed in, the failures in the test, etc.  We would look at all the failing tests and see which ones had similar attributes.  Then we looked at the passing tests and looked to see which pieces of evidence could 'disprove' the likeliness of a cause.

For example, say 10 tests failed.  All 10 involving a Brazilian page. 7 of those touched checkout and 5 of those ordered an item.  We would assume that the Brazilian language is the flaw if all tests failed, as that might be the most common issue.  However, if we had passing tests involving Brazilian, then that seems less likely, so we would see if we could at least establish if all checkout failures had no passing tests involving checkout.  If none had, we would say there was a good chance that checkout was broken and notify manual testers to investigate that part of the system first.  It worked really well and solved a lot of bugs quickly.

I do admit I am skipping some of the details in this example, like we did consider variables in concert, like Brazilian tests that involved checkout might be considered together rather than just as separate variables, but I hope this is enough that if you wanted to you could build your own solution.

Now your turn.  Talk about your framework triumphs.  Blog about them and if you want to put a link in the comments.

Word of the Week: Oracle

Oracle Test Definitions

Thanks to:
Isaac Howard and Wayne J. Earl who had a great deal to do with the editing and formulation of this article.

Like my previous word of the week on Heuristics and Algorithms, this is a complicated one. According to wiki,

An oracle is a mechanism used by software testers and software engineers for determining whether a test has passed or failed.

According to the wiki citation, this comes from BBST, which has some thoughts about what an Oracle is or isn't.  Specifically it talks a lot about Oracle Heuristics, an interesting combination that Bach roughly states as a way to get the right answer some of the time.  I don't feel I have a problem with that, but then we go back into the BBST class and things get confusing.  On Slide 92 of the 2010 BBST course, it says,

How can we know whether a program has passed or failed a test?  Oracles are heuristics 

Slide 94 says:

An oracle is a reference program. If you give the same inputs to the software under test and the oracle, you can tell whether the software under test passed by comparing its results to the oracle's. 

Which later goes on to say that the definition is wrong.  The slides does so because of the claim that Oracles are heuristics.

Classic Problems With The Definitions

But how can this be so if Oracles know all?  They are the truth tellers.  Well the problem in software is that Oracles are not absolute like in the stories.  They give you an answer, but the answer might be wrong.

For example, you might test Excel and compare it to a calculator.  You take the calculator and enter 2.1 * 1 getting back the value 2.  Now perhaps the calculator is setup to provide integers, but when you compare it to Excel's output, you find that Excel gives back 2.1.  This appears to be a failure in Excel, but in reality it is a configuration issue.  The heuristic is in assuming your Oracle is right.  This might of course be a false assumption, or it might be only right in some circumstances.  Interestingly, one of the creators of BBST, Cem Kaner, has revised the definition of Oracle slightly in a posting about Oracles and automation,

...a software testing oracle is a tool that helps you decide whether the program passed your test.

Partial Oracle

While I don't want to get too far off track, I do want to note that Partial Oracles do exist, and from what I can tell, they are Oracles that tell you if the answer is even reasonable.  For example, for two positive integers, you can say that when you add them together, you will get a larger number than either of the separate digits.  1+1=2.  Thus 1<2.  3+4=7.  Thus 4<7.  In both cases the larger number is always smaller than the sum of the two numbers, for ANY two numbers.

New Questions

Let me chart out the idea of an Oracle:

1. Test: Tester runs test.  Example: Login with valid user.
1. Result: Login takes 5 seconds and goes to a internal page.
2. Request for Data: Make a request out to the Oracle; Was 5 seconds too long?
1. Process: Oracle considers the answer.
2. Data: Oracle generates the answer: Yes, 5 seconds is too long.
3. Compare: Verify if Test's Result are acceptable.
4. Output: Test's Results are not acceptable.
5. React: Tester reacts to the result.  Maybe they fail the test.  Maybe they...

Now lets get picky.  Is the monitor (the display) which is beaming information to you an Oracle?  It shows the results you requested and is a tool.  While I noted the parts that are the Oracle, who is this Oracle?  If the Oracle is your mind, then what makes this different from testing?

My colleague Isaac noted that the requirements of a test in most definitions includes making observations and comparing them to expectations.  For example, Elisabeth Hendrickson said,

Testing is a process of gathering information by making observations and comparing them to expectations.

Does this make an Oracle simply part of a test?  Even to be able to come up with the question seems to indicate you might suspect an answer.  Is this too long?  Well, in asking that, one assumes you have a built in answer in your head.  Perhaps you are wrong, but that is part of what an Oracle can be.

Alternatively, maybe an Oracle is an external source, thus it has to be outside of the "Tester".  If that is the case, then can the Oracle be the System Under Test?  Imagine testing using two browsers at the same time doing the above test and the login time has a large difference between browsers.  Is the Oracle the browser or the SUT?

Lets take a different approach.  Lets say you take screenshots after 4 seconds from logging in using an automated process.  You compare the screenshot to the previous version of the result, pixel by pixel.  If the pixels compare differently, the automation reports back a failure.  Where is the Oracle?  The request for data was to get a previous edition of the SUT in image form.  No processing occurred, so the Oracle can't be the processing, but perhaps the image itself is the Oracle.  Or is the Oracle the past edition of the site and the data?  Continuing into this, the automation pulls out each pixel (is that the Oracle?) then compares them.  But wait a minute... someone wrote the automation.  That someone thought to ask the question about comparing the images.  Are they the Oracle?  Since the author is the Tester (checker, whatever) the first time, capturing the first set of images, they saw what was captured and thus became a future Oracle.

Even if the Oracle is always an external source, is it always a tool?  Is a spec a tool?  Is another person (say a manager) a tool?  No, not that sort of tool.  Is a list of valid zip codes a tool or just data?

In case you are wondering, many of the examples given are real with only slight changes to simplify.

How Others Understand Oracles - BBST

Perhaps you feel this detailed "What is it?" questioning is pedantic or irrelevant.  Perhaps we all know what an Oracle is and my attempt to define it is just making a mess.  In order to address that question, I am going to do something a little different from what I have done in the past.  I'm going to open up my BBST experience a little, as I answered the question Kaner wrote about and then talk a little about it. To be clear, this answer has the majority of the content pulled out, as it could be used as an exam question in the future:

Imagine writing a program that allows you to feed commands and data to Microsoft Excel and to Open Office Calc. You have this program complete, and it’s working. You have been asked to test a new version of Calc and told to automate all of your testing. What oracles would you use and what types of information would you expect to get from each?

1. Excel (Windows) – Verify that the most common version of Excel works like Calc does. Are the calculations the same? Do the formulas work the same? If you save a Calc file can Excel open it? What about visa-versa?

a. I’m looking to see if they have similar calculations and similar functionality.

<snippet>

7. Stop watch – Does Calc perform the tasks at the same rough rate as Excel, Google docs? Is it a lot faster or a lot slower?

a. I’m looking to see if we are consistent with current user expectations of speed.

The responses I got were interesting in my peer reviews.  Please note I removed all names and rewrote the wording per the rules in BBST.  One person noted that item 7 would be categorized under performance and that item 1 was looking at consistency with comparable products.  Multiple reviewers felt I was looking at consistency, a heuristic Bach created.  What I find odd about that, is the need to label the Oracle when the Oracle (in my view at the time) was the tool, not the heuristic, therefore citing the heuristic of comparable products was not part of the question.  I got a complaint that I was not testing Excel or Google but Calc, yet the call of the question is about how I would use those Oracles.  One fair issue was, I should have noted I could have compared the old version to the new version using a stop watch, which I had missed.  However, I had cited Old Calc in my full document, so I think that was a relatively minor issue.

Since Oracles are tools, how can I not be implicitly testing Excel?  I kept hearing people say I should name my Oracles, yet to me I was naming them very clearly.  I got into several debates about if something like Self Verifying Data is in fact an Oracle (even though the article clearly has its own opinion on that)!  It seemed like everyone wanted to label the heuristic the Oracle, probably because of the "Heuristic Oracle" label in BBST.  While I did feel BBST failed to make clear what an Oracle is, it did make me think about Oracles a lot more.

Wrapping Up

I'm sorry if that felt a little ranty, but by talking about this subject, I want you to also think about what you see as an Oracle.  Oddly, Kaner himself cite's Doug Hoffman with items which I did not consider an Oracle (such as Self Verifying Data) when I started writing this article.  I think Kaner's own work defends his view point, as he doesn't appear to use his own rule (his definition) to the letter but rather the similarity of the items, a method humans appear to use frequently.

Truth be told, I'm not so sure that Oracle should be a word we in the industry should be using at all.  Isaac does not seem to believe in Oracles anymore, and appears to feel the definition is broken as it really cannot be separated from test.  To me, I see that many people seem to use it and perhaps it can have value if we shrink down the role of the word.  So let me wrap this up with my attempt to patch the definition into something that is usable.

Oracle: One or more imperfect external sources (often tools) that provide data or are data to assist in determining the success or failure of a test.

For a not exactly correct but useful working definition, an Oracle is the value that is the expected result.

What do you think a Oracle is?  Are we missing critical details or other expert views?  Feel free to write us a note in the comments!