Happiness: On the Job and in Life

In this post, Matt Heusser does a great job describing some of the problems around hiring older workers and some of the reasons older workers become less valued.  Obviously, this is a concern for all technology workers in the field, regardless if you are in test, operations or development.  I have not had much experience with it, but I do feel some of the call that those who are in the 40s and 50s start to feel.  I don't want to invest hours upon hours doing development outside of work while working 40 hours a week.  This blog, which I write for often, requires time which I could use to get money from somewhere else.  I don't get paid to do this.  Sometimes I clarify my own thoughts, or use this to prompt me to learn, but more often than not I am actually trying to teach others the hopefully valuable information I have.

However, there is a underlying subject I think is more important than just ageism in the work place.  I want to talk about happiness.

First, a few words, I am not a doctor of any sort, I am not giving advice for depression and if you are experiencing depression you should see a doctor.  One other thing, I'm going to ask you to resist clicking all the exciting links until after you've finished reading this article, as there is a metric ton of data on the subject, but interrupting the narrative will likely distract you.  Now on with the show...

Were We Happy In The Past?

In the past, almost everyone did farming.  There were a few other jobs, like tailor, doctor and merchant, but most people were farmers.  These jobs were fairly well known and a town depended on one (hopefully) competent doctor, one tailor, etc.  A large town might have a few of each, but only in the cities were there really any large quantities of any specialties.  These specialties were all relied upon and considered vital.  While there was anger around the usage of power and prestige these ranks sometimes bestowed, in general they were good jobs.  Life could be brutish and short.  However, the idea of manifest destiny  and excitement of the renaissance show that life in the past could be seen as meaningful.  Is being a farmer happy making?  Or being a doctor or ...?  Sadly, there is little documentation I could find about happiness in the distant past.  However, I did find lots on various problems we see an experience today.  It seems that the general fighting we have today was going on in the past.

It is strange that there should be so little reading in the world, and so much writing. People in general do not willingly read, if they can have any thing else to amuse them. -  Richard Burke to Samuel Johnson, 1783.

The number of technical students has declined constantly... IF THE SAME TEACHER CAN INSTRUCT 500,000 OR MORE [students] SIMULTANEOUSLY? - Radio Electronics, 1956, http://www.americanradiohistory.com/Archive-Radio-Electronics/50s/1956/Radio-Electronics-1956-05.pdf

So while I have been unable to find anything quantitatively correlated to happiness in the distant past (and only a few data point in the recent past), it does seem that the same challenges facing us have been unchanged in the past few hundred years.  We are constantly looking for productivity changes while desiring to keep our hard gotten gains.  Furthermore, while I have yet to read this tome, Steven Pinker suggests that we were actually more violent in the past.  So if violence decreases happiness, which I have found lots of evidence for, it would appear there is little basis for the assumption we were happier in the past.  In fact, it appears arguable that collectively we might be over all happier now then in the past!

Do Jobs Affect Happiness?

The next question is, do we know if farmers of today are happier compared to the general population?
In a word, no.  Happiness and job are nearly unrelated, however there are a few key factors that matter.  One is that you have enough money to survive and not worry about where your next meal will come from.  Another is having meaningful work and yet another is having autonomy.  There are also some negative correlations with jobs.  For example, not getting enough sleep or too much mental stress (some physical stress is good for you).  Lacking creativity can be an indirect form of mental stress as well.  Often this comes in a form known as boredom.  Basically, I think I can sum it all up in a single sentence:

You want a job that pays a living wage with difficult but surmountable and clear goals in which you have enough control to get it done and in which you will get back useful unambiguous feedback and time to recharge.

Great, we're done, right?  Well, I want to explore some more topics around this and how I think the reason our job does not seem highly connected to happiness.  The first is that we don't know how to practice being happy, so why would ones choice of job have an affect?  That is to say, if happiness is fairly mysterious then choice of job will appear to be a random or not related to happiness.  So how do you practice happiness?  For that matter, how can we discuss happiness when there are so many varying view on how to create it.  Is it something that comes without you noticing it or is it something you can actively seek?  If significant increases in wealth doesn't change happiness but for a small and short period of time, what does?

Rather than going on with so many links, I'm going to stop citing so many sources.  The purpose of this essay is not to do research for you but rather to explore the subject and not just trends I have seen in the research.  Furthermore, I wish to explore how to have a happy life, not just being happy on the job.

How We "Need" Society for Happiness

Helping others can, in and of itself hurt those we hope to help, without regards to the permission they give. You might have heard the phrase "What doesn't kill you makes you stronger." The correlation to that would be, "What makes life easier makes you weaker."  Obviously things like chronic pain and long term mental stress eventually do damage to a human being, but in general these two ideas have some truth to them.  Consider the case of the blind, a blind person might appreciate being given an 'extra hand' but at the same time, that might also make them less able to do things on their own when that helping hand isn't around.  Married couples are known to have slightly lower IQ than the unmarried because of dependencies on each other.

In a sense, as John Donne said, no man is an island. That does not equal a learned dependency in my book. Babies have real dependencies. We are dependent on doctors during surgery. We depend on lots of things. However, learned dependencies are not things we are incapable of doing. Maybe we don't want to do them or maybe it would cost too much of our time, but a learned dependency is when you quit trying because you're not expected to try. I suppose you could say I quit trying to change my oil because Jiffy Lube says I'm not expected to try. However, I would argue that my dependency on oil changes is more a matter of time, energy and my personal interest. That being said, it is a fine line.

In regards to happiness, is happiness truly just an internal thing or does it have external dependencies?  Can you live and enjoy pure silence, or do we really need music?  It is an interesting question, but even if you depend on external things, they may not really require other humans.  Enjoying waterfalls or existing books for example require no (new) human work.

Learned dependency, be it having someone else get you around, using a calculator to do math or depending on others for happiness is a problem because now you need something else in order to do the things that you want. I don't know how to give an algorithmic way of judging when it is enabling and undermining independence and when it is a genuine hand up.  Perhaps it is even in part personal taste, as the hermit choose no or few dependencies while the city dweller depends on many people for city services.  The hermit might even see those city services as enabling the city dweller, as the city dweller needs the bus in order to go see the opera they desire to see.  That is always a struggle, and has been one for ages.

If our dependencies on others needs to be managed, it seems only reasonable to ask about the other way around.  How do I prevent asserting myself into other people's need for genuine challenge? The Bible famously asks the question:

Am I my brother's keeper? - Genesis 4:9

In that case it was a way of ignoring responsibility for the murder of his brother, but the question of responsibility and social value for watching out for others has been around for centuries.  I don't pretend I will answer the question today, but I think it is a balancing act.  Rarely do we know what we really want. Our simulators for happiness are poor at best. Dan Gilbert talked about that in his TED talk.  So if we have a hard time knowing what will make us happy, pretending we know what other people want is silly.  This can become a wicked problem when what you want to do is help others but worry about foisting your culture, will or opinions on others without giving them room to grow.  Teachers suffer this question all the time.

 Can Happiness be Treated as a Duty?

The Stoic's claimed virtue is sufficient for happiness and that virtue is created by a will that is aligned with nature.  Kant's Categorical Imperative, in a very rough and simplified form, linked duty to what you imagine a reasonable person would do.  So unless you imagine a reasonable person would choose to be unhappy most of the time, it would appear happiness might in fact be a duty.  There are other philosophical theories like Hedonism which say that you should look to maximize long term happiness.  That is to say, have the day to day discipline to make your life better in the long term.  All of these various ideas about optimizing life with consideration for happiness seem to have something of a pattern to them.  They all push us to grow.  Be it via virtue, by considering how a reasonable person would live or look at the long term and attempting to minimize pain.

So how do you generate growth?  Well one aspect is to not be in fear.  Fear often is the opposite of growth.  You need to feel safe enough that you can work.  Even when in fear, such as when in a castle surrounded by an army, you have enough pause to feel safe enough to gather your wits and think.  Safety by itself isn't enough.  While safety is important, you have to be safe to try experiments, pushing yourself beyond what you have done before or in new ways allows for the growth, which leads towards happiness.  The feeling of growth is in my view, at least one of the causes of happiness.  However, if one only attempts to grow in one area, it can become an addiction.  Growth also has to relate to goals, and in particular unending goals.  Goals you can meet can later feel unsatisfying.  For example,  when you think of buying something, you feel more satisfaction then after you have purchased the item.  Since having meetable goals is less likely to create happiness, how do you create a unmeetable goal?  You could say, "I will weigh 150 lbs in 3 months," but this is an unsatisfying goal.  If you make it, maintaining it is much more difficult.  Instead, I will go to the gym 3 days a week remains satisfying because there is no 'end point'.

In addition to safety, you also need time to develop happiness.  You need to have time to work on your goals, or else you stagnate into surviving life.  Time can mean exact and dedicated time, such as a day of the week or hour in the day, or it can mean just having open variable spots.  Time is perhaps one of the trickier things to give detail on because different people like doing things in different ways.  Some people like things well planned out while others like to be more sporadic.  I have not seen evidence demonstrating that one is superior to the other, but personally I rather have some vague planning rather than chaos or an exact fixed schedule.

While the goals don't change often, they should be allowed some variety.  If you always go to the gym on Monday, Wednesday and Friday, you may end up burnt out.  Instead, you probably should have some time allocated to 'breaking' the goal.  Going to the library once a month instead of your Wednesday workout.  You should feel the pull to get back into the routine which helps make the routine more satisfying.  If you don't feel excited to go back into the routine, it demonstrates something is wrong with your goal.

Finally, you need to be able to see that these goals are working.  Without being able to check up on your goals, without being able to study the results and see if they match expectations you get little satisfaction from 'achieving' them.  While weighing 150 lbs might be a very watchable (E.G. an easy metric), you don't need that to see if going to the gym is helping you or not.  Since using easy metrics tend to make goals that end, those are likely not the first type of metric choices to make.  Saving enough money to buy a house is an admirable goal, but learning the habit of saving should be the real goal.  So using 'enough money to buy a house' as the measurement means you're probably watching the wrong thing.

Now if you have been reading this far, you'll probably notice something interesting.  I'm going to break a general rule I have of not copying and pasting and repeat myself from above.  Basically, I think I can sum it all up in a single sentence:

You want a job that pays a living wage with difficult but surmountable and clear goals in which you have enough control to get it done and in which you will get back useful unambiguous feedback and time to recharge.

If you noticed what I wrote in the job section and in the life section, you can see the life section is very close to a repeat.  In fact I would use the job section as a sort of check sum from what was discussed in the broader life category.

The idea that you can treat happiness as a duty is not exactly correct.  Instead, just like with getting a job that creates happiness, you need to do similar things in life.  It maybe formulaic sounding, but it is more complicated in one's actual life than any author can generalize.  A duty implies it is required, but this is more of a method, backed by out current science and history.  Can other methods work?  Maybe.  Is what I have written 100% correct for your life.  Very likely not.  All situations vary.  But the general ideas seem to remain consistent throughout time.

Disagree with me?  Well, my goal is to engage my readers, so please leave a comment.  I'll be happy to reply.  Oh yes, and now feel free to go click all those links and enjoy all that awesome data.

Words of the Week: Critique and Criticism

Preamble / Ramble

This is a first for me and word of the week, as I typically consider a word without much personal context, but this week I am going to include a context. While at CAST this last week, I heard some valuable input regarding my blog. No matter how I get feedback, I try to take it and make it useful. I tend to be direct with my feedback, but for some, that can sound harsh, so this week I’m going to consider two different words. If you don’t want to hear the backstory, just skip down to the next section. However, for those that care, I do want to give some context and clarity in my own words. I have now heard from multiple sources that they felt my writing reflected a negative viewpoint towards the Software Testing World Cup 2014. I think that to be an inaccurate statement. I did have some issues with it, and I felt I was doing a deep dive analysis of both the pros and the cons of the contest, up to the point I had gotten. In assuming that I was writing a report about my experience with the contest, going into both the good and the bad. I also wanted to talk about what I could gain from the contest and what could be done better. In my second part I wanted to consider what the judges had to say, so I could learn where to improve as well as note what additional feedback would be nice to have. I understand that this is a contest, but I think the more important piece is what can be learned from it. To be fair, perhaps the organizers didn’t want what I had to say, as they actually asked for:

“At the moment we are very curious about your personal STWC experience. We would love to read blog posts or any other kind of written reports about your participation. It doesn’t matter if you write from your personal point of view or from the perspective of your whole team. We are interested in hearing how much fun you had during the competition, what you have learned from this experimental challenge, and what the difference is between testing while having fun together with friends and testing at work. We are also interested in how you communicated with your team members and what your team’s strategy was. Did you start testing right away or did your team follow a strategy? How did you create that strategy? What was your biggest challenge within your team, what kind of bugs were you able to detect and how did you like working with the Agile Manager tool?”

I took this to mean they were excited to receive any feedback regarding the design of their contest. They also wanted to hear what we did and why we did it. I heard that they were open for a critique of the process so that they could attempt to improve it. Furthermore, we were not told not to post scores, as we felt that it was important that those who participated could compare scores and learn where they could improve.  In fact I encouraged others to post scores, but no one actually did so. We also thought it might add value to future judging to see a relatively unemotional reaction to the scores given. We made the judge’s scores more anonymous as it felt important not to provide any identifiable information. However, it seems some saw what I thought was a critique others felt was criticism. This leads me to the two words, critique and criticism. Having re-read my content I can see that it could be read as criticism, and while I won’t take back my words, I will keep that in mind for the future. So on to the words of the week.

Back to our Regularly Scheduled Program

First allow me to define these two words in the way I think of them, without doing much research nor using Google to define the words. I think of criticism and critique as similar, but with a difference in tone and intent. To criticize someone is to give back negative statements, with either the intent to hurt/harm, with no positive points and to lack any possible ways to improve, be they implicit or explicit. Calling someone stupid is to be critical of someone’s intellectual factuality. It is likely intended to harm, has no positive points and does not hint at a way of improving. To critique is to provide feedback with the intent to improve a person or person’s work in some way. There is a mix of some positive points in the comments as well as negative comments. Gushing praise or all negative points is probably not a critique. Furthermore, at least some of the statements need to be actionable. Critique also feels very student/teacher-y. Criticism on the other hand seems like it refers to bullies and angry people shouting at each other. I want to capture this in a readable way for later usage:

Criticism	Critique	Gushing Praise
Negative comments Intended for harm/hurt No method for improvement provided.	Negative and positive comment mix Intended to improve the content, person(s) or future Provides methods for improvement.	Positive comments Maybe intended for manipulation (even if meant to be ‘improve another person’s day’)

Now let’s see how I did. There are a lot of different people’s views on these words, so let me try to capture just a few of them.

Various versions of Critique and Criticism

According to the presently [August 20th, 2014] most popularly rated stack exchange view the difference is zero. They point out how in 1960’s academics started to try to use critique as a form of analysis rather than meant to censure, but that didn't stick.  They cite several modern examples of mixed usage of the terms. In comparison, Paul Brians, the author of Common Errors in English Usage, says:

“Josh critiqued my backhand” means Josh evaluated your tennis technique but not necessarily that he found it lacking. “Josh criticized my backhand” means that he had a low opinion of it.

Clearly their is some difference of opinion on the meaning of the word. In Writing Alone, Writing Together; A Guide for Writers and Writing Groups by Judy Reeves, there is a interesting comparison of criticism and critique:

“[1]Criticism finds fault/Critique looks at structure

[2]Criticism looks for what's lacking/Critique finds what's working

[3]Criticism condemns what it doesn't understand/Critique asks for clarification

[4]Criticism is spoken with a cruel wit and sarcastic tongue/Critique's voice is kind, honest, and objective

[5]Criticism is negative/Critique is positive (even about what isn't working)

[6]Criticism is vague and general/Critique is concrete and specific

[7]Criticism has no sense of humor/Critique insists on laughter, too

[8]Criticism looks for flaws in the writer as well as the writing/Critique addresses only what is on the page”

(NOTE: I numbered the following quote for later convenience)

It is not completely clear some of the differences between criticism and critique, as I can find fault in the structure of a work, which leaves me in no mans land. Wait… wait…

I want to analyze my last sentence, using the rules listed above. In the second rule, I clearly have a criticism of the writing. The the third rule says if I had asked in the form of a question, if I had said “It appears not completely clear some of the differences…” and perhaps ended with a question mark it would have been a critique rather than a criticism. That appears to me to mean the syntax matters more than the semantic content in the author’s opinion?* Oh, this gets us to rule four. While I made my statement objective and honest, it might not be considered kind. I can’t tell if my statement could be seen as negative or positive. It certainly isn’t positive, so I’m going to read between the lines and call it criticism. I think my statement seems concrete and specific, even if no exact example was given. However, it isn’t full of humor, nor is most of my writing.** The final dimension is about the author. With no reference to the author, I think I am addressing only what is on the page. Ultimately it feels that this set of tools has limited value as it takes too much analysis and when you have 5 / 8 on the criticism side, is it criticism? To the definition's benefit, it does try to capture the concepts I stated early about providing some positive points and providing negative points without beating the subject up. It adds to my definition attempt that you should be addressing the content rather than external influences and adds the idea that the critical comments be concrete. However, it fails to address the possibility of simply gushing rather than capturing the areas of improvement.

Obviously my example is contrived, but look at the above paragraph as whole (yes, I’m getting meta) I would call it a criticism using the rules Judy Reeves developed, yet it wasn’t intended to be. So it feels wrong in some way or perhaps I’m simply wrong. In my mind, she is looking for more positive views rather then stressing the analysis. That is to say the message is less important than the presentation. In my opinion, I’m critiquing the content, because I follow my own rules as well as the new rules I have developed.


* Yes, that was intentionally sarcastic. I hope you can appreciate the joke.
** Ignoring the above *’d item. It wasn’t that funny anyway.*

To Sum Things Up

Some people don’t see a difference between criticism and critique. In more academic circles, it appears that a divide is seen, even if the divide is a little fuzz. Some see it as a difference of purpose, some see it as a difference of the message and some see it as a difference of the content.  How do I see it?  Well, let me capture the attributes I think matter:

Criticism	Critique
Negative comments Intended for harm/hurt No method for improvement provided May focus more on the authors over the content "Censure, attack, abuse and name call"	Negative and positive comment mix Intended to improve the content, person(s) or future Provides methods for improvement Address the content rather than the authors Concrete examples "Deep analysis of other people's work"

In reviewing my own work, I think it falls under the label critique, but I can see how others might read something into it.  Written critiques are hard to convey personality.  You can see that with my footnote joke above.  In going so meta, I was intending on injecting some levity into an otherwise highly intellectual process.  But others might see it as 'unprofessional' or they might read my words as snarky.  In getting my article reviewed, Isaac said he thought it was a little on the snarky side, but that in his view it was just right to make the point.  Outside of getting my content reviewed, I am not sure I can defend against hurting people with a wide audience and perhaps that is Judy's point.  By being kind, you avoid the question completely.  Do you have a critique or feedback for me?  Feel free to post in the comments.

As a sort of post script, I happened upon this site after I wrote this article.  As I didn't want to shoe horn it in, yet felt it was of some value, so I put it here.

Word of the Week: Oracle

Oracle Test Definitions

Thanks to:
Isaac Howard and Wayne J. Earl who had a great deal to do with the editing and formulation of this article.

Like my previous word of the week on Heuristics and Algorithms, this is a complicated one. According to wiki,

An oracle is a mechanism used by software testers and software engineers for determining whether a test has passed or failed.

According to the wiki citation, this comes from BBST, which has some thoughts about what an Oracle is or isn't.  Specifically it talks a lot about Oracle Heuristics, an interesting combination that Bach roughly states as a way to get the right answer some of the time.  I don't feel I have a problem with that, but then we go back into the BBST class and things get confusing.  On Slide 92 of the 2010 BBST course, it says,

How can we know whether a program has passed or failed a test?  Oracles are heuristics 

Slide 94 says:

An oracle is a reference program. If you give the same inputs to the software under test and the oracle, you can tell whether the software under test passed by comparing its results to the oracle's. 

Which later goes on to say that the definition is wrong.  The slides does so because of the claim that Oracles are heuristics.

Classic Problems With The Definitions

But how can this be so if Oracles know all?  They are the truth tellers.  Well the problem in software is that Oracles are not absolute like in the stories.  They give you an answer, but the answer might be wrong.

For example, you might test Excel and compare it to a calculator.  You take the calculator and enter 2.1 * 1 getting back the value 2.  Now perhaps the calculator is setup to provide integers, but when you compare it to Excel's output, you find that Excel gives back 2.1.  This appears to be a failure in Excel, but in reality it is a configuration issue.  The heuristic is in assuming your Oracle is right.  This might of course be a false assumption, or it might be only right in some circumstances.  Interestingly, one of the creators of BBST, Cem Kaner, has revised the definition of Oracle slightly in a posting about Oracles and automation,

...a software testing oracle is a tool that helps you decide whether the program passed your test.

Partial Oracle

While I don't want to get too far off track, I do want to note that Partial Oracles do exist, and from what I can tell, they are Oracles that tell you if the answer is even reasonable.  For example, for two positive integers, you can say that when you add them together, you will get a larger number than either of the separate digits.  1+1=2.  Thus 1<2.  3+4=7.  Thus 4<7.  In both cases the larger number is always smaller than the sum of the two numbers, for ANY two numbers.

New Questions

Let me chart out the idea of an Oracle:

1. Test: Tester runs test.  Example: Login with valid user.
1. Result: Login takes 5 seconds and goes to a internal page.
2. Request for Data: Make a request out to the Oracle; Was 5 seconds too long?
1. Process: Oracle considers the answer.
2. Data: Oracle generates the answer: Yes, 5 seconds is too long.
3. Compare: Verify if Test's Result are acceptable.
4. Output: Test's Results are not acceptable.
5. React: Tester reacts to the result.  Maybe they fail the test.  Maybe they...

Now lets get picky.  Is the monitor (the display) which is beaming information to you an Oracle?  It shows the results you requested and is a tool.  While I noted the parts that are the Oracle, who is this Oracle?  If the Oracle is your mind, then what makes this different from testing?

My colleague Isaac noted that the requirements of a test in most definitions includes making observations and comparing them to expectations.  For example, Elisabeth Hendrickson said,

Testing is a process of gathering information by making observations and comparing them to expectations.

Does this make an Oracle simply part of a test?  Even to be able to come up with the question seems to indicate you might suspect an answer.  Is this too long?  Well, in asking that, one assumes you have a built in answer in your head.  Perhaps you are wrong, but that is part of what an Oracle can be.

Alternatively, maybe an Oracle is an external source, thus it has to be outside of the "Tester".  If that is the case, then can the Oracle be the System Under Test?  Imagine testing using two browsers at the same time doing the above test and the login time has a large difference between browsers.  Is the Oracle the browser or the SUT?

Lets take a different approach.  Lets say you take screenshots after 4 seconds from logging in using an automated process.  You compare the screenshot to the previous version of the result, pixel by pixel.  If the pixels compare differently, the automation reports back a failure.  Where is the Oracle?  The request for data was to get a previous edition of the SUT in image form.  No processing occurred, so the Oracle can't be the processing, but perhaps the image itself is the Oracle.  Or is the Oracle the past edition of the site and the data?  Continuing into this, the automation pulls out each pixel (is that the Oracle?) then compares them.  But wait a minute... someone wrote the automation.  That someone thought to ask the question about comparing the images.  Are they the Oracle?  Since the author is the Tester (checker, whatever) the first time, capturing the first set of images, they saw what was captured and thus became a future Oracle.

Even if the Oracle is always an external source, is it always a tool?  Is a spec a tool?  Is another person (say a manager) a tool?  No, not that sort of tool.  Is a list of valid zip codes a tool or just data?

In case you are wondering, many of the examples given are real with only slight changes to simplify.

How Others Understand Oracles - BBST

Perhaps you feel this detailed "What is it?" questioning is pedantic or irrelevant.  Perhaps we all know what an Oracle is and my attempt to define it is just making a mess.  In order to address that question, I am going to do something a little different from what I have done in the past.  I'm going to open up my BBST experience a little, as I answered the question Kaner wrote about and then talk a little about it. To be clear, this answer has the majority of the content pulled out, as it could be used as an exam question in the future:

Imagine writing a program that allows you to feed commands and data to Microsoft Excel and to Open Office Calc. You have this program complete, and it’s working. You have been asked to test a new version of Calc and told to automate all of your testing. What oracles would you use and what types of information would you expect to get from each?

1. Excel (Windows) – Verify that the most common version of Excel works like Calc does. Are the calculations the same? Do the formulas work the same? If you save a Calc file can Excel open it? What about visa-versa?

a. I’m looking to see if they have similar calculations and similar functionality.

<snippet>

7. Stop watch – Does Calc perform the tasks at the same rough rate as Excel, Google docs? Is it a lot faster or a lot slower?

a. I’m looking to see if we are consistent with current user expectations of speed.

The responses I got were interesting in my peer reviews.  Please note I removed all names and rewrote the wording per the rules in BBST.  One person noted that item 7 would be categorized under performance and that item 1 was looking at consistency with comparable products.  Multiple reviewers felt I was looking at consistency, a heuristic Bach created.  What I find odd about that, is the need to label the Oracle when the Oracle (in my view at the time) was the tool, not the heuristic, therefore citing the heuristic of comparable products was not part of the question.  I got a complaint that I was not testing Excel or Google but Calc, yet the call of the question is about how I would use those Oracles.  One fair issue was, I should have noted I could have compared the old version to the new version using a stop watch, which I had missed.  However, I had cited Old Calc in my full document, so I think that was a relatively minor issue.

Since Oracles are tools, how can I not be implicitly testing Excel?  I kept hearing people say I should name my Oracles, yet to me I was naming them very clearly.  I got into several debates about if something like Self Verifying Data is in fact an Oracle (even though the article clearly has its own opinion on that)!  It seemed like everyone wanted to label the heuristic the Oracle, probably because of the "Heuristic Oracle" label in BBST.  While I did feel BBST failed to make clear what an Oracle is, it did make me think about Oracles a lot more.

Wrapping Up

I'm sorry if that felt a little ranty, but by talking about this subject, I want you to also think about what you see as an Oracle.  Oddly, Kaner himself cite's Doug Hoffman with items which I did not consider an Oracle (such as Self Verifying Data) when I started writing this article.  I think Kaner's own work defends his view point, as he doesn't appear to use his own rule (his definition) to the letter but rather the similarity of the items, a method humans appear to use frequently.

Truth be told, I'm not so sure that Oracle should be a word we in the industry should be using at all.  Isaac does not seem to believe in Oracles anymore, and appears to feel the definition is broken as it really cannot be separated from test.  To me, I see that many people seem to use it and perhaps it can have value if we shrink down the role of the word.  So let me wrap this up with my attempt to patch the definition into something that is usable.

Oracle: One or more imperfect external sources (often tools) that provide data or are data to assist in determining the success or failure of a test.

For a not exactly correct but useful working definition, an Oracle is the value that is the expected result.

What do you think a Oracle is?  Are we missing critical details or other expert views?  Feel free to write us a note in the comments!

Words of the Week: Heuristic [& Algorithm]

Isaac and I were debating the meaning of Heuristics the other day, trying to come to some common ground.  We ended up going into some interesting places, but it lead into a good question about what they are and how they are used.  Let me start with my off the cuff definition using no wiki links, then we'll hit into a more formal look.  A heuristic in my mind is a way of getting a not always right answer, but an answer one hopes is good enough.  In comparison, an algorithm will always provide a 'correct' answer.  This means that the output is consistent given the input and will always provide the best answer it can. This leads into the question of, can a computer give a non-algorithmic (heuristic) response?

Now when Isaac and I were talking through this, he noted that a computer always gives the same answer if you lock down all of the inputs.  Random.Int() uses a seed, and if you replay with that same seed it will do the same thing.  If you change the configuration or time of the computer and that has an affect, that is an input, and in theory if you lock that down too as an input.  If the number of CPU cycles for the process is an input, that too would be locked down.  Now given my informal definition, are all heuristics really algorithms, just with inputs that are hard to define?

Lets flip this on its head.  What about people?  Isaac asked me, "What would you call that plant?"  I said, "Green" to his chagrin.   He said, "No, the name of the plant is?"  A co-worker interjected, "George."  Obviously it wasn't Isaac's day, but the point Isaac was going for was, if I didn't know the name and then he told it to me, would I then be able to get a closer to accurate answer.  Even if Isaac was wrong, I had some knowledge base to draw on and could use that to give a heuristically better answer, but I could still be wrong, given that the name might also be George.  The problem is...  What if we locked down all of my life experiences, genes and repeat the experiment?  Obviously it can't be done, but am I really an algorithm that is just a walking input/output device of large complexity?  We are hitting into the realm of determinism and free will.  Is everything an algorithm?  I don't really want to get too far into the weeds, but I believe a point will emerge from this.

The time is now that we start looking at more formal and test oriented definitions to Heuristics.  In wiki it says,

In computer science, artificial intelligence, and mathematical optimization, a heuristic is a technique designed for solving a problem more quickly when classic methods are too slow, or for finding an approximate solution when classic methods fail to find any exact solution. This is achieved by trading optimality, completeness, accuracy, or precision for speed.

 Bach and Kaner say,

A heuristic is a fallible method of solving a problem or making a decision.

So now that we know that, I want us to be able to contrast it to Algorithm, the other word that incidentally is being considered.  Again, let us consider wiki:

In mathematics and computer science, an algorithm is a step-by-step procedure for calculations. Algorithms are used for calculation, data processing, and automated reasoning.  ... While there is no generally accepted formal definition of "algorithm," an informal definition could be "a set of rules that precisely defines a sequence of operations."

In both definitions, Heuristics acknowledge failure as a possibility.  On the other side, the Algorithms definition notes that their is no formal definition and the only 'failure' I noted that was somewhat on topic was the question if an Algorithm needed to stop.  If an Algorithm does not care if it gives back a correct value, just that it has a set of finite steps, then it too acknowledges failure is allowed.  In theory, some might note that it should end eventually, depending on if you think a program that has a halting state is an Algorithm but this is the only question about outcome.  I suppose you could say an Algorithm can also fail, as success is not a required condition, only halting.  In all the definitions, their is some method for finding a result.  The only difference appears that Heuristics specifically acknowledge limits and stopping points and Algorithms don't.

So what is the difference between Heuristic and Algorithm?  One of the popular StackOverflow answers says:

• An algorithm is typically deterministic and proven to yield an optimal result
• A heuristic has no proof of correctness, often involves random elements, and may not yield optimal results.

So in this very formal world, an Algorithm requires mathematical proof of correctness (within a given context, such as assuming our universe's constraints).  Heuristics on the other hand need no such formal proof.  In that case, most code is in fact Heuristical in nature and most of our testing is also Heuristical in nature.  This starts to lead into the question of sapient testing vs checking, but still, I don't want to get into that yet.  Well not much.  I do want to address one other quote from Bach,

There are two main issues: something that helps you solve a problem without being a guarantee. This immediately suggests the next issue: that heuristics must be applied sapiently (meaning with skill and care).

The idea that Heuristics require skill and care is an interesting one.  When I write an automated test or when I write a program, I use skill and care.  Am I using Heuristics in my development or is my Algorithm the Heuristic?  When I test, am I exploring a system using a Heuristic but when I write automation, the Heuristic of my exploration is lost after the writing of the test, and then it becomes something else, an Algorithm to formally prove if the two systems can interact the same way controlling for the majority of the inputs?  What happens when computers start getting vision and understand a little of the context?  Are they now sapient in a given context (meaning are they skilled and take care to manage the more important aspects compared to the less important aspects)?

I don't intend on going on in this questioning manner, but rather to hit you with a surprise left.  Sometimes, words are so squirrelly, that when one person attempts to pin them down, they end up creating a unintended chain of events.  They create just another version of the meaning of the word.  Next thing you know, no one really knows what the word means or what the difference is between two words is.  I have done way more research on this than most do, and yet I don't think there is a good answer.  I too will attempt to put my finger in the dike, but I don't expect to stop the flow of definitions:

• A Heuristic is an attempt to create a reasonable solution in a reasonable amount of time.  Heuristics are always Algorithmic in that they have a set of steps, even if those steps are not formal.
• A Algorithm is a series of steps that given the control of all inputs will consistent give back a result without necessarily considering other external factors, such as time or resources.  These steps have some formal rigor.

Word of the Week: Manumatic

Before I go into the depths of what I mean, I should first talk about what the wiki defines manumatic as.  According to the dictionary, Manumatic is a type of semi-automatic shifter used for vehicles.  This is not what I am talking about, even though it shares some of the same flavor.  What I am talking about is semi-automated testing (or is that semi manual checking?).  Some testers like the term tool-assisted testing and I can imagine a half dozen other terms like tool driven testing.  Whatever you want to call it, I tend to call it a manumatic process or manumatic test.

The idea is that you have a manual process that is cumbersome or difficult to do.  However, either some part of the test is hard to automate or the validation of the results requires human interpretation.  There are many different forms this can come in, and my attempt to define it may be missing some corner cases (feel free to QA me in the comments), but allow me to give some examples.

At a previous company I worked for I had to find a way to validate thousands of pages did not change in 'unexpected' ways, but unexpected was not exactly defined.  Unexpected included JavaScript errors, pictures that did not load, html poorly rendering and the likes.  QA had no way of knowing that anything had in fact changed, so we had to look at the entire set every time and these changes were done primarily in production to a set of pages even the person who did the change may not have known.  How do you test this?  Well, you could go through every page every day and hope you notice any subtle changes that occur.  You could use layout bugs detectors, such as the famous fighting layout bugs (which is awesome by the way), but that doesn't catch nearly all errors and certainly not subtle content changes.

We used a sort of custom screenshot comparison with the ability to shut off certain html elements in order to hide things like date/time displays.  We did use some custom layout bug detectors and did some small checking, but primarily the screenshots were our tool of choice.  Once the screenshots were done, we would manually look at the screenshots and determine which changes were acceptable and which were not.  This is a manumatic test, as the automation does do some testing, but a "pass" meant nothing changed (in so far as the screenshots were concerned), and finding a diff or change in the layout didn't always mean "fail".  We threw away the "test results", only keeping the screenshots.

In manually testing, often we need new logins.  It requires multiple sql calls and lots of data to create a new login, not to mention some verifications that other bits are created.  It is rather hard to do, but we wrote automation to do it.  So with a few edits, an automated 'test' was created that allows a user to fill in the few bits of data that usually matter and lets the automation safely create a user.  Since we have to maintain the automation already, this means every tester need not have the script on their own box and fight updates as the system changes.  This is a manumatic process.

Let me give one more example.  We had a page that interacted with the database based upon certain preset conditions.  In order to validate the preset conditions, we need to do lots of different queries, each of which was subtly connected to other tables.  Writing queries and context switching was a pain, so we wrote up a program to do the queries and print out easy to read HTML.  This is a manumatic process.

I honestly don't care what you call it; I just want to blur the lines between automated testing and manual testing, as I don't think they are as clear as some people make them out to be.

Word of the Week: Trilemma

Sometimes when reading, I run into interesting words.  In this case I found the word "trilemma", which like a dilemma, is about a difficult choice.  The only difference is that Tri literally means three options but a dilemma is a uncounted number of options.  I sometimes randomly look through wiki entries to see what additional data I might gain, and think critically about a subject (E.G. Keeping the saw sharp).   If you want to read this, you will need to have at least glanced at that wiki link, as I will reference it multiple times.

 In wiki they seem to cite multiple different styles of usage.  The computer-oriented example they give is the very well know "Good, Fast, Cheap, pick two" phrase in reference to computers.  This doesn't sound exactly like a trilemma to begin with, but rather a dilemma of some sort, where you can choose two out of a set of three.  Then, when you think more deeply about it, there are three bad groupings you can pick: "Fast-Cheap", "Fast-Good", "Good-Cheap."  This makes it a trilemma and now you can label that triangle.

Another style of this is the Apologetic trilemma, roughly simplified that Jesus is either a "Lunatic, Liar, or Lord."  I don't want to comment on the merits of the argument other than to note that it appears only like a trilemma because the list of options was artificially limited to three.  Where is Legend, Lithographic error or any of the other possible choices?

As the last "style" of trilemma I want to talk is the trilemma in law example given.  Basically it is 3 singular choices, each of which is direct and unique.  Either you can swear to something and lie, keep silent and go to jail or tell the truth and go to jail.  This is the easiest to identify and it doesn't appear there are any other options.  That means it really is a trilemma.

Ultimately, I just enjoy pulling out my thinking error bingo cards when reading a wiki page, not because I think Wikipedia is intentionally misleading, but because the sometimes author's perspectives are limited and concentrated on the work at hand.  QA is a second (or Nth) set of eyes to consider a topic.