Word of the Week: Oracle

Oracle Test Definitions

Thanks to:
Isaac Howard and Wayne J. Earl who had a great deal to do with the editing and formulation of this article.

Like my previous word of the week on Heuristics and Algorithms, this is a complicated one. According to wiki,

An oracle is a mechanism used by software testers and software engineers for determining whether a test has passed or failed.

According to the wiki citation, this comes from BBST, which has some thoughts about what an Oracle is or isn't.  Specifically it talks a lot about Oracle Heuristics, an interesting combination that Bach roughly states as a way to get the right answer some of the time.  I don't feel I have a problem with that, but then we go back into the BBST class and things get confusing.  On Slide 92 of the 2010 BBST course, it says,

How can we know whether a program has passed or failed a test?  Oracles are heuristics 

Slide 94 says:

An oracle is a reference program. If you give the same inputs to the software under test and the oracle, you can tell whether the software under test passed by comparing its results to the oracle's. 

Which later goes on to say that the definition is wrong.  The slides does so because of the claim that Oracles are heuristics.

Classic Problems With The Definitions

But how can this be so if Oracles know all?  They are the truth tellers.  Well the problem in software is that Oracles are not absolute like in the stories.  They give you an answer, but the answer might be wrong.

For example, you might test Excel and compare it to a calculator.  You take the calculator and enter 2.1 * 1 getting back the value 2.  Now perhaps the calculator is setup to provide integers, but when you compare it to Excel's output, you find that Excel gives back 2.1.  This appears to be a failure in Excel, but in reality it is a configuration issue.  The heuristic is in assuming your Oracle is right.  This might of course be a false assumption, or it might be only right in some circumstances.  Interestingly, one of the creators of BBST, Cem Kaner, has revised the definition of Oracle slightly in a posting about Oracles and automation,

...a software testing oracle is a tool that helps you decide whether the program passed your test.

Partial Oracle

While I don't want to get too far off track, I do want to note that Partial Oracles do exist, and from what I can tell, they are Oracles that tell you if the answer is even reasonable.  For example, for two positive integers, you can say that when you add them together, you will get a larger number than either of the separate digits.  1+1=2.  Thus 1<2.  3+4=7.  Thus 4<7.  In both cases the larger number is always smaller than the sum of the two numbers, for ANY two numbers.

New Questions

Let me chart out the idea of an Oracle:

1. Test: Tester runs test.  Example: Login with valid user.
1. Result: Login takes 5 seconds and goes to a internal page.
2. Request for Data: Make a request out to the Oracle; Was 5 seconds too long?
1. Process: Oracle considers the answer.
2. Data: Oracle generates the answer: Yes, 5 seconds is too long.
3. Compare: Verify if Test's Result are acceptable.
4. Output: Test's Results are not acceptable.
5. React: Tester reacts to the result.  Maybe they fail the test.  Maybe they...

Now lets get picky.  Is the monitor (the display) which is beaming information to you an Oracle?  It shows the results you requested and is a tool.  While I noted the parts that are the Oracle, who is this Oracle?  If the Oracle is your mind, then what makes this different from testing?

My colleague Isaac noted that the requirements of a test in most definitions includes making observations and comparing them to expectations.  For example, Elisabeth Hendrickson said,

Testing is a process of gathering information by making observations and comparing them to expectations.

Does this make an Oracle simply part of a test?  Even to be able to come up with the question seems to indicate you might suspect an answer.  Is this too long?  Well, in asking that, one assumes you have a built in answer in your head.  Perhaps you are wrong, but that is part of what an Oracle can be.

Alternatively, maybe an Oracle is an external source, thus it has to be outside of the "Tester".  If that is the case, then can the Oracle be the System Under Test?  Imagine testing using two browsers at the same time doing the above test and the login time has a large difference between browsers.  Is the Oracle the browser or the SUT?

Lets take a different approach.  Lets say you take screenshots after 4 seconds from logging in using an automated process.  You compare the screenshot to the previous version of the result, pixel by pixel.  If the pixels compare differently, the automation reports back a failure.  Where is the Oracle?  The request for data was to get a previous edition of the SUT in image form.  No processing occurred, so the Oracle can't be the processing, but perhaps the image itself is the Oracle.  Or is the Oracle the past edition of the site and the data?  Continuing into this, the automation pulls out each pixel (is that the Oracle?) then compares them.  But wait a minute... someone wrote the automation.  That someone thought to ask the question about comparing the images.  Are they the Oracle?  Since the author is the Tester (checker, whatever) the first time, capturing the first set of images, they saw what was captured and thus became a future Oracle.

Even if the Oracle is always an external source, is it always a tool?  Is a spec a tool?  Is another person (say a manager) a tool?  No, not that sort of tool.  Is a list of valid zip codes a tool or just data?

In case you are wondering, many of the examples given are real with only slight changes to simplify.

How Others Understand Oracles - BBST

Perhaps you feel this detailed "What is it?" questioning is pedantic or irrelevant.  Perhaps we all know what an Oracle is and my attempt to define it is just making a mess.  In order to address that question, I am going to do something a little different from what I have done in the past.  I'm going to open up my BBST experience a little, as I answered the question Kaner wrote about and then talk a little about it. To be clear, this answer has the majority of the content pulled out, as it could be used as an exam question in the future:

Imagine writing a program that allows you to feed commands and data to Microsoft Excel and to Open Office Calc. You have this program complete, and it’s working. You have been asked to test a new version of Calc and told to automate all of your testing. What oracles would you use and what types of information would you expect to get from each?

1. Excel (Windows) – Verify that the most common version of Excel works like Calc does. Are the calculations the same? Do the formulas work the same? If you save a Calc file can Excel open it? What about visa-versa?

a. I’m looking to see if they have similar calculations and similar functionality.

<snippet>

7. Stop watch – Does Calc perform the tasks at the same rough rate as Excel, Google docs? Is it a lot faster or a lot slower?

a. I’m looking to see if we are consistent with current user expectations of speed.

The responses I got were interesting in my peer reviews.  Please note I removed all names and rewrote the wording per the rules in BBST.  One person noted that item 7 would be categorized under performance and that item 1 was looking at consistency with comparable products.  Multiple reviewers felt I was looking at consistency, a heuristic Bach created.  What I find odd about that, is the need to label the Oracle when the Oracle (in my view at the time) was the tool, not the heuristic, therefore citing the heuristic of comparable products was not part of the question.  I got a complaint that I was not testing Excel or Google but Calc, yet the call of the question is about how I would use those Oracles.  One fair issue was, I should have noted I could have compared the old version to the new version using a stop watch, which I had missed.  However, I had cited Old Calc in my full document, so I think that was a relatively minor issue.

Since Oracles are tools, how can I not be implicitly testing Excel?  I kept hearing people say I should name my Oracles, yet to me I was naming them very clearly.  I got into several debates about if something like Self Verifying Data is in fact an Oracle (even though the article clearly has its own opinion on that)!  It seemed like everyone wanted to label the heuristic the Oracle, probably because of the "Heuristic Oracle" label in BBST.  While I did feel BBST failed to make clear what an Oracle is, it did make me think about Oracles a lot more.

Wrapping Up

I'm sorry if that felt a little ranty, but by talking about this subject, I want you to also think about what you see as an Oracle.  Oddly, Kaner himself cite's Doug Hoffman with items which I did not consider an Oracle (such as Self Verifying Data) when I started writing this article.  I think Kaner's own work defends his view point, as he doesn't appear to use his own rule (his definition) to the letter but rather the similarity of the items, a method humans appear to use frequently.

Truth be told, I'm not so sure that Oracle should be a word we in the industry should be using at all.  Isaac does not seem to believe in Oracles anymore, and appears to feel the definition is broken as it really cannot be separated from test.  To me, I see that many people seem to use it and perhaps it can have value if we shrink down the role of the word.  So let me wrap this up with my attempt to patch the definition into something that is usable.

Oracle: One or more imperfect external sources (often tools) that provide data or are data to assist in determining the success or failure of a test.

For a not exactly correct but useful working definition, an Oracle is the value that is the expected result.

What do you think a Oracle is?  Are we missing critical details or other expert views?  Feel free to write us a note in the comments!