It is in Java and was intended as part of a demo for a 2 hour presentation. Because of the complexity of the system, I'll write some notes about it here. I used IntelliJ to develop this code and recommend using it to view the code. It does use Maven for the libraries, including TestNG which you can run from IntelliJ. Many of the concepts could be translated to C# with little difficulty.
So what does it do? It demonstrates a few different, simple examples of reflections and then a build up of methods for generating test data in reflective and possibly smarter fashion (depending on context). I'm sure your sick of hearing about reflections from me, so I'll try to make this my last talk on them for a while, unless I come up with something new and clever.
As a brief aside, Isaac claims that while this is a valiant attempt to write out a code walkthrough, it really needs to be a video or audio recording. Perhaps so, but I don't want to devote the time unless people want it or will find it useful. As I have my doubts, I'm going to let the text stand and see if I get anyone requesting a video. If I do maybe I'll put some time into it. Maybe. :)
Now on to the code...
First the simple examples which I do use in my framework but not as simple as this. The two simple examples are DebugData and ExampleOfList and both are under test/java/SimpleReflections. DebugData shows how you can use reflections to print out a simple object one level. down. It 'toStrings' each field in the object given. Obviously if you wanted sub-fields that would take more complex code, but this is often useful. ExampleOfList takes a list of string and runs a method on each item on the list and returns back the modified list. Obviously this could be any command, but for simplicity of the demo I limited it to methods that did not take arguments.Now all the rest of the code is around different methods for generating data. I will briefly describe each of them and if you want to you can review the code.
The HardcodedNaiveApproach is where all of the values are hard coded by using quoted strings. E.G. x.setValue("Hard coded"); This is a good method for 1-3 tests but if you need more you probably don't want to copy and paste that data. It is hard to maintain, so you might go to the HardcodedSmarterApproach. This method uses functions to return objects with static data so you can follow the DRY principle. However, all the data is the same each time. So you add some random value, maybe append it to the end. The problem is what are your equivalent class values? For example, do you want to generate all Unicode characters? What about the error and 'null' characters? Are negative numbers equally valid to positive numbers. If not, then your methods are less DRY than you might want as you will need different methods for each boundary, if that matters. Also you are writing setters for each value, which might fail when a new property is added. We haven't even talked about validation yet, which would require custom validators based upon the success/failure criteria generated by the functions you write. That is to say if you generate a negative number and it should fail for that, not only does your generator have to handle that but your validator does as well. What to do?
Perhaps reflections could help solve these problems? The ReflectiveNaiveApproach instead uses the typing system to determine what to generate for each field in a given class. An integer would generate a random integer and a string would generate a random string. We know the field name and class type so we could add if statements for each field/type but that puts in the same maintenance of new properties we had with the hard coded approaches. If we didn't do that we could still handle new properties assuming we knew how to set the type, but it might not fit the rules of the business logic and we have no way to know if it should work or not. For fuzz testing this is alright, but not functional testing. Is there any solutions? Maybe.
The final answer I currently have is the ReflectiveSmarterApproach. In effect when you need to generate lots of different data for lots of different fields, you need to have custom generators per class of fields. What is needed is an annotation for each field telling it what needs to be generated. An example of that can be found in the Address class. Here is a partial example of this:
public class Address {
@FieldData(dataGenerators = AverageSizedStringGenerator.class)
private String name;
@FieldData(dataGenerators = AddressGenerator.class)
private String address1;
//...
}
Now let us look at an example generator:
public class AddressGenerator extends GenericGenerator {
@Override
public List<dynamicdata> generateFields() {
List<dynamicdata> fields = new ArrayList<dynamicdata>();
fields.add(new DynamicData(RandomString.randomAddress1(), "Address", DynamicDataMetaData.PositiveTest));
fields.add(new DynamicData("", "Empty",
new DynamicDataMetaData[] {DynamicDataMetaData.NegativeTest, DynamicDataMetaData.EmptyValue}).
setErrorClass(InvalidDataError.class));
return fields;
}
}
This generator generates a random address as well as an empty address. It is clear one of these addresses is valid while the empty address appears to be a negative test.
Through the power of reflections you can do something like this:
List<DynamicDataMetaData> exclude = new ArrayList<DynamicDataMetaData>(); exclude.add(DynamicDataMetaData.NegativeTest); ReflectiveData<Address> shippingAddress = new CreateInstanceOfData<Address>().setObject(new Address(), exclude);
The exclude piece is where you might filter out generating certain values. Say you want to only do positive (as in expected to be successful) tests, you might filter out the negative tests (those that expect not to complete the task and possibly cause an error). The third line generates you an object with all the properties that have the attached annotation and values. Now this does not handle new fields automatically but it could certainly be designed to error out if it found any un-annotated fields (it is not at present designed to do this) and if you embed the code in your production code, it would be more obvious to the developer they need to add a generator.
Now how do we pick which value to test when we could test with the empty address or a real address? At present it picks it randomly because according to James Bach, random only takes roughly 2x to get equal coverage to pair wise testing. Since we know all about the reason for generating a particular value (what error it would cause, etc.) we can at run time say how the validation should occur. The example validation is probably a bit complex, but I was running out of time and got a bit slap-dash on that part. One issue with this method is it is hard to know what your coverage is. You can serialize the objects for later examination and even create statistical models around what you generated if needed.
Summary
Obviously this is a somewhat heavy framework for generating say 20 test data values. But when you have a much larger search space that approaches infinite this is a really valuable tool. I have generated as many as 50 properties/fields about 400,000 times in a 24 hour period. That is to say, generating roughly 400,000 tests. I have found bugs that even with our generator would only be seen 1 : 40,000 times and would probably have never been found in manual testing (but would likely be seen in production). The version I use at work has more than a years worth of development and research, supporting a lot more complexity than exists in this example, but I also don't think it could be easily be adapted as it was built around our particular problems.This simple version can be made to support other environments with relatively little code modification. It took much of the research and ideas I had and implemented in a simpler fashion which is more flexible. You should easily be able to hook up your own class, create annotations and generates and have tests being generated within a day (once you understand how). On the other hand it might take a little longer to figure out how to do the validation as that can be tricky.
One problem I have with what I have generated is there is no word or phrase to describe it. In some sense it is designed to create exploratory data. In another sense it is a little like model driven testing in that it generates data, has an understanding of what state it should go to and a method to validate it went to the correct state. However it doesn't traverse multiple states and isn't designed like a traditional MDT system. Data Driven Testing describes a method for testing using static data from a source like a csv or database. While similar, this creates dynamic tests that no tester may have imagined. Like combinatorics, this creates combinations of values, but unlike pairwise testing, the goal isn't just generating the combinations (which can be impossible/unpractical to enumerate) but to generate almost innumerable values and pick a few to test with, while enforcing organization of your test data. This method also encourages usage of ideas like random values while combinatorics is designed to have a more static set of values. Yes you can make combinatorial ideas works with non-static sets, but it requires more abstraction (E.G. Create a combination of Alpha, Alpha-numeric, ... and this set of Payment method, now use the string type to choose what generate you use) and complexity. Finally combinatoric methods can have difficulties when you have too many variables, depending on implementation. This is a strange hybrid of multiple different techniques. I suppose that means it is up to me to try to name it. Let's call it:
I would say don't expect any major changes/additions to the design unless I start hearing people using it and needing support. That being said I love feedback, both positive and negative.
While researching for this article I came across this which is cool, but I found no good place to cite it. So here is a random freebie: http://en.wikipedia.org/wiki/Curse_of_dimensionality
