Showing posts with label Design. Show all posts
Showing posts with label Design. Show all posts

Wednesday, September 23, 2015

Link Mania: 75 Links with Commentary on Testing, Security, Tech and Life

I have a friend who wants to get into security testing and asked for my help in broadening their education.  I assumed that my friend would have a good number of links on security, and so I would try to start with a broader set of testing knowledge.  They have a degree and know basic computer science, but in discussing needs, I decided that my primary goal would be to provide a broader understanding around testing.  I started by emailing some links I thought might be useful... but that quickly became clear that it was less manageable, so over a few months I put together groups of links I thought might be of value into a Google Doc.

I also wrote some notes on why I thought they were of value for my friend.  It should be noted that I cite myself frequently.  This is not because I'm full of myself but because I addressed specific questions my friend had and felt that my research + links I had in my posts would be of some benefit.  I did do some light editing of my notes, removing personal comments for privacy reasons, but this is roughly what I came up with.

One limitation to this sort of document is I could go on forever.  I certainly could have written several dozen posts around these links.  Before I turn you loose to all these links, let me give some advice.  I look in detail for things I need to know soon and skim general topics for data I might value in the future.  It’s a method that allows me to work well with a large set of disparate data.  It’s why I can talk about all sorts of subjects at some level, but am not an expert in any.  Knowing who actually is worth reading helps me filter the 'must reads' from the 'skims'.  So as you look at these links, look for what you want to learn about, and then look at the author.  If the data is useful, look for more of that author, even links I did not suggest. If it wasn't helpful, try a different author.  If you try about 3 different links in a row in this set, and all are not useful, either you are not primed to learn about the subject, or the type of data/authors I have gathered are not optimal for how you learn.  You will have to decide which.

Happy reading!

Links I Came Up With Before I Started Organizing [Mostly Test]



Link

JCD's notes
http://about98percentdone.blogspot.com/2015/01/leveling-up-your-testing-skills.html I literally wrote this for people trying to improve themselves.
http://bbst.info/?page_id=23 Lectures 2,3,5,6 in particular.  They are about 30 mins per lecture.  Great testing stuff.

The lecture has power point slides, which can be downloaded and are useful by themselves, HOWEVER, he talks about different points not just captured in the slides.
http://about98percentdone.blogspot.com/2013/09/where-cdt-fails-rebuttal.html Isaac’s effort to describe what testers should learn in ‘levels’. Good stuff.
http://www.testingreferences.com/software_testing_bookstore.phpThe books I read that they recommend are good, thus I trust their recommendation enough to suggest taking a look at the list.
http://www.developsense.com/blog/ An interesting thinker in the testing space.
http://www.satisfice.com/blog/archives/1346James Bach, one of a hand full of people to affect the modern software testing world greatly with his words.  In one of his various attempt to define testing.  This one is of particular interesting, because it is about the art and act of doing testing.



http://about98percentdone.blogspot.com/2014/06/my-current-test-framework-testing-large.html
Reflections are a mind-expanding technique to make code think about itself.





http://steve-yegge.blogspot.com/2006/10/egomania-itself.html
I really love Steve Yegge, he maybe one of the best writer-programmer combinations I know of.  I come back to his work frequently.  But he is a programmer, and as such, the one that really matters is the top link.  The rest are very programmer centric.
http://blog.codinghorror.com/ Heard of Stack Overflow?  This guy made that (along with Joel Spolsky of Joel On Software, but Joel’s stuff is getting old).  He writes on a variety of topics and on his good days is really good.  That said, he’s still a developer, thinking like a developer.


Research in Life / Happiness / Living / The Mind / General Career Advice



Link

JCD's notes

http://www.ted.com/talks/brene_brown_listening_to_shame?language=en
These have made me think about how I live my life.

http://about98percentdone.blogspot.com/2014/06/what-is-highest-level-of-skill-in.html
Learn about what you should do to make yourself more valuable in life.  Learning about what matters in learning.
http://about98percentdone.blogspot.com/2014/02/being-fraud-and-failure.html How to deal with feeling like you don’t know enough, when in fact you’re driven to know more.

I have referenced this exact blog post more in my comments to other people’s blog entries than any other.  Often the people who care the most feel this way.
http://about98percentdone.blogspot.com/2013/12/book-consideration-introduction-to.html Read the bullet points at the bottom.  In particular, “The Answer”.  The book reviewed is a little bit of a personal and spiritual look at science.  Somewhat like Sagan, but revolved around thinking.  You might like the book.
http://about98percentdone.blogspot.com/2013/09/testing-hiring-process-for-testers.html

http://about98percentdone.blogspot.com/2013/09/a-subject-of-hiring-process.html

http://about98percentdone.blogspot.com/2013/09/my-interviewing-start-and-changes-ive.html
Isaac and I wrote a little bit about getting hired. These are some of my early blog entries, but you also get both the view of a person being hired by Isaac as well as, Isaac, the hiring manager's method of choosing to hire people like me.

https://sites.google.com/site/steveyegge2/miracle-interview
More interview related thoughts, but from a developer side.
https://sites.google.com/site/steveyegge2/age-racecar-driver (same guy as above, but more philosophical questions in specialization.)



http://www.stickyminds.com/article/helpful-tips-hiring-better-testers
The last set of articles Isaac and I have on interviewing.
http://www.moserware.com/2009/01/wetware-refactorings.html Fairly good.  Almost all true.  Interesting ideas and good set of resources.
http://www.moserware.com/2008/03/what-does-it-take-to-become-grandmaster.html I would give this a lower priority because it’s long and this is not the first time I have found someone who says this and put it in the list.  Hopefully you see the patterns… and will learn.
http://breakingsmart.com/An interesting description of the tug between the past and the future.  It seems to be a little dismissive of some problems, but the general picture is not wrong.
http://blog.codinghorror.com/level-5-means-never-having-to-say-youre-sorry/I hate the title, but the content is pretty good. It provides some insights into why generating reactionary, scripted systems does not scale well in creative work.

Paul Graham is fantastically interesting in general.  This is an interesting attempt to correlate technological outcomes to culture.  I recently heard from a friend who lived in Japan for a year about these Japanese workers at this company, whose company was bought out by an American company after it started to fail, had a real difficult time letting go of the quality of a product in order to get the product out the door. That story feels like a sort of quick CRC check for me, meaning this article probably does have some veracity.
http://www.ribbonfarm.com/the-gervais-principle/I know I have talked about this before, but it is an interesting model of human behavior.  While I’m not actually a fan of The Office, I found it was mostly ‘translatable’.  It also offers small notes on the Peter and Dilbert principles which are also worth looking at.


Testing



Link

JCD's notes
https://www.youtube.com/watch?v=j_JviA5nvS0&list=PLSIUOFhnxEiDFckNDSjKWqOCtd8ksJrh4GTAC is fairly good.  Some of the content might be on security. I have not watched all the 2014 (and soon 2015) videos, only the 2013 videos which were often very good.
http://www.associationforsoftwaretesting.org/conference/cast-2015/ CAST 2015 is probably worth watching. I have yet to watch it myself, but I attended last year.
http://www.testingreferences.com/testingnews.phpI use this as a tool, looking for things of interesting, read them, keeping note of who wrote it and if I decide I don’t like someone (for any reason), I mentally filter them out of the list.  It’s the firehose method, good for going in all different places, but you never know what you might get.

How SQL joins work. They are often used both for getting data for testing as well as part of how many reports are generated.
http://angryweasel.com/blog/In broad strokes, I agree with Alan and appreciate his perspective with about 20 years at Microsoft.
http://oredev.org/2013/wed-fri-conference/balancing-atdd-gui-automation-and-exploratory-testing A video I watched long ago that is still in my notes.  Might be interesting from an automation vs exploratory testing perspective.  I have little memory of it though. Most people only teach the basics :(
http://www.huibschoots.nl/wordpress/ Like Alan, I agree with Huib broadly, and I find his view very interesting as he comes from a European background.


General Tech



Link

JCD's notes
https://howdns.works/episodes/ How DNS works.
http://blog.codinghorror.com/on-software-engineering/Why software engineering is so difficult, and why consultants are sometimes looked down upon.
http://www.dreamincode.net/forums/topic/223324-an-interesting-interview-with-steve-yegge-and-james-duncan-about-java/ Interesting set of interviews, the biggest point is that you should know your tools well enough you know what is wrong with them.
http://www.moserware.com/2009/03/how-net-regular-expressions-really-work.htmlI think the data here is interesting, however, the meta is important too, even at a security level.  A DDOS attack is a security attack, but if I can DDOS you with one click, it is a security issue too.  Knowing how regular expressions work tells you about how a lot of systems work inside, which is how samy was able to defeat the defenses of MySpace ( http://namb.la/popular/ ).  It’s also a huge part of getting things done.
http://edge.org/annual-question/what-do-you-think-about-machines-that-think An interesting set of speakers, most very smart, all on the same topic.  The 2014 topic was interesting too. This might generate some interesting questions, such as:


What would happen if/when the #1 job goes away (Driving, at least in the USA; currently biggest economy in the world; last count ~ 3 million souls in the US do this)?  
What do we do with these people?  
What of those that aren’t capable or interested in more mind-oriented work or who are too old to change careers?  
What of my grandmother who doesn’t interact well with online work, in part because her hands are too crippled to do much with keyboards and did not grow up with this world?
What happens when machines categorize someone as an outlaw in some countries which others counties would recognize as moral and legal (such as being gay, or a political dissidents)? Who is legally and morally responsible for a 'thinking' machine?

https://sites.google.com/site/steveyegge2/math-every-day
Here is a mission in life -- how do you solve problems in which there is no solution?  (My only hint is find another problem.)  However, the point is, solving one of these non-trivial problems is a lifetime’s worth of work, if you want it.  See the intro of the second piece to see what I mean.
http://www.amazon.com/review/RUGSCP3XBNBUVSuggested by Yegge’s post… it’s an interesting read in and of itself.  It’s got some ideas I’ve not really researched.


Security



Link

JCD's notes
https://www.youtube.com/watch?v=n9-Gz1U87CI&index=9&list=PLQB4l9iafcelXpJnK6IyDsoFeEb1icqrl Brilliant! I attend this live and you’ll see me in the end asking a question.  All the rest of the talks (I kept the list) are test related, but this is security related.






(Others…)
https://heimdalsecurity.com/blog/best-internet-security-blogs/
Famous security researchers blog.
https://krebsonsecurity.com/category/how-to-break-into-security/ How to learn to be a security professional.
https://www.schneier.com/blog/archives/2013/04/nice_security_m.html Why we are all frauds and failures sometimes (As address in a link above).


A *practical* piece of tech that really applies around security of the web, he has other good posts too.

http://www.guru99.com/learn-penetration-testing.html
I liked the graph, purely from a ‘what do you mean by “security testing”’ question. NOTE: My friend suggested http://www.techrepublic.com/blog/it-security/the-five-phases-of-a-successful-network-penetration/ might also be useful.
https://www.cs.purdue.edu/homes/xyzhang/fall07/Papers/sw-test.pdf Read this a long time ago and it was a little helpful.
http://www.veracode.com/security/software-security-testingHappened upon this, no idea if it is any good. I did not get chance to really review it.
http://blogs.msdn.com/b/oldnewthing/archive/2013/12/24/10484402.aspxMight be interesting, taken from a comment from...
http://blog.codinghorror.com/welcome-to-the-internet-of-compromised-things/here.  Also interesting.
https://www.grc.com/fingerprints.htmThey have some interesting stuff regarding SSL but more importantly they might be a generally useful resource.
https://www.youtube.com/watch?v=wKDE_upBlfcComplex example of a hack and how it can be done by defeating multiple layers and understanding history.


Other



Link

JCD's notes
http://blog.incubaid.com/2012/03/28/the-game-of-distributed-systems-programming-which-level-are-you The technical subject is interesting and it provides insight into architecture. It ALSO gives a nice idea of levels and how you can’t really know what level you are at until you see the next level up. I actually have a more complex theory on this, but I have yet to write the blog post.
http://www.moserware.com/2009/07/just-enough-mba-to-be-programmer.html Just plain useful in a practical sort of way.
http://archive.wired.com/wired/archive/4.12/ffglass_pr.htmlI have not yet completely read it but it was recommended to me by a trusted source.  It’s long.
http://archive.wired.com/wired/archive/8.04/joy.htmlVery long indeed, but worth reading.

You made it to the bottom?  And you counted them?  Only 74 links and you want your money back? Alright! Fine. Well I'm sure I can come up with a 75th link just for you, my bean counting friend.  How about something interesting?  Like really interesting.  Here you go: http://www.damninteresting.com/the-zero-armed-bandit/

Thursday, April 9, 2015

The Future of Load Testing

First of all, I won't pretend I actually know what the future of Load Testing will look like, but I want to describe some of the different ideas I have seen and done.  Some of these things I have not seen or heard of by anyone else, certainly not on the internet.  So hopefully these will expand your thinking around Load Testing.

What is the Purpose of Load Testing?


Functional testing is designed to exercise the functionality of the system.  Frequently people talk about using Selenium or QTP to exercise a particular piece of functionality, often in the UI.  With the Test Pyramid it is suggested that these sorts of tests should try to hit below the UI level for various reasons.  No matter, if you test an API, a UI, Console App or some other hook below the UI, if your concern is around the Testing Pyramid, it is likely you are trying to test the functionality.  You are often interested in large sets of behaviors and the way the system responds.  When you do Load Testing, in most cases, few of the broad tests are as important.  You are not trying to see if the system will handle all the corner cases and often Load Tests don't even check anything besides a response code.  The raison d'etre for a Load Test on the other hand often has less to do with how the system functions but in ways the system reacts to many different inputs occurring in a relatively quick fashion.  Granted some Load Tests are less about the number of inputs and more about the style of the input (E.G. large files) or other types of constraints (e.g. with less RAM).  To sum it up in a general statement, Wikipedia describes it this way:
Load testing is the process of putting demand on a system or device and measuring its response. - http://en.wikipedia.org/wiki/Load_testing
However, the majority of Load Tests simply want to understand how many inputs a system can handle given a certain profile of inputs.

All of that sounds rather abstract, but if you go reading Microsoft's very hand guide on types of performance tests, you will see that the underlying purpose of this sort of testing varies.  They use terms like load test, performance test and stress test for different meanings.  I think all of this data is really useful and valid.  I however, am going to use Load Test (in capitals) to describe any number of different purposes.  Instead, what I want to look at is what sorts of ways we can organize our testing to make for a better long term experience.  These ideas could be applied to many of the various purposes of Load Testing, so assuming you understand your Load Testing goals you can tailor these ideas to your organization.

35-50% of the Internet Traffic


The first idea I think worth exploring is the Netflix model of Load Testing.  Effectively, trying to have a production-like QA system is silly for Netflix because that would be like having a second set of the internet for QA.  In fact, if you were like Netflix, you would then have to have an insane number of additional systems to generate a load anything like your customers...  or you could just have production traffic mirrored between the two systems.  Having a second prod-like environment is of course not going to work.  So they came up with a radical set of strategies, but I think this sums it up nicely:

We have found that the best defense against major unexpected failures is to fail often. By frequently causing failures, we force our services to be built in a way that is more resilient.  - http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html
The best method they found was basically to attack production and see how it would respond.  This however still leaves the question of how do they specifically test their code will handle new loads.  While they do some Load Testing, their biggest defenses are the scaled down production traffic mirrors they run, the ability to go back to old code and the fact that AWS allows you to spin up new instances all the time.  In effect, they turn the problem on its head, but this really only is helpful if you can use AWS to grow quickly and if your traffic is relatively uniform.  Also, at Netflix scale, you can hire a lot of engineering talent to build this up.


The Limits of Load Test Systems


When trying to leverage user traffic doesn't work, you have to start looking for other options.  Using something like JMeter is interesting.  I call out JMeter because that is what Netflix uses above and beyond their production traffic mirror.  A tool like JMeter takes traffic from a proxy and feeds it into a script (I guess you could hand code it if you are crazy).  Then you edit the script and parameterize it.  You run the script over and over again using multiple threads and try to create load.   These tools might instrument the systems under test or you might have to do that yourself.  In either case, the data gathered is outputted and left for some poor soul to try to understand.  After having been in this position several times, let me say that it truly is difficult to understand these results, in particular because you had to use record-playback from a proxy.  Just as it is a bad idea to use record-playback in automating your functional tests, I think it is a bad idea to do so with such tools.

In one of my former companies, they had one specialist whose only job was the deal with these proxy-recorded scripts.  They are a mess and I'm not going to pretend I know how to fix them.  However, I do have a few ideas and all of them involve your already created friends, the functional tests.

Functional Tests == Load Tests: With or Without UI

When you have a functional test, you might have a complex setup and tear down.  However, the test itself is often fairly simple.  There are two major ways you can create functional tests, one is using a UI and the other is to hit just below the UI, perhaps at an API level.  So logically you can do two things to create load.  One is to scale up your UI tests.  I have seen this done and know of others who have tried to do it.  It is a fairly big engineering feat to create a Load Test using Selenium, but I know it can be done.  Be warned, this can be very expensive, as it requires one OS per hand full of threads you want to run plus overhead for selenium hub nodes.  The other option is to use your existing API tests and create a Load Test overtop of that.  You might have to simplify the data creation and you might have to remove the validation if those take too long, but this is a very easy method of Load Testing the system.  I have personally built several Load Tests around this idea.

Now we have talked for years about how functional tests should be run in a CI environment.  If you are writing your Load Tests like you write your functional tests, using the same systems, then why not run your Load Tests nightly?  Obviously there are some questions you want to ask up front, like if someone will be alerted because of it or what impact that has on your functional tests.  Another question you might consider is what sort of load do you want?  If you need to actively watch to make sure the system stays up, a nightly Load Test would not make much sense.  On the other hand, what if you did a small load for a short period of time?  You could capture the resulting data and plot it on a chart.  This way as you gathered more nightly data, you would have a rough understanding of what you expect.  Now, you aren't running a Load Test once a sprint with little idea of what changes might cause impact, with specialized scripts that take a lot of effort to maintain.  Instead, you have a trend line and will notice changes.  Now this won't tell you when you will fall over or some other data points, but it does give you a change detector.  Furthermore, when you have to fix your functional test, your fix automatically goes into the Load Test.

The next piece is you can run multiple 'threads' at once, not all of which are load related.  If your Load Test can't do validation, while running the Load Test you can run some functional tests to see if the system still appears to function.  Since this is a job in your CI manager, it should be easy to kick off.  You can even manually test while your Load Test runs.

Eventually, you might realize that your functional and load tests only vary in how much load there is and how complex the setup/validation is.  You might realize, like the Load Test, you can metrickitize the functional tests so that you see if a functional test start taking longer.

Load Test?  What is that?

If you notice my description, my efforts are to make the code base easier to maintain and have as much data as possible.  This ultimately makes your Load Testing efforts and your functional efforts look very similar.  They live in the same code base, call the same functions, effectively the concepts merge.  The only differences are in the design, setup, cleanup and how heavy the validation is.  I do think there is value in having different words to describe the intent, but merging the code allows you to get more done with less resources.  One of the biggest advantages I have personally seen is that I actually understand what the test does, where as when I was using tools like JMeter, I often had no idea how it worked.  I have learned a ton about HTTP and HTTPS just by building my own tools.  Now not everyone has time for that and I think ultimately we will want tooling to help make this easier.  However, I am not sure if the costs of having a different tech stack and code base is worth the value the current tools provide, so you might have to make your own for now.

If you have found your Load Testing tools are working well today, then feel free to ignore this.  I know not everyone has the same needs as we have.  I know that the tools we have today do serve some purposes, but my experience hints that they frequently are as much trouble as the value they add.

"In the Year 2000"

- Conan O'Brien, et al.

In trying to predict the future it is really difficult to say what will or will not happen.  Conan O'Brien has been predicting what will happen in the year 2000 for more than 15 years, but unlike him, I have not had the benefit of seeing the future.  With that said, I suspect that in the future we will see more machine learning style systems that take our Load Test data and create load profiles based upon real time data.  Such a system would be able to adaptively adjust based upon metrics in the systems under test and would also detect what changes happened and what code appears to have caused a particular slowdown.  It will correlate this and help find what is causing systems to fail.

I also suspect that we will have a better set of load profiles that can push or break systems.  Load Test systems of the future might go through those profiles on a daily or build-basis and inform you when a build/day has uncharacteristic results, again based upon some form of machine learning.  It will start looking a lot more like functional tests which you only examine when something strange occurs or when you are auditing your tests.

Obviously all of this takes some fair amount of work and effort to produce.  We presently don't have the tooling to do this and while bits and pieces have been worked at, I have heard of no one actually doing this.

What sorts of things would you like to see in future load testing systems?  What areas have you struggled with?  I have found very little experiential data around load testing.  I'd love to hear from others on this topic, even if you just dump a link to your blog post.

Friday, June 27, 2014

My Current Test Framework: Testing large datasets

I recently wrote how I felt few people talk about their framework design in any detail.  I feel this is a shame, and should be corrected as soon as possible.  Unfortunately, most companies don't allow software, including testing automation to be released into the public.  So most of the code we see is from consultants with companies occasionally okaying something.  In my case, I did something like a clean room implementation of my code.  It is much simplified, does not demonstrate anything I did for my company nor does it directly reference them.  It is open source and free to use.  Without further delay, here is the link: https://github.com/jc-d/Demos.

It is in Java and was intended as part of a demo for a 2 hour presentation.  Because of the complexity of the system, I'll write some notes about it here.  I used IntelliJ to develop this code and recommend using it to view the code.  It does use Maven for the libraries, including TestNG which you can run from IntelliJ.  Many of the concepts could be translated to C# with little difficulty.

So what does it do?  It demonstrates a few different, simple examples of reflections and then a build up of methods for generating test data in reflective and possibly smarter fashion (depending on context).  I'm sure your sick of hearing about reflections from me, so I'll try to make this my last talk on them for a while, unless I come up with something new and clever.

As a brief aside, Isaac claims that while this is a valiant attempt to write out a code walkthrough, it really needs to be a video or audio recording.  Perhaps so, but I don't want to devote the time unless people want it or will find it useful.  As I have my doubts, I'm going to let the text stand and see if I get anyone requesting a video.  If I do maybe I'll put some time into it.  Maybe. :)

Now on to the code...

First the simple examples which I do use in my framework but not as simple as this.  The two simple examples are DebugData and ExampleOfList and both are under test/java/SimpleReflections.  DebugData shows how you can use reflections to print out a simple object one level. down.  It 'toStrings' each field in the object given.  Obviously if you wanted sub-fields that would take more complex code, but this is often useful.  ExampleOfList takes a list of string and runs a method on each item on the list and returns back the modified list.  Obviously this could be any command, but for simplicity of the demo I limited it to methods that did not take arguments.

Now all the rest of the code is around different methods for generating data.  I will briefly describe each of them and if you want to you can review the code. 

The HardcodedNaiveApproach is where all of the values are hard coded by using quoted strings.  E.G. x.setValue("Hard coded");  This is a good method for 1-3 tests but if you need more you probably don't want to copy and paste that data.  It is hard to maintain, so you might go to the HardcodedSmarterApproach.  This method uses functions to return objects with static data so you can follow the DRY principle.  However, all the data is the same each time.  So you add some random value, maybe append it to the end.  The problem is what are your equivalent class values?  For example, do you want to generate all Unicode characters?  What about the error and 'null' characters?  Are negative numbers equally valid to positive numbers.  If not, then your methods are less DRY than you might want as you will need different methods for each boundary, if that matters.  Also you are writing setters for each value, which might fail when a new property is added.  We haven't even talked about validation yet, which would require custom validators based upon the success/failure criteria generated by the functions you write.  That is to say if you generate a negative number and it should fail for that, not only does your generator have to handle that but your validator does as well.   What to do?

Perhaps reflections could help solve these problems?  The ReflectiveNaiveApproach instead uses the typing system to determine what to generate for each field in a given class.  An integer would generate a random integer and a string would generate a random string.  We know the field name and class type so we could add if statements for each field/type but that puts in the same maintenance of new properties we had with the hard coded approaches.  If we didn't do that we could still handle new properties assuming we knew how to set the type, but it might not fit the rules of the business logic and we have no way to know if it should work or not.  For fuzz testing this is alright, but not functional testing.  Is there any solutions?  Maybe.

The final answer I currently have is the ReflectiveSmarterApproach.  In effect when you need to generate lots of different data for lots of different fields, you need to have custom generators per class of fields.  What is needed is an annotation for each field telling it what needs to be generated.  An example of that can be found in the Address class.  Here is a partial example of this:

public class Address {
 @FieldData(dataGenerators = AverageSizedStringGenerator.class)
 private String name;
 @FieldData(dataGenerators = AddressGenerator.class)
 private String address1;
 //...
}

Now let us look at an example generator:


public class AddressGenerator extends GenericGenerator {
 @Override
 public List<dynamicdata> generateFields() {
  List<dynamicdata> fields = new ArrayList<dynamicdata>();
  fields.add(new DynamicData(RandomString.randomAddress1(), "Address", DynamicDataMetaData.PositiveTest));
  fields.add(new DynamicData("", "Empty",
   new DynamicDataMetaData[] {DynamicDataMetaData.NegativeTest, DynamicDataMetaData.EmptyValue}).
   setErrorClass(InvalidDataError.class));

  return fields;
 }
}


This generator generates a random address as well as an empty address.  It is clear one of these addresses is valid while the empty address appears to be a negative test.

Through the power of reflections you can do something like this:


List<DynamicDataMetaData> exclude = new ArrayList<DynamicDataMetaData>();
exclude.add(DynamicDataMetaData.NegativeTest);
ReflectiveData<Address> shippingAddress = new CreateInstanceOfData<Address>().setObject(new Address(), exclude);


The exclude piece is where you might filter out generating certain values.  Say you want to only do positive (as in expected to be successful) tests, you might filter out the negative tests (those that expect not to complete the task and possibly cause an error).  The third line generates you an object with all the properties that have the attached annotation and values.  Now this does not handle new fields automatically but it could certainly be designed to error out if it found any un-annotated fields (it is not at present designed to do this) and if you embed the code in your production code, it would be more obvious to the developer they need to add a generator. 

Now how do we pick which value to test when we could test with the empty address or a real address?  At present it picks it randomly because according to James Bach, random only takes roughly 2x to get equal coverage to pair wise testing.  Since we know all about the reason for generating a particular value (what error it would cause, etc.) we can at run time say how the validation should occur.  The example validation is probably a bit complex, but I was running out of time and got a bit slap-dash on that part.  One issue with this method is it is hard to know what your coverage is.  You can serialize the objects for later examination and even create statistical models around what you generated if needed.

Summary

Obviously this is a somewhat heavy framework for generating say 20 test data values.  But when you have a much larger search space that approaches infinite this is a really valuable tool.  I have generated as many as 50 properties/fields about 400,000 times in a 24 hour period.  That is to say, generating roughly 400,000 tests.  I have found bugs that even with our generator would only be seen 1 : 40,000 times and would probably have never been found in manual testing (but would likely be seen in production).  The version I use at work has more than a years worth of development and research, supporting a lot more complexity than exists in this example, but I also don't think it could be easily be adapted as it was built around our particular problems.

This simple version can be made to support other environments with relatively little code modification.  It took much of the research and ideas I had and implemented in a simpler fashion which is more flexible.  You should easily be able to hook up your own class, create annotations and generates and have tests being generated within a day (once you understand how).  On the other hand it might take a little longer to figure out how to do the validation as that can be tricky.

One problem I have with what I have generated is there is no word or phrase to describe it.  In some sense it is designed to create exploratory data.  In another sense it is a little like model driven testing in that it generates data, has an understanding of what state it should go to and a method to validate it went to the correct state.  However it doesn't traverse multiple states and isn't designed like a traditional MDT system.   Data Driven Testing describes a method for testing using static data from a source like a csv or database.  While similar, this creates dynamic tests that no tester may have imagined.  Like combinatorics, this creates combinations of values, but unlike pairwise testing, the goal isn't just generating the combinations (which can be impossible/unpractical to enumerate) but to generate almost innumerable values and pick a few to test with, while enforcing organization of your test data.  This method also encourages usage of ideas like random values while combinatorics is designed to have a more static set of values.  Yes you can make combinatorial ideas works with non-static sets, but it requires more abstraction (E.G. Create a combination of Alpha, Alpha-numeric, ... and this set of Payment method, now use the string type to choose what generate you use) and complexity.  Finally combinatoric methods can have difficulties when you have too many variables, depending on implementation.  This is a strange hybrid of multiple different techniques.  I suppose that means it is up to me to try to name it. Let's call it:  JCD's awesome code.  Reflective Test Data Model Generation

I would say don't expect any major changes/additions to the design unless I start hearing people using it and needing support.  That being said I love feedback, both positive and negative.

While researching for this article I came across this which is cool, but I found no good place to cite it.  So here is a random freebie: http://en.wikipedia.org/wiki/Curse_of_dimensionality

Wednesday, October 2, 2013

Refactoring: Improving The Design Of Existing Code

Having read many books on the basics of programming, building up queries and the basics of design, I have found that almost none really talk about how to deal with designing code outside abstract forms. Most books present design in a high level, possibly where a UML is shown and revolves around most of the OOP connections. Some talk over how to use design patterns or create data structures of various types. All seem to be under the illusion that, we as programmers actually can apply abstract concepts into real every day practical methodologies. Almost always they use things like animals or shapes or other "simple" (but not practical) OOP examples. They are always well designed, and seem relatively simple. In spite of that, I think I have learned how to do proper design to some degree by years of trial and error.

Refactoring, on the other hand starts out with a highly simple, yet more real world, movie system, and then slowly but surely unwraps the design over 50 pages. It is one of the most wonderful sets of “this code works, but let us make it better” I have ever seen, and it is dead simple in implementation. The book starts with a few classes, including a method to render to a console. Then blow by blow, the shows differences via a bolding of each change. The book talks design choices like how having a rendering method that contains what most would call “business logic” prevents you from easily having multiple methods of rendering (I.E. Console and html) without duplicate code. They also make a somewhat convincing argument against temporary variables, although I am not 100% convinced. Their reasoning is that it is harder to refactor with temp. variables, but sometimes temp variables (in my opinion) can provide clarity, not to mention they tend to show up in debugger watch windows. To be fair, later on, the author notes that adding back in temporary variables for clarity, but it is not emphasized nearly as much.

As I continued through the book, a second and important point was brought up. Refactoring requires reasonable amounts of unit testing in order to ensure that you do not accidentally break the system by the redesign. The argument continues that all coding falls into two categories. One is creating new functionality and the other is refactoring. When creating the initial code, you need testing around it, which connects into the TDD. The point is to be careful with refactors because they can cause instability, and to work slowly in the refactoring. They talk about constantly hitting the compile button just to be sure you have not screwed something up.

Sometimes, it is comforting to know that I am not the only one who runs into some types of trouble. One of my favorite statements in the book is in chapter 7 where the author notes after 10+ years of OOP, he still gets the placement of responsibilities in classes wrong. This to me is perhaps the entire key to refactoring. The point is, we as humans are flawed, even if we are all flawed in unique ways; it is something we have in common. This book, in my opinion is not ultimately how to correctly design, but how to deal with the mess that you will inevitably make. This is so important I think it bears repeating; this book says that no one is smart enough to build the correct software the first time.  Instead, it says that you must bravely push on with your current understanding and then, when you finally do have something of a concept of what you are doing, go back and refactor the work that you just created. That is hard to put into practice, since you just got it working, and now you have to go tear it up and build again. The beauty of the refactor is that you don’t have to go it alone, you have techniques that allow you to make wise choices on what the change and the changes should have no affect on the final product.

One final thing I think is worth mentioning. The way the book is laid out, you can easily re-visit any given refactoring technique that you didn’t “get” the first time, as it is fairly rationally structured, grouping refactors together but keeping each refactor individualized. It makes me wonder how many times they refactored the book before they felt they had it right?