Showing posts with label TDD. Show all posts
Showing posts with label TDD. Show all posts

Thursday, April 9, 2015

The Future of Load Testing

First of all, I won't pretend I actually know what the future of Load Testing will look like, but I want to describe some of the different ideas I have seen and done.  Some of these things I have not seen or heard of by anyone else, certainly not on the internet.  So hopefully these will expand your thinking around Load Testing.

What is the Purpose of Load Testing?


Functional testing is designed to exercise the functionality of the system.  Frequently people talk about using Selenium or QTP to exercise a particular piece of functionality, often in the UI.  With the Test Pyramid it is suggested that these sorts of tests should try to hit below the UI level for various reasons.  No matter, if you test an API, a UI, Console App or some other hook below the UI, if your concern is around the Testing Pyramid, it is likely you are trying to test the functionality.  You are often interested in large sets of behaviors and the way the system responds.  When you do Load Testing, in most cases, few of the broad tests are as important.  You are not trying to see if the system will handle all the corner cases and often Load Tests don't even check anything besides a response code.  The raison d'etre for a Load Test on the other hand often has less to do with how the system functions but in ways the system reacts to many different inputs occurring in a relatively quick fashion.  Granted some Load Tests are less about the number of inputs and more about the style of the input (E.G. large files) or other types of constraints (e.g. with less RAM).  To sum it up in a general statement, Wikipedia describes it this way:
Load testing is the process of putting demand on a system or device and measuring its response. - http://en.wikipedia.org/wiki/Load_testing
However, the majority of Load Tests simply want to understand how many inputs a system can handle given a certain profile of inputs.

All of that sounds rather abstract, but if you go reading Microsoft's very hand guide on types of performance tests, you will see that the underlying purpose of this sort of testing varies.  They use terms like load test, performance test and stress test for different meanings.  I think all of this data is really useful and valid.  I however, am going to use Load Test (in capitals) to describe any number of different purposes.  Instead, what I want to look at is what sorts of ways we can organize our testing to make for a better long term experience.  These ideas could be applied to many of the various purposes of Load Testing, so assuming you understand your Load Testing goals you can tailor these ideas to your organization.

35-50% of the Internet Traffic


The first idea I think worth exploring is the Netflix model of Load Testing.  Effectively, trying to have a production-like QA system is silly for Netflix because that would be like having a second set of the internet for QA.  In fact, if you were like Netflix, you would then have to have an insane number of additional systems to generate a load anything like your customers...  or you could just have production traffic mirrored between the two systems.  Having a second prod-like environment is of course not going to work.  So they came up with a radical set of strategies, but I think this sums it up nicely:

We have found that the best defense against major unexpected failures is to fail often. By frequently causing failures, we force our services to be built in a way that is more resilient.  - http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html
The best method they found was basically to attack production and see how it would respond.  This however still leaves the question of how do they specifically test their code will handle new loads.  While they do some Load Testing, their biggest defenses are the scaled down production traffic mirrors they run, the ability to go back to old code and the fact that AWS allows you to spin up new instances all the time.  In effect, they turn the problem on its head, but this really only is helpful if you can use AWS to grow quickly and if your traffic is relatively uniform.  Also, at Netflix scale, you can hire a lot of engineering talent to build this up.


The Limits of Load Test Systems


When trying to leverage user traffic doesn't work, you have to start looking for other options.  Using something like JMeter is interesting.  I call out JMeter because that is what Netflix uses above and beyond their production traffic mirror.  A tool like JMeter takes traffic from a proxy and feeds it into a script (I guess you could hand code it if you are crazy).  Then you edit the script and parameterize it.  You run the script over and over again using multiple threads and try to create load.   These tools might instrument the systems under test or you might have to do that yourself.  In either case, the data gathered is outputted and left for some poor soul to try to understand.  After having been in this position several times, let me say that it truly is difficult to understand these results, in particular because you had to use record-playback from a proxy.  Just as it is a bad idea to use record-playback in automating your functional tests, I think it is a bad idea to do so with such tools.

In one of my former companies, they had one specialist whose only job was the deal with these proxy-recorded scripts.  They are a mess and I'm not going to pretend I know how to fix them.  However, I do have a few ideas and all of them involve your already created friends, the functional tests.

Functional Tests == Load Tests: With or Without UI

When you have a functional test, you might have a complex setup and tear down.  However, the test itself is often fairly simple.  There are two major ways you can create functional tests, one is using a UI and the other is to hit just below the UI, perhaps at an API level.  So logically you can do two things to create load.  One is to scale up your UI tests.  I have seen this done and know of others who have tried to do it.  It is a fairly big engineering feat to create a Load Test using Selenium, but I know it can be done.  Be warned, this can be very expensive, as it requires one OS per hand full of threads you want to run plus overhead for selenium hub nodes.  The other option is to use your existing API tests and create a Load Test overtop of that.  You might have to simplify the data creation and you might have to remove the validation if those take too long, but this is a very easy method of Load Testing the system.  I have personally built several Load Tests around this idea.

Now we have talked for years about how functional tests should be run in a CI environment.  If you are writing your Load Tests like you write your functional tests, using the same systems, then why not run your Load Tests nightly?  Obviously there are some questions you want to ask up front, like if someone will be alerted because of it or what impact that has on your functional tests.  Another question you might consider is what sort of load do you want?  If you need to actively watch to make sure the system stays up, a nightly Load Test would not make much sense.  On the other hand, what if you did a small load for a short period of time?  You could capture the resulting data and plot it on a chart.  This way as you gathered more nightly data, you would have a rough understanding of what you expect.  Now, you aren't running a Load Test once a sprint with little idea of what changes might cause impact, with specialized scripts that take a lot of effort to maintain.  Instead, you have a trend line and will notice changes.  Now this won't tell you when you will fall over or some other data points, but it does give you a change detector.  Furthermore, when you have to fix your functional test, your fix automatically goes into the Load Test.

The next piece is you can run multiple 'threads' at once, not all of which are load related.  If your Load Test can't do validation, while running the Load Test you can run some functional tests to see if the system still appears to function.  Since this is a job in your CI manager, it should be easy to kick off.  You can even manually test while your Load Test runs.

Eventually, you might realize that your functional and load tests only vary in how much load there is and how complex the setup/validation is.  You might realize, like the Load Test, you can metrickitize the functional tests so that you see if a functional test start taking longer.

Load Test?  What is that?

If you notice my description, my efforts are to make the code base easier to maintain and have as much data as possible.  This ultimately makes your Load Testing efforts and your functional efforts look very similar.  They live in the same code base, call the same functions, effectively the concepts merge.  The only differences are in the design, setup, cleanup and how heavy the validation is.  I do think there is value in having different words to describe the intent, but merging the code allows you to get more done with less resources.  One of the biggest advantages I have personally seen is that I actually understand what the test does, where as when I was using tools like JMeter, I often had no idea how it worked.  I have learned a ton about HTTP and HTTPS just by building my own tools.  Now not everyone has time for that and I think ultimately we will want tooling to help make this easier.  However, I am not sure if the costs of having a different tech stack and code base is worth the value the current tools provide, so you might have to make your own for now.

If you have found your Load Testing tools are working well today, then feel free to ignore this.  I know not everyone has the same needs as we have.  I know that the tools we have today do serve some purposes, but my experience hints that they frequently are as much trouble as the value they add.

"In the Year 2000"

- Conan O'Brien, et al.

In trying to predict the future it is really difficult to say what will or will not happen.  Conan O'Brien has been predicting what will happen in the year 2000 for more than 15 years, but unlike him, I have not had the benefit of seeing the future.  With that said, I suspect that in the future we will see more machine learning style systems that take our Load Test data and create load profiles based upon real time data.  Such a system would be able to adaptively adjust based upon metrics in the systems under test and would also detect what changes happened and what code appears to have caused a particular slowdown.  It will correlate this and help find what is causing systems to fail.

I also suspect that we will have a better set of load profiles that can push or break systems.  Load Test systems of the future might go through those profiles on a daily or build-basis and inform you when a build/day has uncharacteristic results, again based upon some form of machine learning.  It will start looking a lot more like functional tests which you only examine when something strange occurs or when you are auditing your tests.

Obviously all of this takes some fair amount of work and effort to produce.  We presently don't have the tooling to do this and while bits and pieces have been worked at, I have heard of no one actually doing this.

What sorts of things would you like to see in future load testing systems?  What areas have you struggled with?  I have found very little experiential data around load testing.  I'd love to hear from others on this topic, even if you just dump a link to your blog post.

Monday, December 1, 2014

Shorts: Ideas described in less words; Code Coverage, Metrics, Reading and More

Most times I write more 'essay' style articles, but Isaac and I have sometimes had small ideas we wanted to discuss but didn't feel like they were big enough to post on their own.  So I'm trying this out, with a series of small short ideas that might be valuable but are not too detailed.  Please feel free to comment on any of these shorts or on the idea of these smaller, less essay-style posts.  If you are really excited about a topic and ask interesting questions, I might try to follow it up with another essay-style post.

Code Coverage


Starting with a quote:
Recently my employer Rapita Systems released a tool demo in the form of a modified game of Tetris. Unlike "normal" Tetris, the goal is not to get a high score by clearing blocks, but rather to get a high code coverage score. To get the perfect score, you have to cause every part of the game's source code to execute. When a statement or a function executes during a test, we say it is "covered" by that test. - http://blog.jwhitham.org/2014/10/its-hard-to-test-software-even-simple.html

The interesting thing here is the idea of having manual testing linked to code coverage.  While there are lots of different forms of coverage, and coverage has limits, I think it is an interesting way of looking at code coverage.  In particular, it seems like it might be interesting if it was ever integrated into a larger exploratory model.  Have I at least touched all the code changes since the last build?  Am I exploring the right areas?  Does my coverage line up with the unit test coverage and between the two what are we missing?  This sort of tool would be interesting for that sort of query.  Granted you wouldn't know if you covered all likely scenarios, much less do complete testing (which is impossible), but more knowledge in this case feels better than not knowing.  At the very least, this knowledge allows action, where as just plain code coverage from unit tests as a metric isn't often used in an actionable way.

I wonder if anyone has done this sort of testing?  Did it work?  If you've tried this, please post a comment!

 Mario’s Minus World


Do you recall playing Super Mario Brothers on the Nintendo Entertainment System ("NES")? Those of you who do will have more appreciation for this short but I will try to make it clear to all.  Super Mario Bros, which made Nintendo's well known series of Mario games famous.  In it you are a character who has to travel through a series of 2D planes with obstacles including bricks and turtles that bite, all to save a princess.  What is interesting is that fans have in fact cataloged a large list of bugs for a game that came out in 1985.  Even more interesting, I recall trying to recreate one of the bugs back in my childhood, which is perhaps the most famous bug in gaming history.  It's known as the minus world bug.  The funny thing is in some of these cases, if a tester had found these bugs and they had been fixed and the tester would have tested out value rather than adding value in, at least for most customers.  I am not saying that we as testers should ignore bugs, but rather one man's bug can in some cases be another man's feature.

How Little We Read


I try not to talk much about the blog in a blog post (as it is rather meta) or post stats, but I do actually find them interesting.  They to some degree give me insight into what other testers care about.  My most read blog post was about the Software Test World Cup, with second place going to my book review of Exploratory Software Testing.  The STWC got roughly 750 hits and EST got roughly 450 hits.  Almost every one has heard of Lessons Learned in Software Testing (sometimes called "the blue test book").  It is a masterpiece made by Cem Kaner, James Bach and Bret Pettichord back in 2002.  I just happened upon a stat that made me sad to see how little we actually read books.  According to Bret Pettichord, "Lessons Learned in Software Testing...has sold 28,000 copies (2/09)".  28,000 copies?!?  In 7 years?!?  While the following assertion is not fully accurate and perhaps not fair, that means there is roughly 30k actively involved testers who consider the context drive approach.  That means my blog post which had the most hits saw roughly 3% of those testers.  Yes, the years are off, yes I don't know if those purchased books were read or if they were bought by companies and read by multiple people.  Lots of unknowns.  Still, that surprised me.  So, to the few testers I do reach, when was the last time you read a test related book?  When are you going to go read another?  Are books dead?

 Metrics: People Are Complicated

Metrics are useful little buggers.  Humans like them.  I've been listening to Alan and Brent in the podcast AB Testing, and they have some firm opinions on how it is important to measure users who don't know they are (or how they are) being measured.  I also just read Jeff Atwood's post about how little we read when we do read (see my above short).  Part of that appears to be those who want to contribute are so excited to get involved (or in a pessimistic view, spew out our ideology) that they fail to actually read what was written.  In Jeff Atwood's article, he points to a page that only exists in the Internet Archives, but had a interesting little quote.  For some context, the post was about a site meant to create a community of forum-style users, using points to encourage those users to write about various topics.

Members without any pre-existing friends on the site had little chance to earn points unless they literally campaigned for them in the comments, encouraging point whoring. Members with lots of friends on the site sat in unimpeachable positions on the scoreboards, encouraging elitism. People became stressed out that they were not earning enough points, and became frustrated because they had no direct control over their scores.
How is it that a metric, even a metric as meaningless as score stressed someone out?  Alan and Brent also talked about gamers buying games just to get Xbox gamer points, spending real money to earn points that don't matter.  Can that happen with more 'invisible metrics' or 'opaque metrics'?  When I try to help my grandmother on the phone deal with Netflix, the fact that they are running 300 AB tests.  What she sees and what I see sometimes varies to my frustration.  Maybe it is the AB testing or maybe it is just a language and training barrier (E.G. Is that flat square with text in it a button or just style or a flyout or a drop down?).

Worse yet, for those who measure, these AB tests don't explain why one is preferred over another.  Instead, that is a story we have to develop afterwards to explain the numbers.  In fact, I just told you several stories about how metrics are misused, but those stories were at least in part told by numbers.  In more theoretical grounds, let us consider a scenario.  Is it only mobile users who are expert level liked the B scenario while users of tablets and computers prefer A?  Assuming you have enough data, you have to ask, 'does your data even show that'?  Knowing the device is easy in comparison, but which devices count as tablets?  How do you know someone is an expert?  Worse yet, what if two people share an account, which one is the expert?  Even if you provide sub-accounts (as Netflix does), not everyone uses one and not consistently.  I'm not saying to ignore the metrics, just know that statistics are at best a proxy for the user's experience.

Monday, November 10, 2014

James Bach and Dr. James Whittaker: An attempt to understand viewpoints


Introduction, Intentions and Limits


I don't have the political ties to many of the big thinkers of the industry.  I have seen James Bach speak, had him address my blog, addressed his blog, read his book and I have even asked him a question at CAST, but I don't know him personally.  I have attempted to address Dr. Whittaker in his blog, but have never got a response and I have never meet him.  I have read Dr. Whittaker's (in)famous exploratory testing book, which is currently the second most popular posts I have written.  I take that to mean I am not the only one who has interest in Whittaker's views and I suspect I'm not the only one to never have met him.  In my consideration of both these individuals, I am not attempting to make it personal.  As you read this, please keep in mind that this is my attempt to understand their viewpoints.  This is an attempt to parse the words of each of them to gain insights.

I came from CAST a few months ago and while I'm not going to directly talk about CAST I want to talk about a subject near and dear to my heart. In listening to Jame Bach's talk, most of which was around the process of testing, but one piece was clearly controversial. This was the Testing vs Checking and the ideas around it. In talking with Bach, I now know I hold disagreements with some of his model. I mostly kept silent regarding this, except for my consideration of Heuristics and Algorithms where I pondered if what humans and computers do is equal, it just happens human are complex and unknown currently compared to that of a computer. This leads down the free will question that is unanswerable, which is why I didn't deep dive into that subject. I still am resisting that particular topic for now, but I may return to it on a later date.

Bach has interesting ways of subdividing his world, with the terms Testing and Checking, with the word Test meaning something different in the past, even for Bach. Some day I will do a word of the week on Test, but that isn't today's subject either. In fact I'm still dodging giving my detailed, full opinion on checking vs testing, but be assured it will come up. Today's topic actually is about my studies of James Whittaker and what appears in my view as James Bach's answer to that viewpoint. I choose these two in part because I was interested in why Whittaker left testing and for Bach's part, he is a prolific exemplar of a testing professional for me to consider.  Bach also has address some areas that Whittaker has talked about, making him a easy to cite exemplar as well. This is not meant to be name calling, but rather I am trying to describe how I see these thinkers in test hold positions and viewpoints that are deeply held by many individuals. Certainly other people have variations of these viewpoints and certainly I maybe miss-reading parts of these viewpoints. I struggle not to put words in people's mouths, but I have to admit some of this is speculation based upon the comments I've found that Whittaker has made between 2008ish and 2014. Those are limited and I have studied them over months, so my sourcing is limited. James Bach, on the other hand, is prolific, which means I might be missing data. With that said, on with the show!

Warning: This is a long winded personal discovery post, you have been warned.

 What Dr. Whittaker has said on Testing


In .NET Rocks 408 from 2009, Whittaker complained about only having guiding practices, with no solid knowledge, saying,
“I'll just be happy when testing works... because they have guiding practices, guiding theory....”   
Later he noted how more testers means, in his words, lazy developers:
“We claim in many groups a 1 : 1 dev/testing ratio [in Microsoft]... Google [has]... 1:10 tester ratio... 1:1, you think oh wow, that is a lot of testing, also a lot of lazy developers to be perfectly honest... At Google they have no choice but to be a little more careful in development.”
Whittaker wants to remove as much inventive thinking from the process as possible, saying things like “In Visual Studio 2010... much of that [change analysis] we will automate completely..." and
“What can you tell me about the new tools... we think you will be able to reproduce every bug.”  That is a fairly wide and amazing claim.  He even starts talking about intelligent code that will find bugs all by itself,  “Only if it is the right sort of unit testing. If it isn't finding actionable bugs... throw it out. If the test cases themselves were feature aware, they know, they know what bugs they are good at finding... this is the future I want.”  Why do you need to inject intelligent reasoning when all problems have been solved before, but are hard-coded to one particular program?  “I'm convinced, on this planet, everything that can be tested, has been tested.”  In fact, why not remove testing as an activity all together?  Just have the machine figure out what needs to be tested, even before you hit compile!  “We[test] need to be invited in to be a full fledged member... we have to get to the point where we [test] are contributing more. Can we get software testing to be more like spell check is in word.... this needs to be the focus of software testers. These late cycle heroics and these testers who are married to late cycle heroes are Christmas Tester Past. ”

Dr. Whittaker imagines a future where testers are more-or-less developers. They might have a few test skills but basically once the magic tools exist, then developers should be solving these problems.  But of course he might have changed his mind since then?  I mean that was in 2009!  What about modern times?

Quality is no less important, of course, but achieving it requires a different focus than in the past. Hiring a bunch of testers will ensure that you need those testers. Testers are a crutch. A self-fulfilling prophesy. The more you hire the more you will need. Testing is much less an actual role now as it is an activity that has blended into the other activities developers perform every day. You can continue to test like its 1999, but why would you? - March 25, 2014, http://www.stickyminds.com/interview/leadership-and-career-success-purpose-interview-james-whittaker
Nope, Dr. Whittaker's viewpoint has gotten even more extreme. Dr. Whittaker might say I am a developer given the 'advanced' automation I write, but he certainly thinks most testers should just go away. He would rather have testers all fired, because [manual?] testers just enable developers to write bad code.  Why does he feel this way? Well first of all, he's in a super-large company that hires a great deal of the best talent there is.  Microsoft has some amazing developers and clearly they can write some amazing code, so maybe he sees them and assumes everyone else is like that.  Before he was at Microsoft, he worked for Google, so his perspective might be skewed.  Perhaps Microsoft and Google can get away with fewer or no testers.  Facebook seems to do it. He certainly seems to hint at it with his most recent blog post on the subject.  His infamous 'traditional test is dead' talk also seems to indicate that you need enough people to dog food your product, which requires a large community of people willing to file bugs for you. That does not apply to everyone, but he might not realize that?  Dr. Whittaker's history has, as best I can tell, been mostly about bigger companies and teaching. Perhaps there are other reasons I have not been able to discover.  I'd love to hear from Dr. Whittaker!  Clearly James Bach, has a reply...

James Bach and Automation


James Bach, a man who consults with and trains hundreds (or more?) of testers each year also has an interesting vantage point.  He mostly works on testing from an intelligent gathering of information through investigation.  He argues, in essence that any investigation made by a programmer in order to write automated tests (i.e. "checks") or executing the automated tests is in fact not equal to that of a human being.  All of Dr. Whittaker's magic tools make less sense in the mind of Bach.  Not that Bach doesn't want these tools, but he would argue without an intelligent human, the tool provides no additional value.  He has gone so far as to start labeling automated testing as checking. Not saying they have no value, but saying that their scope is limited compared to a human. I think Bach would even argue an AI-oriented test method, load testing, model driven testing and other "beyond human" testing could be done by a tester, given infinite amounts of time and/or humans. To be clear, James Bach seems to not view it as possible that there are tests that are beyond human, but I will address this later.  Therefore, I could simplify the argument to this:

if(Human.Testing > Computer.Testing) then throw new DoNotOverloadWord("Computer Testing and Human Testing are not the same!");

Ironically, Dr. Whittaker might agree with that simplification if only you change the ">" to "<".  While I might agree with James Bach in his basic point, I think that there are a few concerns.  One is that trying to change a word used in many different cultures, not just software testing to something new just makes another standard.  He's just added another definition to an already complicated word, making it even harder to understand. Worse yet, it seems like in defining the word that way, Bach has accidentally made his own standard for the word test, which will compete with other standards. I don't know if Bach wants to be a sort of defacto-ISO standard or not, but that's how it might become.  I certainly sometimes see how his passionate arguments could be perceived as an attempt to be sort of standard.  On the other hand, I also hate the answer 'it depends on context' (without something to back it up), so at least when I read Bach's blogs, I know what he means when he says checking.  Dr. Whittaker doesn't seem to put in nearly as much rigor in his words, making his ideas a little more slippery.  I don't know that there is an ideal way of handling this problem.  Perhaps that is part of why schools of testing showed up, so you know whose standard definitions you use.

When you say "Test" which definition are you using?  The Bachian 2014 definition.  Oh, you've not moved on to the 2015 definition?  No, it's out now?

Back to the question of computers and if they can go beyond humans in testing/checking.  I take it that Bach feels that computer's ability to test/check cannot go beyond people from this graphic:


Machine Checking is inside Testing and James Bach defines Testing as:
Testing is the process of evaluating a product by learning about it through exploration and experimentation, which includes to some degree: questioning, study, modeling, observation, inference, etc. 
So if you run a load test, the load that the machine generates allows you to observe the system under load, using machine generated metrics like page load times or database CPU usage. The question is where does the load test code belong?  Is it 'checking' in that it navigates the pages?  I suppose it does blow up if it can't load the page, which is a check, but the primary point of load tests is not about the functions of the page as much as the non-functional requirements. Is the data gathered by the load test a model?  If I have a graph of users to page load times, isn't that a model... created by a machine?  Bach does not seem to have a reply to this, other than to say that a machine embodies its script and thus is separate from humans. Whittaker on the other hand does not care as much about the distinction as he does about getting the technology good enough to get the data into engineers hands without a middleman tester writing a specialized load test script. Better yet, in Whittaker's mind, get your users to do the testing by deploying the code.




Another issue is that while the test executes, it is limited to the data and code written, even if you have a genetic algorithm for testing and use Google as a datasource.  These are powerful algorithms and Google as a datasource outmatches my own knowledge base in a sense. While I agree this is perhaps different than a human who is perhaps in some way more advanced than said algorithms, it can also be beyond human.  Perhaps because humans can reason and the algorithms are limited to the code written there is a distinction to be made, which is fair enough, even if I still am not 100% convinced.  Is reasoning just another term for 'a process we don't fully understand' while code is something we can?  Is 'tacit knowledge' and 'cultural understand' really limited to just humans or will computers with enough data be able to generate similar models some day?  I admit human brains seem different, maybe we have a soul, maybe we have free will, maybe....  But to be fair, even with my imagined super-advanced testing system, I know 98% of automated testing systems are not attempting to approach these sorts of advancements.  Even if our automation systems aren't nearly as good as I think they can be, does that mean that testing vs checking will always be a dichotomy or will programs eventually get good enough that we can't tell the difference?  And what if humans are always special and better, what about diminishing returns? Hand-washed clothing maybe cleaner and possibly more water efficient, but a washing machine is preferred by nearly everyone in the modern world.  Perhaps all of this doesn't matter right now, but Dr. Whittaker seems to think we're there.  Maybe places like Microsoft are there?  Maybe "Tester" as a role makes less sense as more of that role is automated away, and maybe when you throw enough developers to create the automation, you can have most of your testing automated?

So then there is another question that comes to mind, if executing a "automated test" is checking, while the human is writing the algorithm to do the 'checking', is that testing?  In speaking with Bach, it is not in his mind.  My brain, thinking about the problem, writing code, creating models in my head about the system under 'test' (system under check?) is not testing according to Bach.  Very interesting viewpoint, making my claim of about this site being 98% about testing untrue in Bach's viewpoint.  Dr. Whittaker would seem to say that of course that is testing, even if I personally shouldn't be doing it or at least I should also be writing production code.

To be fair, Dr. Whittaker also argues that users are better than machines or testers.  This is a point both Whittaker and Bach might agree on, even if Bach might say only in some context.  But, in saying that I think Dr. Whittaker loses the point that users of a product can change products quickly and are fickle.  The mighty IE 6 failed and clearly IE is no longer top dog.  Listening to users might have been part of that, but an insider advocating for the users also might have been helpful! Users often don't know what they want and what they want changes quickly. This has known this for years. What makes Dr. Whittaker think that all products have a single user base that only wants one set of features? One man's bug is another man's feature.  For that matter, what about products that don't have a exploitable user base?  How do you dog food software for shipping products (OMS) when it costs money every minute the software is broken?  How about we dog food a pace maker; that will work, right?  James Bach would note that quality is not a thing you can build into software but rather a sort of relationship.  I suspect that if you break the software too often, you harm the user's interest in that software and they will move on.

What can I see from here?


I very much appreciate that these giants of the industry allow me to stand on their shoulders and look out upon the sea of possibilities.  I don't know exactly where I stand in an 'official' sense, but I am sure that both of them hold good ideas, ideas that should be investigated... nay, should be tested.  To his credit, James Bach agrees and wants to continue to make his ideas better.  The problem is that with different context there are different needs.  I have pondered from time to time if we could lock down context (e.g. E-Commerce website, X-Y income, X-Y products sold, Z types of browsers), could we say something about which style of testing would make more sense?  In considering both arguments, I have to say that in my experience, automation can go beyond what a human can test, but the value provided often is different than that provided by manual testing.  On the other hand, often the results require human analysis, meaning that human judgment is still required. Like a load test that knocks over a system after throwing 500% of normal load, a human is required to do the analysis and render a value judgment.  I having also seen some development shops that have gone without testers. At least in one case, they seemed to have almost pulled it off in a superficial view, but once you dig in, I think it became clear that some people or places do need testers.  I don't think there is a way of giving detailed advice that is universal, and without some well controlled studies, it maybe impossible to tell when a particular methodology is superior in a given context.

While I can't speak to what method is the best or which are superior to others, I can speak to what worked and what didn't for me in my context. Using only my own contexts and what I have seen I can make a few broad statements.  Few people do automation well.  The most successful automation does not involve a UI or when it does, it is in a manumated process. Those who do automation well are treasures.  Few, but more people do manual testing well.  These people are also treasures.  If you have only bench testing by the developers, adding either automated or manual testing (or checking) is likely to increase the quality for the product in small to medium size organizations.  I am not sure how to logically organize test/development for large organizations (more than 500 people in software development).

Not everything is easy to be tested by code.  Not everything is easy to test by people.  If you can't get people to do the testing, either it might not matter enough or you should automate it and accept the limits of checking.  If your automation keeps breaking either your software is flaky or perhaps it shouldn't be automating (yet, if ever).  Testing is hard and you need people who understand testing, be they developers or testers.  For that matter, you also need people to understand software development, the business, the customers and probably the accounting.  How you label these people may affect how they do the job, and maybe it matters, but what matters most is that the job is done as much as it needs to be done.

Ultimately, software development is hard.  Dr. Whittaker and James Bach come at the problems from very different places, but both want to improve the quality of software.  Even with their extremely different views both have contributed towards their goals.  These extremes do cause some polarization and sadly this post might sound negative towards both their views, but it is hard to find common ground between two extreme views. On the other hand, Whittaker helped make Visual Studio 2008 and 2010 better at testing while Bach has been working on perfecting RST!  Both these men have worked towards their visions and have created useful artifacts from their near disparate visions. While I have disagreements with both sides, I am glad they were around making my job of testing better and easier.

So if I haven't said it yet, thank you.

One last thing.  If Dr. James Whittaker or James Bach respond to this with any corrections, I will update this post.  I maybe challenging my understanding of their ideas but I am not trying to put words in their mouths.

Monday, October 21, 2013

The case for an Automation Developer

Disclaimers: As is always true, context matters. Your organization or needs may vary. This is only based upon my experience in the hardware, B2B, ecommerce and financial industries. Given the number of types of businesses and situations, I assume you can either translate this to your situation or see how it doesn't translate to your situation.

Automation


Automation within this context is long living, long term testing somewhere between vertical integration testing (e.g., Unit testing including all layers) and system testing (including load testing).  These activities include some or all of the following activities:
  • Writing database queries to get system data and validate results.
  • Writing if statements to add logic, about things like the results or changing the activities upon environment.
  • Creating complex analysis of results such as reporting those to an external system, rerunning failed tests, assigning like reasons for failure, etc.
  • Capturing data about the system state when a failure occurs, such as introspection of the logs to detect what happened in the system.
  • Providing feedback to the manual testers or build engineers in regards to the stability of the build, areas to investigate manual, etc.
  • Documenting the state of automation, including what has been automated and what hasn't been.
  • Creating complex datasets to test many variations of the system, investigating the internals of the system to see what areas can or should be automated.
  • Figuring out what should and shouldn't be automated.
Developer

Developer within this context is the person who can design complex systems.  They need to have a strong grasp on the current technology sets and be able to speak to other developers at roughly the same level.  They need to be able to take very rough high level ideas and translate them into working code.  They should be able to do or speak to some or all of the following activities:
  • Design
  • Understand OOP
    • Organization
  • Database
    • Design
    • Query
  • Refactor
  • Debug
  • Reflections
Automation Developer

You will notice that the two lists are somewhat similar in nature.  I tried to make the first set feel more operational and the second set to be a little more skills based, but in order to do those operations, you really have to have the skills of a developer.  In my experience, you need at least one developer-like person on a team of automators.  If you want automation to work, you have to have someone who can treat the automation as a software development project.  That also of course assumes your automation is in fact a software development project.  Some people only need 10 automated tests, or record-playback is good enough for their work.  For those sorts of organizations, a 'manual' tester (that is to say, a person who has little programming knowledge) is fine for those sorts of needs.

Automation Failures

I know of many stories of automation failure.  Many of the reasons revolve around expectations, leadership and communication.  As that is an issue everywhere I don't want to consider those in too much depth other than to say a person who doesn't understand software development will have a hard time to clearly state what they can or can't do.

Technical reasons for failure involve things as simple as choosing the wrong tool to building the wrong infrastructure.  For example, if you are trying to build an automated test framework, have you got an organized structure defining the different elements and sub elements.  These might be called "categories" and "pages" with multiple pages in a category and multiple web elements in a page.  How you organize the data is important.  Do you save the elements as variables, call getters or embed that in actions in the page?  Do actions in the page return other pages or is the flow more open?  What happens when the flow is changed based upon the user type?  Do you verify that the data made it into the database or just check the screen?  Are those verifications in the page layer or in a database layer?  Organization matters and sticking to that organization or refactoring it as need be is a skill set most testers don't have initially.  This isn't the only technical development skill most testers don't have, but I think it illustrates the idea. Maybe they can learn it, but if you have a team for automation, that team needs a leader.

Real Failure

These sorts of problems I talk about aren't new (Elisabeth Hendrickson from 1998) which is why I hesitate to enumerate the problems with much more detail.  The question is how have we handled such failures as a community?  Like I said, Elisabeth Hendrickson said in 1998 (1998! Seriously!):
Treat test automation like program development: design the architecture, use source control, and look for ways to create reusable functions.
 So if we knew this 15 years ago, then why have we as a community failed to do so?  I have seen suggestions that we should separate the activities into two camps, checking vs testing, with checking being a tool to assist in testing, but not actually testing.  This assumes that automation purely assists because it doesn't have the ability to come up with judgment.  This may be insightful in trying to denote roles, but this doesn't really tell you much about who should do the automating.  CDT doesn't help much, they really only note that it depends on external factors.

When often automation fails or at least seems to have limited value, who can suggest what we should do?  My assertion is that testers typically don't know enough about code to evaluate the situation other than to say "Your software is broken" (as that is what testers do for a living,).  Developers tend to not want to test is typically noted when talking about developers doing testing.  Furthermore, what developer ever intentionally writes a bug (that is to say, we are often blind to our own bugs)?

A Solution?

I want to be clear, this is only one solution, there maybe others which is why the subheading starts with "A".  That being said, I think a mixed approach is reasonable.  What you want is a developer-like person leading the approach, doing the design and enforcing the code reviews.  They 'lead' the project's framework while the testers 'lead' the actual automated tests.  This allows for the best of both worlds.  The Automation Developer is mostly doing code as a software development project while the testers do what they do best, develop tests.  Furthermore, the testers then have buy-in in the project and they know what actually is tested.

Thoughts?

Wednesday, October 16, 2013

Reflections


I have been recently been reading over some of Steve Yegge's old posts and they reminded me of a theme I wanted to cover.  There is a idea we call meta-cognition, that testers often use to defocus and focus, to occasionally come back for air and look for what we might have missed.  It is a important part of our awareness.  We try to literally figure out what we don't know and transfer that into a coherent question(s) or comment(s).  Part of what we do is reflect on the past, using a learned sub-conscious routine and attempt to gather data.

In the same way, programming too has ways of doing this, in some cases, in some frames of reference.  This is the subject I wish to visit upon and consider a few different ways.  In some languages this is called reflections, which uses a reification of typing to introspect on the code.  Other languages allow other styles of the same concept and they call them 'eval' statements.  No matter the name, the basic idea is brilliant.  Some computer languages literally can consider things in a meta sense intelligently.

Reflections

So lets consider an example. Here is the class under consideration:
class HighScore {
 String playerName;
 int score;
 Date created;
 int placement;
 String gameName;
 String levelName;
 //...Who knows what else might belong here.
}

First done poorly in pseudo code, here is a way to inject test variables for HighScore:
function testVariableSetup(HighScore highScore) {
highScore.playerName = RandomString();
highScore.gameName = RandomString();
highScore.levelName = RandomString();
highScore.score = RandomNumber();
highScore.create = RandomDate();
//... I got tired.
}

Now here is a more ideal version:
function testVariableSetup(Object class) {
for each variable in class.Variables {
 if(variable.type == String) then variable.value = RandomString();
 if(variable.type == Number) then variable.value = RandomNumber();
 if(variable.type == Date) then variable.value = RandomDate();
}

Now what happens when you add a new variable to your class?  For that matter, what happens when you have 2 or more classes you need to do this in?  The first version can be applied to anything that has Strings, Dates and Numbers.  Perhaps we are missing some types, like Booleans, but that doesn't take too much effort to get the majority of the simple types.  Once you have that, you only have to pass in a generic object and it will magically set all fields.  Perhaps you want filtering, but that too is just another feature in the method.

The cool thing is, this can also be used to get all the fields without knowing what the fields are. In fact, this one is so simple, I am going to show a real life example, done in Java:

//import java.lang.reflect.Field;
//import java.util.*;

 public static List<String> getFieldNames(Object object) {
  List<String> names = new ArrayList<String>();
  for(Field f : object.getClass().getFields()) {
   f.setAccessible(true);
   names.add(f.getName());
  }
  return names;
 }

 public static Object getFieldValue(String fieldName, Object object) {
  try{
   Field f = object.getClass().getDeclaredField(fieldName);
   f.setAccessible(true);
   return f.get(object);
  }catch (Throwable t) {
   throw new Error(t);
  }
 }

 public static Map<String, Object> getFields(Object object) {
  HashMap<String, Object> map = new HashMap<String, Object>();
  for(String item : getFieldNames(object)) {
   map.put(item, getFieldValue(item, object));
  }
  return map;
 }

Lets first define the term "Field."  This is a Java term for a variable, be it public or private.  In this case, there is code to get all the field names, get any field value and get a map of field names to values. This allows you to write really quick debug strings by simply automatically reflecting any object and spitting out name/value pairs. Furthermore, you can make it so that it would filter out private variables, variables with a name like X or rather than getting fields, use it to get properties rather than variables.  Obviously this can be rather powerful.


Eval

Let me give one other consideration of how reflective like properties can work.  Consider the eval statement, a method of loading in code dynamically.  First, starting with a very simple JavaScript function, let me show you what eval can do:

  var x = 10;
  alert( eval('(x + 2) * 5'));

This would display an alert with the value 60. In fact, an eval can execute any amount of code, including very complex logic. This means you can generate logic using strings rather than hard code it.

While I believe the eval statement is rather slow (in some cases), it can be useful for generating dynamically generated code.  I'm not going to write out an exact example for this, but I want to give you an idea of a problem:

for(int a = 0; a!=10; a++) {
  for(int b = 0; b!=10; b++) {
    //for ... {
      test(X[a][b][...]);
    //}
  }
}

First of all, I do know you could use recursion to deal with this problem. That's actually a hard problem to solve, hard to follow and hard to debug. If you were in a production environment, maybe that would be the way to go for performance reasons, but for testing, performance is often not as critical. Now imagine if you had something that generated dynamic strings? I will again attempt to pseudo code an example:

CreateOpeningsFor(nVariables, startValue, endValue) {
  String opening = "for(a{0} = {1}; a{0}!={2}; a{0}++) {";  
  String open = "";
  for(int i = 0; i!=nVariables; i++) {
    open = open + "\n" + String.Format(opening, i, startValue, endValue);
  }
  return open;
}


eval(CreateOpenFor(5, 0, 10) + CreateFunctionFor("test", "X", 5) + CreateCloseFor(5));
//TODO write CreateFunctionFor, CreateCloseFor...  
//Should look like this: String function = "{0}({1}{2});" String closing = "}";

While I skipped some details, it should be obvious to someone who programs that this can be completed. Is this a great method? Well, it does the same thing as the hard coded method, yet it is dynamically built, thus is easily changed. You can log the created function even and place it in code if you get worried about performance. Then if you need to change it, change the generator instead of the code. I don't believe this solves all problems, but it is a useful way of thinking about code. Another tool in the tool belt.

The next question you might ask is, do I in fact use these techniques? Yes, I find these tools to be invaluable for some problem sets. I have testing code that automatically reflects and generates lots of different values and applies them to various fields or variables. I have used it in debugging. I have used it to create easily filterable dictionaries so I could get all variables with names like X. It makes the inflexibly type system into a helpful systems. I have even used it in creating a simple report database system which used reflections to create insert/update statements where the code field names are the column names in the database. Is it right for every environment? No, of course not, but be warned, as a tool it can feel a little like the proverbial hammer which makes everything look like nails. You just have to keep in mind that it is often a good ad-hoc tool, but not always a production worthy tool without significant design consideration. That being said, after a few uses and misuses, most good automators can learn when to use it and when not to.

Another reasonable question is what other techniques are useful and similar to this? One closely related tool would be regular expressions, a tool I don't wish to go in depth on as I feel like there is a ton of data on it. The other useful technique is known as annotations or attributes. These are used to define meta data about a field, method or class. As I think there is a lot more details to go over, I will try to write a second post on this topic in combinations with reflections as they are powerful together.

Wednesday, October 2, 2013

Refactoring: Improving The Design Of Existing Code

Having read many books on the basics of programming, building up queries and the basics of design, I have found that almost none really talk about how to deal with designing code outside abstract forms. Most books present design in a high level, possibly where a UML is shown and revolves around most of the OOP connections. Some talk over how to use design patterns or create data structures of various types. All seem to be under the illusion that, we as programmers actually can apply abstract concepts into real every day practical methodologies. Almost always they use things like animals or shapes or other "simple" (but not practical) OOP examples. They are always well designed, and seem relatively simple. In spite of that, I think I have learned how to do proper design to some degree by years of trial and error.

Refactoring, on the other hand starts out with a highly simple, yet more real world, movie system, and then slowly but surely unwraps the design over 50 pages. It is one of the most wonderful sets of “this code works, but let us make it better” I have ever seen, and it is dead simple in implementation. The book starts with a few classes, including a method to render to a console. Then blow by blow, the shows differences via a bolding of each change. The book talks design choices like how having a rendering method that contains what most would call “business logic” prevents you from easily having multiple methods of rendering (I.E. Console and html) without duplicate code. They also make a somewhat convincing argument against temporary variables, although I am not 100% convinced. Their reasoning is that it is harder to refactor with temp. variables, but sometimes temp variables (in my opinion) can provide clarity, not to mention they tend to show up in debugger watch windows. To be fair, later on, the author notes that adding back in temporary variables for clarity, but it is not emphasized nearly as much.

As I continued through the book, a second and important point was brought up. Refactoring requires reasonable amounts of unit testing in order to ensure that you do not accidentally break the system by the redesign. The argument continues that all coding falls into two categories. One is creating new functionality and the other is refactoring. When creating the initial code, you need testing around it, which connects into the TDD. The point is to be careful with refactors because they can cause instability, and to work slowly in the refactoring. They talk about constantly hitting the compile button just to be sure you have not screwed something up.

Sometimes, it is comforting to know that I am not the only one who runs into some types of trouble. One of my favorite statements in the book is in chapter 7 where the author notes after 10+ years of OOP, he still gets the placement of responsibilities in classes wrong. This to me is perhaps the entire key to refactoring. The point is, we as humans are flawed, even if we are all flawed in unique ways; it is something we have in common. This book, in my opinion is not ultimately how to correctly design, but how to deal with the mess that you will inevitably make. This is so important I think it bears repeating; this book says that no one is smart enough to build the correct software the first time.  Instead, it says that you must bravely push on with your current understanding and then, when you finally do have something of a concept of what you are doing, go back and refactor the work that you just created. That is hard to put into practice, since you just got it working, and now you have to go tear it up and build again. The beauty of the refactor is that you don’t have to go it alone, you have techniques that allow you to make wise choices on what the change and the changes should have no affect on the final product.

One final thing I think is worth mentioning. The way the book is laid out, you can easily re-visit any given refactoring technique that you didn’t “get” the first time, as it is fairly rationally structured, grouping refactors together but keeping each refactor individualized. It makes me wonder how many times they refactored the book before they felt they had it right?