Showing posts with label Exploratory. Show all posts
Showing posts with label Exploratory. Show all posts

Sunday, March 25, 2018

Code Camp Presentations

I present two hours at code camp earlier this month on the topics of blackbox and whitebox software testing.  The two topic were broken out in a way that anyone could attend one session or the other and get value out of them, but both was better.  Blackbox Testing was the first hour and Whitebox Testing was the second hour.  For those who did not attend my presentation, I should note you maybe missing some context that I gave during my talk but was not recorded in the slides.

One important aspect not in my slides but I told the audience was my blackbox talk was heavily influenced by the BBST courses developed by Cem Kaner and the book Lessons Learned in Software Testing.  So if you have questions, please feel free to bring them up in the comments below.

Blackbox Testing Abstract

Blackbox Testing involves testing software without knowing the internals of the code. This is a survey presentation, covering a broad set of topics meant to expand your interests and provide self-study opportunities. We will cover: 
* Schools of Thought: Where we have been, why testing has changed
* Testing Missions: How to know what you should test 
* Testing Strategies: How to make your testing organized 
* Testing Tactics: How to make your testing better

This will include practical examples as well as theory. You do not need to attend the whitebox session to gain value from this talk. However, these presentations are meant as a pair and will not cover the same material.

Whitebox Testing Abstract

Whitebox Testing involves testing software with deep knowledge of the internals of the code. This is a survey presentation, covering a broad set of topics meant to expand your interests and provide self-study opportunities. We will cover: 
* Schools of Thought: Where we have been, why testing has changed. 
* Limits: Why you need both the light and dark sides. 
* Techniques: Static Analysis, Security Analysis, Unit, Integration and System Testing 
* Tooling to make that blackbox system more white. 
* Automation techniques, including both white and blackbox techniques.

Almost none of the content here will be a repeat of the Blackbox Testing presentation. While it is not required, it is recommended that you attend the Blackbox Testing presentation.

Wednesday, September 23, 2015

Link Mania: 75 Links with Commentary on Testing, Security, Tech and Life

I have a friend who wants to get into security testing and asked for my help in broadening their education.  I assumed that my friend would have a good number of links on security, and so I would try to start with a broader set of testing knowledge.  They have a degree and know basic computer science, but in discussing needs, I decided that my primary goal would be to provide a broader understanding around testing.  I started by emailing some links I thought might be useful... but that quickly became clear that it was less manageable, so over a few months I put together groups of links I thought might be of value into a Google Doc.

I also wrote some notes on why I thought they were of value for my friend.  It should be noted that I cite myself frequently.  This is not because I'm full of myself but because I addressed specific questions my friend had and felt that my research + links I had in my posts would be of some benefit.  I did do some light editing of my notes, removing personal comments for privacy reasons, but this is roughly what I came up with.

One limitation to this sort of document is I could go on forever.  I certainly could have written several dozen posts around these links.  Before I turn you loose to all these links, let me give some advice.  I look in detail for things I need to know soon and skim general topics for data I might value in the future.  It’s a method that allows me to work well with a large set of disparate data.  It’s why I can talk about all sorts of subjects at some level, but am not an expert in any.  Knowing who actually is worth reading helps me filter the 'must reads' from the 'skims'.  So as you look at these links, look for what you want to learn about, and then look at the author.  If the data is useful, look for more of that author, even links I did not suggest. If it wasn't helpful, try a different author.  If you try about 3 different links in a row in this set, and all are not useful, either you are not primed to learn about the subject, or the type of data/authors I have gathered are not optimal for how you learn.  You will have to decide which.

Happy reading!

Links I Came Up With Before I Started Organizing [Mostly Test]



Link

JCD's notes
http://about98percentdone.blogspot.com/2015/01/leveling-up-your-testing-skills.html I literally wrote this for people trying to improve themselves.
http://bbst.info/?page_id=23 Lectures 2,3,5,6 in particular.  They are about 30 mins per lecture.  Great testing stuff.

The lecture has power point slides, which can be downloaded and are useful by themselves, HOWEVER, he talks about different points not just captured in the slides.
http://about98percentdone.blogspot.com/2013/09/where-cdt-fails-rebuttal.html Isaac’s effort to describe what testers should learn in ‘levels’. Good stuff.
http://www.testingreferences.com/software_testing_bookstore.phpThe books I read that they recommend are good, thus I trust their recommendation enough to suggest taking a look at the list.
http://www.developsense.com/blog/ An interesting thinker in the testing space.
http://www.satisfice.com/blog/archives/1346James Bach, one of a hand full of people to affect the modern software testing world greatly with his words.  In one of his various attempt to define testing.  This one is of particular interesting, because it is about the art and act of doing testing.



http://about98percentdone.blogspot.com/2014/06/my-current-test-framework-testing-large.html
Reflections are a mind-expanding technique to make code think about itself.





http://steve-yegge.blogspot.com/2006/10/egomania-itself.html
I really love Steve Yegge, he maybe one of the best writer-programmer combinations I know of.  I come back to his work frequently.  But he is a programmer, and as such, the one that really matters is the top link.  The rest are very programmer centric.
http://blog.codinghorror.com/ Heard of Stack Overflow?  This guy made that (along with Joel Spolsky of Joel On Software, but Joel’s stuff is getting old).  He writes on a variety of topics and on his good days is really good.  That said, he’s still a developer, thinking like a developer.


Research in Life / Happiness / Living / The Mind / General Career Advice



Link

JCD's notes

http://www.ted.com/talks/brene_brown_listening_to_shame?language=en
These have made me think about how I live my life.

http://about98percentdone.blogspot.com/2014/06/what-is-highest-level-of-skill-in.html
Learn about what you should do to make yourself more valuable in life.  Learning about what matters in learning.
http://about98percentdone.blogspot.com/2014/02/being-fraud-and-failure.html How to deal with feeling like you don’t know enough, when in fact you’re driven to know more.

I have referenced this exact blog post more in my comments to other people’s blog entries than any other.  Often the people who care the most feel this way.
http://about98percentdone.blogspot.com/2013/12/book-consideration-introduction-to.html Read the bullet points at the bottom.  In particular, “The Answer”.  The book reviewed is a little bit of a personal and spiritual look at science.  Somewhat like Sagan, but revolved around thinking.  You might like the book.
http://about98percentdone.blogspot.com/2013/09/testing-hiring-process-for-testers.html

http://about98percentdone.blogspot.com/2013/09/a-subject-of-hiring-process.html

http://about98percentdone.blogspot.com/2013/09/my-interviewing-start-and-changes-ive.html
Isaac and I wrote a little bit about getting hired. These are some of my early blog entries, but you also get both the view of a person being hired by Isaac as well as, Isaac, the hiring manager's method of choosing to hire people like me.

https://sites.google.com/site/steveyegge2/miracle-interview
More interview related thoughts, but from a developer side.
https://sites.google.com/site/steveyegge2/age-racecar-driver (same guy as above, but more philosophical questions in specialization.)



http://www.stickyminds.com/article/helpful-tips-hiring-better-testers
The last set of articles Isaac and I have on interviewing.
http://www.moserware.com/2009/01/wetware-refactorings.html Fairly good.  Almost all true.  Interesting ideas and good set of resources.
http://www.moserware.com/2008/03/what-does-it-take-to-become-grandmaster.html I would give this a lower priority because it’s long and this is not the first time I have found someone who says this and put it in the list.  Hopefully you see the patterns… and will learn.
http://breakingsmart.com/An interesting description of the tug between the past and the future.  It seems to be a little dismissive of some problems, but the general picture is not wrong.
http://blog.codinghorror.com/level-5-means-never-having-to-say-youre-sorry/I hate the title, but the content is pretty good. It provides some insights into why generating reactionary, scripted systems does not scale well in creative work.

Paul Graham is fantastically interesting in general.  This is an interesting attempt to correlate technological outcomes to culture.  I recently heard from a friend who lived in Japan for a year about these Japanese workers at this company, whose company was bought out by an American company after it started to fail, had a real difficult time letting go of the quality of a product in order to get the product out the door. That story feels like a sort of quick CRC check for me, meaning this article probably does have some veracity.
http://www.ribbonfarm.com/the-gervais-principle/I know I have talked about this before, but it is an interesting model of human behavior.  While I’m not actually a fan of The Office, I found it was mostly ‘translatable’.  It also offers small notes on the Peter and Dilbert principles which are also worth looking at.


Testing



Link

JCD's notes
https://www.youtube.com/watch?v=j_JviA5nvS0&list=PLSIUOFhnxEiDFckNDSjKWqOCtd8ksJrh4GTAC is fairly good.  Some of the content might be on security. I have not watched all the 2014 (and soon 2015) videos, only the 2013 videos which were often very good.
http://www.associationforsoftwaretesting.org/conference/cast-2015/ CAST 2015 is probably worth watching. I have yet to watch it myself, but I attended last year.
http://www.testingreferences.com/testingnews.phpI use this as a tool, looking for things of interesting, read them, keeping note of who wrote it and if I decide I don’t like someone (for any reason), I mentally filter them out of the list.  It’s the firehose method, good for going in all different places, but you never know what you might get.

How SQL joins work. They are often used both for getting data for testing as well as part of how many reports are generated.
http://angryweasel.com/blog/In broad strokes, I agree with Alan and appreciate his perspective with about 20 years at Microsoft.
http://oredev.org/2013/wed-fri-conference/balancing-atdd-gui-automation-and-exploratory-testing A video I watched long ago that is still in my notes.  Might be interesting from an automation vs exploratory testing perspective.  I have little memory of it though. Most people only teach the basics :(
http://www.huibschoots.nl/wordpress/ Like Alan, I agree with Huib broadly, and I find his view very interesting as he comes from a European background.


General Tech



Link

JCD's notes
https://howdns.works/episodes/ How DNS works.
http://blog.codinghorror.com/on-software-engineering/Why software engineering is so difficult, and why consultants are sometimes looked down upon.
http://www.dreamincode.net/forums/topic/223324-an-interesting-interview-with-steve-yegge-and-james-duncan-about-java/ Interesting set of interviews, the biggest point is that you should know your tools well enough you know what is wrong with them.
http://www.moserware.com/2009/03/how-net-regular-expressions-really-work.htmlI think the data here is interesting, however, the meta is important too, even at a security level.  A DDOS attack is a security attack, but if I can DDOS you with one click, it is a security issue too.  Knowing how regular expressions work tells you about how a lot of systems work inside, which is how samy was able to defeat the defenses of MySpace ( http://namb.la/popular/ ).  It’s also a huge part of getting things done.
http://edge.org/annual-question/what-do-you-think-about-machines-that-think An interesting set of speakers, most very smart, all on the same topic.  The 2014 topic was interesting too. This might generate some interesting questions, such as:


What would happen if/when the #1 job goes away (Driving, at least in the USA; currently biggest economy in the world; last count ~ 3 million souls in the US do this)?  
What do we do with these people?  
What of those that aren’t capable or interested in more mind-oriented work or who are too old to change careers?  
What of my grandmother who doesn’t interact well with online work, in part because her hands are too crippled to do much with keyboards and did not grow up with this world?
What happens when machines categorize someone as an outlaw in some countries which others counties would recognize as moral and legal (such as being gay, or a political dissidents)? Who is legally and morally responsible for a 'thinking' machine?

https://sites.google.com/site/steveyegge2/math-every-day
Here is a mission in life -- how do you solve problems in which there is no solution?  (My only hint is find another problem.)  However, the point is, solving one of these non-trivial problems is a lifetime’s worth of work, if you want it.  See the intro of the second piece to see what I mean.
http://www.amazon.com/review/RUGSCP3XBNBUVSuggested by Yegge’s post… it’s an interesting read in and of itself.  It’s got some ideas I’ve not really researched.


Security



Link

JCD's notes
https://www.youtube.com/watch?v=n9-Gz1U87CI&index=9&list=PLQB4l9iafcelXpJnK6IyDsoFeEb1icqrl Brilliant! I attend this live and you’ll see me in the end asking a question.  All the rest of the talks (I kept the list) are test related, but this is security related.






(Others…)
https://heimdalsecurity.com/blog/best-internet-security-blogs/
Famous security researchers blog.
https://krebsonsecurity.com/category/how-to-break-into-security/ How to learn to be a security professional.
https://www.schneier.com/blog/archives/2013/04/nice_security_m.html Why we are all frauds and failures sometimes (As address in a link above).


A *practical* piece of tech that really applies around security of the web, he has other good posts too.

http://www.guru99.com/learn-penetration-testing.html
I liked the graph, purely from a ‘what do you mean by “security testing”’ question. NOTE: My friend suggested http://www.techrepublic.com/blog/it-security/the-five-phases-of-a-successful-network-penetration/ might also be useful.
https://www.cs.purdue.edu/homes/xyzhang/fall07/Papers/sw-test.pdf Read this a long time ago and it was a little helpful.
http://www.veracode.com/security/software-security-testingHappened upon this, no idea if it is any good. I did not get chance to really review it.
http://blogs.msdn.com/b/oldnewthing/archive/2013/12/24/10484402.aspxMight be interesting, taken from a comment from...
http://blog.codinghorror.com/welcome-to-the-internet-of-compromised-things/here.  Also interesting.
https://www.grc.com/fingerprints.htmThey have some interesting stuff regarding SSL but more importantly they might be a generally useful resource.
https://www.youtube.com/watch?v=wKDE_upBlfcComplex example of a hack and how it can be done by defeating multiple layers and understanding history.


Other



Link

JCD's notes
http://blog.incubaid.com/2012/03/28/the-game-of-distributed-systems-programming-which-level-are-you The technical subject is interesting and it provides insight into architecture. It ALSO gives a nice idea of levels and how you can’t really know what level you are at until you see the next level up. I actually have a more complex theory on this, but I have yet to write the blog post.
http://www.moserware.com/2009/07/just-enough-mba-to-be-programmer.html Just plain useful in a practical sort of way.
http://archive.wired.com/wired/archive/4.12/ffglass_pr.htmlI have not yet completely read it but it was recommended to me by a trusted source.  It’s long.
http://archive.wired.com/wired/archive/8.04/joy.htmlVery long indeed, but worth reading.

You made it to the bottom?  And you counted them?  Only 74 links and you want your money back? Alright! Fine. Well I'm sure I can come up with a 75th link just for you, my bean counting friend.  How about something interesting?  Like really interesting.  Here you go: http://www.damninteresting.com/the-zero-armed-bandit/

Wednesday, July 15, 2015

My Many Years in Testing and Automation Development

I have heard the question expressed in multiple different ways.  "How do I get into QA?"  "How do I progress my career?"  "How do I level up my skill?"  "You read whitepapers?!?"  "What type of automation should I write?"  I have written various articles around this broad range of topics, but I wanted to take a slightly different tact around one of the implied questions.  The implied question I'm considering is how do I continue down the path from where I am to become a better tester and what skills should I spend time working on improving?  Instead of trying to explore the topic in a general sense, I am going to go through a few interesting experiences I have had in both "Testing" and "Automation Development".  While I will be talking about automation, I do not mean to imply that is where you should spend your time learning, just that I will discuss how I made the choice.  I will try to concentrate on interesting stories and lessons learned.

I have been writing code for nearly half my life, but I started working in the development of test automation somewhere between 2004 and 2005.  I started professionally testing in 2002, although I had been writing my own code and testing it earlier.  I also did some testing for others before 2002, but I was not paid for that work.  I will try to describe the companies and context a little, but I will not use company names, to protect the innocent.  But first, a small introduction to perhaps explain how early on I was interested in QA.

When I started my life in QA, I was roughly 6 years old and curious about this clock in a video game called "The Last Ninja" for the NES.  It kept a timer of how long you had been playing the game.  It gave hours, minutes and seconds.  There was a place where no bad guys were with a stream and a park bench.  It was perfect for leaving my ninja to sit and just enjoy the scenery.  In only 99 hours 59 minutes and 59 seconds I could find out what happened when the clock ran out.  I knew Sonic would run off the screen after 3 minutes, so I conducted some smaller tests.  In my smaller tests, I discovered, unlike Sonic, The Last Ninja did not seem to mind waiting for short periods of time, so it seemed easy enough to test the longer case.  The only problem was I didn't want to lose my Nintendo for a week while I waited for the result.  So when I went on vacation, I left my system running.  When I came back, what I found was "0A" and the game was frozen up.  Now, many years later, I know 0A is hex for 10, likely meaning the clock had run over to hex and then the game had crashed.  I didn't understand the result, but I was curious about it.  I was later grounded for leaving my system on for that long.  You've been warned kids!

Years later, I was looking to get a job out of High School.  I had been programming text adventures for some years, but never had a 'real' programming job.  I enjoyed programming, I enjoyed the experience of creating, but still didn't feel like I knew that much.  Still I kept at it.  I had worked at the Post Office for 3 weeks in a Christmas rush season, but had little other professional work under my belt.  I was going to school and studying Computer Science, but that was going to take years and I needed money sooner.  So I started looking for a computer job.  I interviewed once, but didn't know what I was doing and did not get the job.  College and High School don't seriously prepare you for interviewing.  In my second job interview, I took a computerized test and passed all but one question (I tried typing in a search but they expected me to use the dropdown...  No, I'm not still bitter.).

I got the job and started working as a managed contractor for a overly large company who worked on specialized hardware.  I was a tester who did everything from competitive intel to stress testing to environment testing to configuration testing.  However, I had no training and really no idea that testing was a general skill set.  I incidentally learned a few simple QA skills, but nearly all of the learning I did was around social skills in my first year.  Perhaps one the most important skills I learned was to apologize is important and doubly so when you are a manager and have made a mistake.  I moved to a more complex set of testing, but it was a script based.  I learned more about networking and security testing.  I almost got a job in a team that was inspired by James Bach's work around exploratory testing (in one of the places that James and his brother did some of the development around the idea of exploratory testing), but I missed out to someone with a lot more experience.

Then, in a sad set of events a former lead who changed shifts passed away suddenly.  I took his place, an awkward thing for sure.  I learned about this idea that you could 'automate' tests.  I began to apply my existing programming skills in order to test various UI elements.  It worked well in some cases, but we struggled to handle descriptions (a QTP phrase for describing a way of IDing elements in a UI system).  A co-worker of mine had more difficulty picking up the programming as his specialty was in hardware, not software.  He came into work more than once complaining of nightmares around programming.  My coworker did eventually get better at automation, but that was never his strength (Aside: He's now a big-wig manager, likely making way more than me!).

I bounced around from place to place, with a rough total of nine different jobs (not all jobs were title changes).  One thing looking back is how I was always trying to go meta, not just writing good automation but improving the entire process.  In a large company, that can be very difficult.  I ultimately had a conflict with one of the managers and felt the need to leave.  Before I left, I was told to quit trying to improve the organization (by providing useful tools) but instead work on personal projects!  I did and learned SQL and web development while looking for a job.  I also had multiple open source projects under my belt by this point.

If a big company didn't work, I reasoned that a smaller company would be better!  I applied to one company and studied up.  Rather than applying to many companies, I reasoned it was better to be prepared.  This from having done so poorly years ago seemed to work reasonably well.  I got a job offer. I tried to negotiate for more money, but was declined.  I declined the job but was later called back and given a slightly higher offer.  I concluded that because of the recession that a much bigger offer was not going to happen.

I took a job at a 50 person company, and within 2 weeks of me joining the QA manager had quit.  This left the QA group answering to the VP of Engineering and it remained that way for nearly 2 years.  I worked for longer hours at this company than anywhere else.  I lost another co-worker who I was training, and whom I may have been the last person he talked to.  He was hit by a bus.  I learned the 'hit by a bus' scenario is both real and the worry about how the business knowledge loss will be much less impactful than the emotional impact from the loss.  I learned what the term 'death march' meant as I worked 80+ hour work weeks.  I learned how a recession can affect you in spite of the fact that your field is in demand.  I learned the dangers of saying that you won't ship with any [known] bugs.  I learned how working more hours did not equal more productivity.  I wrote nearly 1300 automated UI tests.  I found the most bugs at the company, in part because of the automation I wrote, and because all bugs must be fixed, it must logically be concluded I found the most important ones.  (Yes, that is sarcasm.)  Yet, I was not promoted nor paid more for nearly the entire 3 years I was there.  Only after 50% turn over and the firing of upper management did things change even a little, and by then I was too tired to care.  I left.  In case you want more details, Isaac also detailed his own personal journey at this same company.

Now we are getting into more recent events.  To be perfectly honest, I don't plan on telling anything more about the various company cultures because of how it might affect me.  However, I still learned social and technical lessons.

I learned that 'manual' testers could appreciate automation and automation results.  I, for the first time, got to work with other people with equal or better coding skills than myself as we developed automation.  I got to learn how to play a technical leader, even when I was not promoted into a leadership position nor had any authority over others.  While I had dealt with databases and did a little database testing before, I got to do development in which nearly all the code was  around database usage, including communication between database systems.  For the first time I had to deal with the integration of multiple team's automation efforts which were written in different languages.  By this point in my career I had written test code in C, Java, JavaScript, SQL (various flavors), C#, J#, VB.NET and VB Script.

On the social front, I got to work with both one of the best teams and one of the most difficult teams I ever worked with.  In dealing with the difficult team, I got to see how personality clashes worked and how to work around them.  To be fair, I have my own personality quirks too, but when these hit extremes it is imperative that you figure out how to be a positive force for the team rather than yet another point of contention.  Perhaps one of the biggest lessons I learned was that it is important to have a solid social connection and understanding of an individual before giving a completely honest evaluation of someone's skill set.  In fact, I took the lesson so much to heart that I try to be more careful with my day-to-day speech as a whole.

Finally, I want to describe a little bit of my attempts at professional growth.  I spent a good amount of time embracing various avenues for learning.  I took BBST, and while I only took the class for the first BBST course, I watched all the videos and read all the material.  I have written on this blog for a few years now.  I have spent time giving talks at various venues.  I have written letters, written for other professional venues, testing non-work apps and taken serious amounts of time trying to understand the systems under which work is performed.

So to sum this all up, I think I have just two simple suggestion for you:

Learn

Making mistakes is fine.  Just learn from them.  Learn how you learn.

Don't just do work

We all are making a journey, and at some point it will stop.  Making an impact is important.  But don't make work the only impact in life.

Friday, December 12, 2014

Thinking about No Testers

In attempting to hear out the various viewpoints of those whom subscribe to the idea that Agile should move towards removing the role of tester.  I have yet to see anyone who actually suggested eliminating testing completely, if that is even possible.  So let us unbox this idea and examine it, with as little bias towards keeping my own job as possible.  Here are the rough set of arguments I have seen/heard in TL/DR (Too long, didn't read) form:
  • Limited deploys (to say .01% of the users and deploy to more over time) with data scientists backing it will allow us to get things done faster with more quality.
    • Monitoring makes up for not having testers.
  • Hiring the right team and best developers will make up for the lack of testers.
    • Writing code that "just works" is possible without testing, given loose enough constraints (which leads to the next argument).
  • Since complete testing is impossible, bench testing + unit testing + ... is enough.
  • Quality cannot be tested in.  That is the design/code.
    • Testers make developers lazy.
  • It is just a rename to change the culture so that testers are treated as equals.
    • Testers should be under a dev/team manager, not in a silo.
  • It is a minor role change where testers now work in a more embedded fashion.
  • We hire too many testers who add inefficiencies as we have to wait <N time> before deploying.
    • We only hire testers to do a limit set of activities like security testing.
  • Testers do so many things the name/profession does not mean much of anything.
  • With the web we can deploy every <N time unit>, so if there are bugs we can patch them quickly.
  • As a customer I see those testers as an expensive waste.

That is sure a lot of different reasons, and I'm sure it is not an exhaustive list.  Now I shall enumerate the rough responses I have seen:

  • A little genuine anger and hurt that 'we' are unwanted.
  • A little genuine disgust that 'we' are going through the 80's again.
  • Denial ('this will never work')
  • It can work for some, but probably not all.  (This follows a very CDT 'it depends' point of view)
    • Not everything can easily be instrumented/limited deployed.  For example, I have heard Xbox 360 games cost ~10+k to do an update.
    • In some software, having less bugs is more important than the cost of testing.
  • This is why we need a standard (I never heard this one, but it is an obvious response)
  • Concern about the question of testing as a role vs task.
  • Different customers have different needs they want to fulfill.
  • Good teams create defects too.
  • We need science to back up these claims.
  • Testing should move towards automation to help eliminate the bottlenecks that started this concern.
  • It is very difficult to catch your own mistaken assumptions.
    • If you need someone else to help catch mistakes, why not a specialist?
While I don't have a full set of links I have heard this back and forth viewpoints, but I think it is an interesting subject to consider.  I am going to go through my bulleted list top down, starting with the "no testing" side of the world.  I believe I have seriously considered the limited deployment and letting the tester be the customer in a previous post.  In that same post I noted that at least not all teams can make up for a lack of testers, so again, I think it has been addressed.

Complete Testing is Impossible & Quality Cannot be Tested in


The premise is true, complete testing is in fact impossible, I do agree with that.  The conclusion that hiring testers is not needed is the part I find interesting.  Quality of code and design cannot be tested in is also in interesting statement.  The idea that testers make developers lazy feels very unsubstantiated, with possible correlation-causation issues (and I have seen no studies on the subject), so I am going to leave that part alone for now.

I write software outside of work.  Mostly open source projects, of which I have 5 projects to my name.  Most of them were projects in which I was looking to solve a problem I had and wanted to give to the world.  In more than half of them I wrote no formal tests, no unit tests and to be honest the code quality in some of them is questionable.  There are bugs I know how to work around in some cases, or bugs that never matter to me and thus the quality for me is good enough.  I have had ~10k downloads from my various projects and very few bugs have ever been filed.  Either most people used it and then deleted it or the quality was good enough for them from a free app.  I hired no testers and as the sole dev I was the only one testing the app.  I think this is a +1 for the no tester group, if you work on an open source app for yourself, no tester (maybe) required.  CDT even would agree with this as it does depend.  Context matters.  I am sure other contexts exist, even if I have not seen them.  What I have not seen is specifics around contexts that might require no testers, with the exception of the Microsoft/Google sized companies.

Now can you test code quality in?  If you are about to design the wrong thing and a tester brings that up, did the tester 'test' code quality in?  Well he didn't test any software, the no tester side would say.  Sure, but he did test assumptions, which is often part of testing.  What about the feature the tester notes the competition has that you don't, which was discovered when comparing competitor protect quality?  What if I suppose the dev knows exact what to develop, can the tester add value?  Well I know I have made code suggestions that have gotten a developer thinking.  However, I could have just as easily been an XP developer sitting beside him as he coded rather than a tester.  Well what about TDD?  Is that testing quality in?  If I write tests first, I have to think about my design.  That is testing quality in, but again that is just the developer.  What if the tester and the developer co-design the TDD or the tester recommends additional tests added during development.  Again, this could be a developer, but a tester is often thinking about the problem differently.  So while I think the statement is false, if I look at it from the spirit, I think it might be true in some cases.  There are some really great developers who do think like testers and they could do it.  However, I think that is only some of the developers.  There will always be junior developers and there are always other mind set factors that play in.

 What is in a Name & Quick Deploys?


What's Montague? it is nor hand, nor foot,
Nor arm, nor face, nor any other part
Belonging to a man. O, be some other name!
What's in a name? that which we call a rose
By any other name would smell as sweet;
- Juliet, Romeo and Juliet
Does the fact that we are in a in-group called 'test' and some see developers as an outsider matter?  Visa-versa is probably true too.  In fact, the other day Isaac and I had a small debate about correcting a tester in some very minor minutia about deep mathematical and development theory.  To me, perhaps a developer who was neck deep in building new languages would find such a correction helpful, but a tester who was doing some exploration around the subject didn't need such a correction.  In fact such a correction might make things more confusing to anyone who didn't have a deep understanding of the subject.  I myself am guilty of the insider/outsider mentality.  We all categorize things and use those categories to make choices.  Perhaps renaming us all as human beings would make things better.  If an organization is particularly combative, this might be a useful change.  That being said, there is also a group identity for people who see themselves as testers or QA.  There is a specific field of study around it, and that also adds value.  Also, by having a sub-group, the tension, even if minor, might possibly help people grow.

This is a mushy, non-business oriented view of the world, and perhaps those who are just interested in the how fast things go today would prefer avoiding the conflict in order to get things done.  The fear of doing this from a QA point of view comes from the 1980s, where defects were a big deal.  Renaming people feels like going back to that era.  The argument back is that this isn't the 80s, we can ship software almost instantly.  The argument to this is that some software can't just be shipped with broken code.  Work on many different areas must be 'mostly right' the first time.  Also, this grand new world that people want to move into has not been vetted by time.  Wait a while and see if it is the panacea it is believed to be.  I recognize the value in renaming positions in order to improve team cohesion and believe it comes from a genuine desire to improve.  Just keep in mind what is lost and the risks in doing so.

I should note, I have seen some of this implemented before in organizations I have worked in.  Working under a single development manager, the development manager would often ignore the advice given by the "QA" person.  Perhaps a full rename would have fixed things, but often development wants to move forward with the new shiny thing.  Having no testers is perhaps a new shiny, and so development would be on board.  The people who are more aware of risk of change are the QA and operations people.  This is a generalization, unsubstantiated by anything but my observation, but it feels like QA is now calling this a risk and because it threatens our jobs we are being told we are biased, so we can be ignored.  Psychology and politics are complex things and anything I write will not adequately cover this subject.  I don't object to people trying this, maybe it will work.  I still see it as a risk.  I hope some people take it and report back their observations.

Perhaps the final argument about us is that we are many different roles tied with one name.  This is a complexity that came up at WHOSE.  We talked less about testing than you might expect.  Communication and Math and Project Management and ...  We had a list of about 200 skills a tester might need.  We are a swiss-army knife.  We are mortar for the projects developed, and because every project is different, we have to fill in different cracks for each project.  We happened to do that best in some ways because we happen to see many different areas of a system.  We have to worry about security, the customers who use it, the code behind it, how time works, the business, and lots more.  That is ignoring the testing pieces, like what to test, when to test and how to justify the bugs you find.  Maybe we could remove the justification piece, but sometimes the justification matters even to the developer, because you don't know if it is worth fixing or what fix to choose.  I think we can't help being many different things because what we do is complicated.

Customers Don't Want QA?


While that is a lie in regards to me personally (I get excited when I see QA in the credits of video games), perhaps it is different for others.  I have written elsewhere how users of mobile devices are accepting more defects.  Perhaps that is true in general.  We know there is such a thing as too much QA.  To go to the absurd, when QA bankrupts the company, it is clearly too much QA.  We also know there is too little QA, when the company goes bankrupt from quality issues.  Perhaps we could say when there are regulatory constraints that close the company down because of a lack of QA could also be there.  Those are the edge cases, and like a typical tester, I like starting at boundary cases.  Customers clearly want some form of quality, and as long as testers help enhance quality (E.G. Value to someone who matters), testers are part of that equation.  Are they the only way to achieve quality?  Maybe not, but that involves either re-labeling (see above) or giving the task to others, be they the developers or the customers or someone else.  Some customers like the cheaper products with less quality, and that does exist.  Having looked at the games that exist on mobile devices that are free, my view is the quality is not nearly as good.  Then again, I want testers.  Maybe I'm biased.  Even if I was a developer for a company, I would like to have testers.

Should the ideas of no testers be explored?  I have no objection, but I would say that it should be done with caution and be carefully examined.  Which makes the companies who try it...well, testers.

 Brief Comment on the Community


I have heard a lot of good responses, albeit limited.  I have also seen some anger and frustration.  Sometimes silly things are said in the heat of the moment, likely from both sides.  We also have some cultural elements whom are jagged edged.  These can create an atmosphere of conflict, which can be good, but often just causes shouting without much communication.  One of the problems is most people don't have the time or energy to write long winded detailed posts like this.  We instead use twitter and use small slogans like #NoTesting.  I personally think twitter as a method for communicating is bad for the community, but I acknowledge it exists.  How can we improve the venue of these sorts of debates?  How can we improve our communication?  I don't pretend to have an answer, but maybe the community does.

UPDATE: I wrote a second piece around this topic.

Monday, December 1, 2014

Shorts: Ideas described in less words; Code Coverage, Metrics, Reading and More

Most times I write more 'essay' style articles, but Isaac and I have sometimes had small ideas we wanted to discuss but didn't feel like they were big enough to post on their own.  So I'm trying this out, with a series of small short ideas that might be valuable but are not too detailed.  Please feel free to comment on any of these shorts or on the idea of these smaller, less essay-style posts.  If you are really excited about a topic and ask interesting questions, I might try to follow it up with another essay-style post.

Code Coverage


Starting with a quote:
Recently my employer Rapita Systems released a tool demo in the form of a modified game of Tetris. Unlike "normal" Tetris, the goal is not to get a high score by clearing blocks, but rather to get a high code coverage score. To get the perfect score, you have to cause every part of the game's source code to execute. When a statement or a function executes during a test, we say it is "covered" by that test. - http://blog.jwhitham.org/2014/10/its-hard-to-test-software-even-simple.html

The interesting thing here is the idea of having manual testing linked to code coverage.  While there are lots of different forms of coverage, and coverage has limits, I think it is an interesting way of looking at code coverage.  In particular, it seems like it might be interesting if it was ever integrated into a larger exploratory model.  Have I at least touched all the code changes since the last build?  Am I exploring the right areas?  Does my coverage line up with the unit test coverage and between the two what are we missing?  This sort of tool would be interesting for that sort of query.  Granted you wouldn't know if you covered all likely scenarios, much less do complete testing (which is impossible), but more knowledge in this case feels better than not knowing.  At the very least, this knowledge allows action, where as just plain code coverage from unit tests as a metric isn't often used in an actionable way.

I wonder if anyone has done this sort of testing?  Did it work?  If you've tried this, please post a comment!

 Mario’s Minus World


Do you recall playing Super Mario Brothers on the Nintendo Entertainment System ("NES")? Those of you who do will have more appreciation for this short but I will try to make it clear to all.  Super Mario Bros, which made Nintendo's well known series of Mario games famous.  In it you are a character who has to travel through a series of 2D planes with obstacles including bricks and turtles that bite, all to save a princess.  What is interesting is that fans have in fact cataloged a large list of bugs for a game that came out in 1985.  Even more interesting, I recall trying to recreate one of the bugs back in my childhood, which is perhaps the most famous bug in gaming history.  It's known as the minus world bug.  The funny thing is in some of these cases, if a tester had found these bugs and they had been fixed and the tester would have tested out value rather than adding value in, at least for most customers.  I am not saying that we as testers should ignore bugs, but rather one man's bug can in some cases be another man's feature.

How Little We Read


I try not to talk much about the blog in a blog post (as it is rather meta) or post stats, but I do actually find them interesting.  They to some degree give me insight into what other testers care about.  My most read blog post was about the Software Test World Cup, with second place going to my book review of Exploratory Software Testing.  The STWC got roughly 750 hits and EST got roughly 450 hits.  Almost every one has heard of Lessons Learned in Software Testing (sometimes called "the blue test book").  It is a masterpiece made by Cem Kaner, James Bach and Bret Pettichord back in 2002.  I just happened upon a stat that made me sad to see how little we actually read books.  According to Bret Pettichord, "Lessons Learned in Software Testing...has sold 28,000 copies (2/09)".  28,000 copies?!?  In 7 years?!?  While the following assertion is not fully accurate and perhaps not fair, that means there is roughly 30k actively involved testers who consider the context drive approach.  That means my blog post which had the most hits saw roughly 3% of those testers.  Yes, the years are off, yes I don't know if those purchased books were read or if they were bought by companies and read by multiple people.  Lots of unknowns.  Still, that surprised me.  So, to the few testers I do reach, when was the last time you read a test related book?  When are you going to go read another?  Are books dead?

 Metrics: People Are Complicated

Metrics are useful little buggers.  Humans like them.  I've been listening to Alan and Brent in the podcast AB Testing, and they have some firm opinions on how it is important to measure users who don't know they are (or how they are) being measured.  I also just read Jeff Atwood's post about how little we read when we do read (see my above short).  Part of that appears to be those who want to contribute are so excited to get involved (or in a pessimistic view, spew out our ideology) that they fail to actually read what was written.  In Jeff Atwood's article, he points to a page that only exists in the Internet Archives, but had a interesting little quote.  For some context, the post was about a site meant to create a community of forum-style users, using points to encourage those users to write about various topics.

Members without any pre-existing friends on the site had little chance to earn points unless they literally campaigned for them in the comments, encouraging point whoring. Members with lots of friends on the site sat in unimpeachable positions on the scoreboards, encouraging elitism. People became stressed out that they were not earning enough points, and became frustrated because they had no direct control over their scores.
How is it that a metric, even a metric as meaningless as score stressed someone out?  Alan and Brent also talked about gamers buying games just to get Xbox gamer points, spending real money to earn points that don't matter.  Can that happen with more 'invisible metrics' or 'opaque metrics'?  When I try to help my grandmother on the phone deal with Netflix, the fact that they are running 300 AB tests.  What she sees and what I see sometimes varies to my frustration.  Maybe it is the AB testing or maybe it is just a language and training barrier (E.G. Is that flat square with text in it a button or just style or a flyout or a drop down?).

Worse yet, for those who measure, these AB tests don't explain why one is preferred over another.  Instead, that is a story we have to develop afterwards to explain the numbers.  In fact, I just told you several stories about how metrics are misused, but those stories were at least in part told by numbers.  In more theoretical grounds, let us consider a scenario.  Is it only mobile users who are expert level liked the B scenario while users of tablets and computers prefer A?  Assuming you have enough data, you have to ask, 'does your data even show that'?  Knowing the device is easy in comparison, but which devices count as tablets?  How do you know someone is an expert?  Worse yet, what if two people share an account, which one is the expert?  Even if you provide sub-accounts (as Netflix does), not everyone uses one and not consistently.  I'm not saying to ignore the metrics, just know that statistics are at best a proxy for the user's experience.