Monday, November 9, 2015

New Adventures in Software Testing

I recently described my past years in testing, but this post is more about the future, using data to back it up.  However, we can only look into the future using the mirror of of the past.  In writing my last 70 or so articles, I have gather a good number of thoughts, and I have posted on average 1-2 in depth posts a month.  In using blogger's data, I had assumed that I was reaching an audience of substance.  However, in investigating my actual reach, it is substantially smaller than I thought.  I have two purposes for writing.  One is to collect my own thoughts, the other is to document my ideas for others to use.  Until today.

I am backing away from my editorial adventures for the time being in order to focus on another adventure.  But wait, don't leave yet.  You've not even asked what it is.  Well, here is the last article I plan to write in the near future.

Often times, I have been asked what sort of code someone should write when they want to learn to write code.  I often come back with the reply, well what sorts of problems do you want to solve?  Why write code at all?  If you have no vision for what to write, being motivated to write code can be incredibly hard.  Even if the answer is I want to write web applications to get a job, that at least provides a hint into the motivations and what sorts of study you need.  Some people have no focus at all and just want to do "something cool".  I have found these people rarely have the interest in completing projects, as cool moves so fast.

In learning to code, I started out with games, like most other people my age.  I built up a set of programs over the years, some of which I spent years developing.  Often times, these programs were designed for a particular need I experienced.  Like my advice, I work on what I find I need rather than what is cool.  Several years ago I became a land lord and found I needed a way to track my write offs.  I also had a job interview that involved PHP, so I wrote a simple tool to track write offs using PHP and Postgres.  I didn't get the job.  My PHP application was clunky and required me to record all the data by hand, yet my bank knew about these transactions.  Why couldn't I just automate the task?  I didn't enjoy working in PHP so I moved to .net and started building a windows application.

I worked for about 3 weeks and had a rough framework designed, primarily using reflections and a Excel-like control.  I started to refine it and within a month I had an application I was happy with.  I kept refining it over a few years, but it was always just meant for me.  I fixed big bugs, but it was meant to please me, so I didn't worry too much about it.

Then, just this last year, my boss, Isaac asked me a question.  He asked me if I was ever going to sell my project.  I thought a while around this and decided it would be a interesting effort and a good learning experience.  I have always wanted to go to work for myself, so why not?  I started in on my adventure.  My adventure included multiple iterations and efforts.  Many of these efforts have been running parallel over the last year.

What Features Do I Need?


I knew what I needed.  I needed one serious feature, the ability to mark items as write offs.  No one else except for the massive applications supported that sort of feature.  These mazes of confusion and madness were more complex than I wanted.  Most of the small open source applications either didn't support the feature or were equally complicated.  Furthermore, the thing I personally needed only a little is one of the big words in personal financing, budgeting.  So, like the heuristic of comparable products, I started examining other products.  I worked on developing features and creating test maps around what would need testing.  But more on that later.

This is a more complex question than you might otherwise think.  For example, what does your EULA look like?  How do you authenticate your key?  What other non-functional requirements am I missing?  I developed a large list of requirements and had to build them all out.

How Do I Start a Business?


I had to start investigating how to license software.  Even the word license is tricky, as I don't sell software, I license its usage.  That affects taxes.  You have to gather all sorts of documentation for the government to allow you to start a business.  Then there is the logo.  What about the store front?  The list goes on and on.  On the good side, I have no employees, so I that simplifies a great deal.

Marketing?


Let me be frank, to the few people who read this regularly.  I'm not good at marketing.  I choose privacy over announcing my existence over and over again.  To make even a personal blog big, it appears one must do lots of advertising.  Isaac suggested it takes 2 years of effort to maybe hit it big, ignoring Black Swans.  I still don't have a personal twitter handle, although I do have one for business to my chagrin.  This will likely be the hardest part for me.  I have a few tweets queued up, and several blog posts about personal finance written.  Hopefully my efforts will pay off, but one never knows.

I also am starting to learn about Google Ads and Bing Ads.  In reality, things are way more complicated than you might imagine.  Google's Ad system is crazy-complicated.  Campaigns connect to Ads which tie to keywords or to a screen scrapper Google uses.  Google Ads use a complicated bidding system where the winner doesn't pay what they bid but the max the last guy bid.  The more I learn the more I learn I know nothing.

Documentation?


I grabbed a free for a year AWS instance of a wiki system and started writing.  I kept writing and writing.  I have written documentation for several open source systems and have done some fairly extensive documentation in the past.  In this case, I tried tackling the problem from several different ways.  The first way I tried was to write out different features of the system.  Then I tried writing things from a different tact.  I wrote about how you can accomplish tasks.

Testing?


As I built out new features, I discovered how complicated my code base started to get.  I had never done any serious regression on my work since it was built just for me.  I had added features I thought might be useful, but it was still poorly tested.  So I built up a list of all the features and major feature interactions and kept a copy in google docs.  Once my program was in a mostly stable state I started rigorously testing it.  I wish I could claim I didn't find any bugs, but that would not be true.  Instead, I found more than two dozen bugs.  Many of them were relatively minor, others were usability.  My list of needed features grew even larger.  I spent 10-20 hours a week for several weeks testing while I spent my normal time on my day job.

One interesting thing I want to note to all my fellow testers is how hard it is to do both the software development and testing.  While I certainly did testing, I found giving some time between developing the code and doing in depth testing gave me some time to forget exactly how I coded something.  That way, my bias in expectations would be gone, allowing me to note usability issues much more clearly.  I also noticed how it was hard to remain motivated.  Testing for 40 hours a week and then going home and doing more of the same is a challenge.  If you're on a death march, imagine how unmotivated a developer is to do any testing of their software.

Shipping is a Feature


Ultimately, I got my application to a stable state.   While I do still occasionally find a bug, I have not found any big bugs in weeks.  I am still doing occasional testing in that I use it for myself.  I do find value in some of the new features I have added in the past 12 months, so I spend more time in my application than I use to.  It's still fairly trim compared to many of my competitors, but I think that is a feature.  Simpler system often have their own sort of value.  I think it is ready for release.  So, on Oct 31st, I released my program, Money Pig, into the world.




I'm also stepping away from our blog for a while, while I work on Money Pig.  I shall return!

- JCD

Wednesday, September 23, 2015

Link Mania: 75 Links with Commentary on Testing, Security, Tech and Life

I have a friend who wants to get into security testing and asked for my help in broadening their education.  I assumed that my friend would have a good number of links on security, and so I would try to start with a broader set of testing knowledge.  They have a degree and know basic computer science, but in discussing needs, I decided that my primary goal would be to provide a broader understanding around testing.  I started by emailing some links I thought might be useful... but that quickly became clear that it was less manageable, so over a few months I put together groups of links I thought might be of value into a Google Doc.

I also wrote some notes on why I thought they were of value for my friend.  It should be noted that I cite myself frequently.  This is not because I'm full of myself but because I addressed specific questions my friend had and felt that my research + links I had in my posts would be of some benefit.  I did do some light editing of my notes, removing personal comments for privacy reasons, but this is roughly what I came up with.

One limitation to this sort of document is I could go on forever.  I certainly could have written several dozen posts around these links.  Before I turn you loose to all these links, let me give some advice.  I look in detail for things I need to know soon and skim general topics for data I might value in the future.  It’s a method that allows me to work well with a large set of disparate data.  It’s why I can talk about all sorts of subjects at some level, but am not an expert in any.  Knowing who actually is worth reading helps me filter the 'must reads' from the 'skims'.  So as you look at these links, look for what you want to learn about, and then look at the author.  If the data is useful, look for more of that author, even links I did not suggest. If it wasn't helpful, try a different author.  If you try about 3 different links in a row in this set, and all are not useful, either you are not primed to learn about the subject, or the type of data/authors I have gathered are not optimal for how you learn.  You will have to decide which.

Happy reading!

Links I Came Up With Before I Started Organizing [Mostly Test]



Link

JCD's notes
http://about98percentdone.blogspot.com/2015/01/leveling-up-your-testing-skills.html I literally wrote this for people trying to improve themselves.
http://bbst.info/?page_id=23 Lectures 2,3,5,6 in particular.  They are about 30 mins per lecture.  Great testing stuff.

The lecture has power point slides, which can be downloaded and are useful by themselves, HOWEVER, he talks about different points not just captured in the slides.
http://about98percentdone.blogspot.com/2013/09/where-cdt-fails-rebuttal.html Isaac’s effort to describe what testers should learn in ‘levels’. Good stuff.
http://www.testingreferences.com/software_testing_bookstore.phpThe books I read that they recommend are good, thus I trust their recommendation enough to suggest taking a look at the list.
http://www.developsense.com/blog/ An interesting thinker in the testing space.
http://www.satisfice.com/blog/archives/1346James Bach, one of a hand full of people to affect the modern software testing world greatly with his words.  In one of his various attempt to define testing.  This one is of particular interesting, because it is about the art and act of doing testing.



http://about98percentdone.blogspot.com/2014/06/my-current-test-framework-testing-large.html
Reflections are a mind-expanding technique to make code think about itself.





http://steve-yegge.blogspot.com/2006/10/egomania-itself.html
I really love Steve Yegge, he maybe one of the best writer-programmer combinations I know of.  I come back to his work frequently.  But he is a programmer, and as such, the one that really matters is the top link.  The rest are very programmer centric.
http://blog.codinghorror.com/ Heard of Stack Overflow?  This guy made that (along with Joel Spolsky of Joel On Software, but Joel’s stuff is getting old).  He writes on a variety of topics and on his good days is really good.  That said, he’s still a developer, thinking like a developer.


Research in Life / Happiness / Living / The Mind / General Career Advice



Link

JCD's notes

http://www.ted.com/talks/brene_brown_listening_to_shame?language=en
These have made me think about how I live my life.

http://about98percentdone.blogspot.com/2014/06/what-is-highest-level-of-skill-in.html
Learn about what you should do to make yourself more valuable in life.  Learning about what matters in learning.
http://about98percentdone.blogspot.com/2014/02/being-fraud-and-failure.html How to deal with feeling like you don’t know enough, when in fact you’re driven to know more.

I have referenced this exact blog post more in my comments to other people’s blog entries than any other.  Often the people who care the most feel this way.
http://about98percentdone.blogspot.com/2013/12/book-consideration-introduction-to.html Read the bullet points at the bottom.  In particular, “The Answer”.  The book reviewed is a little bit of a personal and spiritual look at science.  Somewhat like Sagan, but revolved around thinking.  You might like the book.
http://about98percentdone.blogspot.com/2013/09/testing-hiring-process-for-testers.html

http://about98percentdone.blogspot.com/2013/09/a-subject-of-hiring-process.html

http://about98percentdone.blogspot.com/2013/09/my-interviewing-start-and-changes-ive.html
Isaac and I wrote a little bit about getting hired. These are some of my early blog entries, but you also get both the view of a person being hired by Isaac as well as, Isaac, the hiring manager's method of choosing to hire people like me.

https://sites.google.com/site/steveyegge2/miracle-interview
More interview related thoughts, but from a developer side.
https://sites.google.com/site/steveyegge2/age-racecar-driver (same guy as above, but more philosophical questions in specialization.)



http://www.stickyminds.com/article/helpful-tips-hiring-better-testers
The last set of articles Isaac and I have on interviewing.
http://www.moserware.com/2009/01/wetware-refactorings.html Fairly good.  Almost all true.  Interesting ideas and good set of resources.
http://www.moserware.com/2008/03/what-does-it-take-to-become-grandmaster.html I would give this a lower priority because it’s long and this is not the first time I have found someone who says this and put it in the list.  Hopefully you see the patterns… and will learn.
http://breakingsmart.com/An interesting description of the tug between the past and the future.  It seems to be a little dismissive of some problems, but the general picture is not wrong.
http://blog.codinghorror.com/level-5-means-never-having-to-say-youre-sorry/I hate the title, but the content is pretty good. It provides some insights into why generating reactionary, scripted systems does not scale well in creative work.

Paul Graham is fantastically interesting in general.  This is an interesting attempt to correlate technological outcomes to culture.  I recently heard from a friend who lived in Japan for a year about these Japanese workers at this company, whose company was bought out by an American company after it started to fail, had a real difficult time letting go of the quality of a product in order to get the product out the door. That story feels like a sort of quick CRC check for me, meaning this article probably does have some veracity.
http://www.ribbonfarm.com/the-gervais-principle/I know I have talked about this before, but it is an interesting model of human behavior.  While I’m not actually a fan of The Office, I found it was mostly ‘translatable’.  It also offers small notes on the Peter and Dilbert principles which are also worth looking at.


Testing



Link

JCD's notes
https://www.youtube.com/watch?v=j_JviA5nvS0&list=PLSIUOFhnxEiDFckNDSjKWqOCtd8ksJrh4GTAC is fairly good.  Some of the content might be on security. I have not watched all the 2014 (and soon 2015) videos, only the 2013 videos which were often very good.
http://www.associationforsoftwaretesting.org/conference/cast-2015/ CAST 2015 is probably worth watching. I have yet to watch it myself, but I attended last year.
http://www.testingreferences.com/testingnews.phpI use this as a tool, looking for things of interesting, read them, keeping note of who wrote it and if I decide I don’t like someone (for any reason), I mentally filter them out of the list.  It’s the firehose method, good for going in all different places, but you never know what you might get.

How SQL joins work. They are often used both for getting data for testing as well as part of how many reports are generated.
http://angryweasel.com/blog/In broad strokes, I agree with Alan and appreciate his perspective with about 20 years at Microsoft.
http://oredev.org/2013/wed-fri-conference/balancing-atdd-gui-automation-and-exploratory-testing A video I watched long ago that is still in my notes.  Might be interesting from an automation vs exploratory testing perspective.  I have little memory of it though. Most people only teach the basics :(
http://www.huibschoots.nl/wordpress/ Like Alan, I agree with Huib broadly, and I find his view very interesting as he comes from a European background.


General Tech



Link

JCD's notes
https://howdns.works/episodes/ How DNS works.
http://blog.codinghorror.com/on-software-engineering/Why software engineering is so difficult, and why consultants are sometimes looked down upon.
http://www.dreamincode.net/forums/topic/223324-an-interesting-interview-with-steve-yegge-and-james-duncan-about-java/ Interesting set of interviews, the biggest point is that you should know your tools well enough you know what is wrong with them.
http://www.moserware.com/2009/03/how-net-regular-expressions-really-work.htmlI think the data here is interesting, however, the meta is important too, even at a security level.  A DDOS attack is a security attack, but if I can DDOS you with one click, it is a security issue too.  Knowing how regular expressions work tells you about how a lot of systems work inside, which is how samy was able to defeat the defenses of MySpace ( http://namb.la/popular/ ).  It’s also a huge part of getting things done.
http://edge.org/annual-question/what-do-you-think-about-machines-that-think An interesting set of speakers, most very smart, all on the same topic.  The 2014 topic was interesting too. This might generate some interesting questions, such as:


What would happen if/when the #1 job goes away (Driving, at least in the USA; currently biggest economy in the world; last count ~ 3 million souls in the US do this)?  
What do we do with these people?  
What of those that aren’t capable or interested in more mind-oriented work or who are too old to change careers?  
What of my grandmother who doesn’t interact well with online work, in part because her hands are too crippled to do much with keyboards and did not grow up with this world?
What happens when machines categorize someone as an outlaw in some countries which others counties would recognize as moral and legal (such as being gay, or a political dissidents)? Who is legally and morally responsible for a 'thinking' machine?

https://sites.google.com/site/steveyegge2/math-every-day
Here is a mission in life -- how do you solve problems in which there is no solution?  (My only hint is find another problem.)  However, the point is, solving one of these non-trivial problems is a lifetime’s worth of work, if you want it.  See the intro of the second piece to see what I mean.
http://www.amazon.com/review/RUGSCP3XBNBUVSuggested by Yegge’s post… it’s an interesting read in and of itself.  It’s got some ideas I’ve not really researched.


Security



Link

JCD's notes
https://www.youtube.com/watch?v=n9-Gz1U87CI&index=9&list=PLQB4l9iafcelXpJnK6IyDsoFeEb1icqrl Brilliant! I attend this live and you’ll see me in the end asking a question.  All the rest of the talks (I kept the list) are test related, but this is security related.






(Others…)
https://heimdalsecurity.com/blog/best-internet-security-blogs/
Famous security researchers blog.
https://krebsonsecurity.com/category/how-to-break-into-security/ How to learn to be a security professional.
https://www.schneier.com/blog/archives/2013/04/nice_security_m.html Why we are all frauds and failures sometimes (As address in a link above).


A *practical* piece of tech that really applies around security of the web, he has other good posts too.

http://www.guru99.com/learn-penetration-testing.html
I liked the graph, purely from a ‘what do you mean by “security testing”’ question. NOTE: My friend suggested http://www.techrepublic.com/blog/it-security/the-five-phases-of-a-successful-network-penetration/ might also be useful.
https://www.cs.purdue.edu/homes/xyzhang/fall07/Papers/sw-test.pdf Read this a long time ago and it was a little helpful.
http://www.veracode.com/security/software-security-testingHappened upon this, no idea if it is any good. I did not get chance to really review it.
http://blogs.msdn.com/b/oldnewthing/archive/2013/12/24/10484402.aspxMight be interesting, taken from a comment from...
http://blog.codinghorror.com/welcome-to-the-internet-of-compromised-things/here.  Also interesting.
https://www.grc.com/fingerprints.htmThey have some interesting stuff regarding SSL but more importantly they might be a generally useful resource.
https://www.youtube.com/watch?v=wKDE_upBlfcComplex example of a hack and how it can be done by defeating multiple layers and understanding history.


Other



Link

JCD's notes
http://blog.incubaid.com/2012/03/28/the-game-of-distributed-systems-programming-which-level-are-you The technical subject is interesting and it provides insight into architecture. It ALSO gives a nice idea of levels and how you can’t really know what level you are at until you see the next level up. I actually have a more complex theory on this, but I have yet to write the blog post.
http://www.moserware.com/2009/07/just-enough-mba-to-be-programmer.html Just plain useful in a practical sort of way.
http://archive.wired.com/wired/archive/4.12/ffglass_pr.htmlI have not yet completely read it but it was recommended to me by a trusted source.  It’s long.
http://archive.wired.com/wired/archive/8.04/joy.htmlVery long indeed, but worth reading.

You made it to the bottom?  And you counted them?  Only 74 links and you want your money back? Alright! Fine. Well I'm sure I can come up with a 75th link just for you, my bean counting friend.  How about something interesting?  Like really interesting.  Here you go: http://www.damninteresting.com/the-zero-armed-bandit/

Thursday, September 10, 2015

Belief: Absolute Conviction or Probability?

I have long been thinking about the nature of the world, the nature of belief, knowledge, faith, etc. I have come to a conclusion that is both obvious and perhaps to some, scary.  This is one of those personal adventures, and it will take a little bit to explain.  It's also rather abstractly connected to testing, so if you are looking for how-to articles on testing,  I'd go looking here or even here instead.  You've been warned.

By modifying the definition of belief ever so slightly, my language in fact sounds completely reasonable, and I suspect that my version of belief is much closer to the reality of what a belief really is in a psychological sense.  In particular, I can say I have observed it with people who do scientific-like work.  Let's first start with dictionary definition of belief.

According to Merriam-Webster, a belief is:

1. a state or habit of mind in which trust or confidence is placed in some person or thing
2. something believed; especially : a tenet or body of tenets held by a group
3. conviction of the truth of some statement or the reality of some being or phenomenon especially when based on examination of evidence

Knowledge in epistemology is well known to be a "True Justified Belief."  Effectively, the difference between belief and knowledge is that a belief may not be true and it might not be justified. You can only claim to know something when it is true and justified (although how much justification is still in question).  I have also done some informal consideration of justification, which can be valuable in justifying a bug.

In a World...


Imagine, if you will, a world where beliefs were held so tightly that they were in fact the way you ran your life with exact and precise calculations.  You might be like some of the Vulcans who dedicate themselves to pure logic, as described in Star Trek, except your dedicate would be to your own beliefs, not logic per-se.  If, for example, you believed in a set of development principles, you would never break any of these principles.  Assuming your principles were designed correctly to never have functional bugs, a functional bug would not exist in a code base you exclusively created because your beliefs would be personal law, with you never straying. The only possible remaining set of functional bugs are those which are not covered by your beliefs or areas where your personal mental model of your belief was not exactly in congruity with the actual statements of the guiding principle, which is just a form of misunderstanding.

For example, if you stated your belief was "No text box I create will have an injection attack possible.", but your mental model was only based upon SQL injection and did not consider command injections.  Then if you had a vulnerability to a command injection attack in your text box, is it the fault of your beliefs that no text box should be vulnerable to injection attacks?  I think not, because you either had a limit to your understanding of your philosophy (what you believe an injection attack to be) or the world (what injection attacks are known to the world).  However, a more subtle question and interesting question is, did you really hold the belief?  I think the answer is complicated.  Internally, the answer is yes, of course you believed.  However, from an external view did you hold the true belief?  Yes, you held the belief in your own way, however, it might not meet some external evaluator's view of how that belief should be held.  This is a communication problem people run into all the time when they define words differently.

Thus, your personal reality would appear to succumb to your beliefs, or your beliefs would say nothing about reality. That is to say you would ignore your own senses when you observed the natural world contradict your beliefs. This occasionally happens at a developer's desk when they say, "That isn't possible." as I demonstrate a bug to them.  Do they see the world shifting out from under them or are they ignoring it?  Do they see their belief ruined and their world is crashing down?  For some people, this Vulcan-like demand for the perfect belief, the idea of the world crashing down might come close to their view of life.  Then there are others who choose to doubt only the facts they don't like, so that their belief is maintained.  But do these groups go from the point where they say “I believe” and run their life exactly according to their proclaimed beliefs?

Perhaps a few do run their life exactly according to their beliefs, but I suspect that it is more likely most have a journey where you slowly reform both themselves and their beliefs.  Even with all the defenses around their beliefs, and all the denial for the facts contradicting, I think given time to consider, people can migrate beliefs.  Like the developer who claims the bug is not possible, they adjust their beliefs over time. So, either most of us are able to hold two opposing beliefs at the same time ("That isn't possible" and "That just happened") as we reform our beliefs, or are able to ignore those former beliefs somehow, thus truth in the mind and actions are not strongly related.  The latter option is sometimes referred to as rationalization.

For the sake of argument, I am going to assume that in fact people can hold two opposing beliefs at the same time. The reasoning I am assuming this is that if belief of a truth and how one acts have limited relations, then we are creatures whose entire rationale does not matter and thus the entire nature of belief doesn’t matter in this context. Another reason for my assumption is that in working with software testers, I have observed this ability to weight two opposing beliefs.  In my observations, it wasn't that they had ignored their belief and just done something else but rather taken context into consideration.

50% Chance of Opposite Day


If we can in fact hold two opposing “beliefs” then those are not beliefs in the traditional sense.  One is not in fact completely confident in their belief.  Instead, what they almost become is an internal struggle for which belief is more accurate in the modeling of the world.  Since different people model the world differently, a realization occurs that multiple models can both be correct.  In a real sense, the person with said beliefs is weighing non-mathematical probabilities.  Now, if that is what we are doing, trying to decide which version of belief is more likely, and we have the ability to replace one belief with another based upon this evaluation, beliefs are just things we attach a probability on.  Another way of saying this is that they are context driven.  The probability that a particular belief will satisfice is determined through life-experienced context and those probabilities are constantly being re-evaluated as more of life is experienced.

I try very hard to avoid politics in this blog, but I wanted to address a good example of differences in mental models where it is not clear what the meaning or intent is.  Recently, a woman claimed that she is black when her parents disagree with her and claim she is white.  The interesting question here is, what is it to be "black"?  Is it a life style?  Is it a social upbringing?  Is it a description of particular genes?  Is it a color?  At what time and under what conditions?  Is it an origin?  How many generations back?  Is it a description of being disadvantaged? Is it an artificial classification?  How many of these checkboxes do you have to have to be considered black and which oracle(s) do you listen to?  Is this another true Scotsman problem?

If beliefs are actually statements we find likely to be true, we are attaching what I will refer to as a probability. It may not be numerically calculated, but a consideration of one’s experiences, perceptions, internal mental structure, and numerous other factors. Ultimately we come to some sorts of conclusions. "I write great code", "I’m in love", "JCD writes good blog posts", ad nauseum. Can you in fact believe both “men are evil” and that “evil does not exist”? If those are just probabilities, then yes, those can in fact both be beliefs at the same time for one person. Keeping in mind that knowledge is a true justified belief; you can’t “know” both of these things at once, because only one can logically exist at once, but which one? Even with all the brains we have, this appears to be a Gordian knot of a problem. Cutting the knot using things like Occam’s razor only goes so far (It is only a razor!).  Worse yet, the question of context pops its head in here.  "In the past year JCD has written good blog posts" adds context that the first statement did not have.  A developer's defense of "No one would do that" is true in the context of "that I like and hasn't made a mistake and ...."  Perhaps with enough context one could claim knowledge, but you know the saying about building an idiot proof box... I suspect if you could build enough context, someone would just come up with new unconsidered situations.

Why not accept our nature and in fact embrace it? If someone says that “men are evil” and provides limited evidence, you can assign a probability to that (let us assign an arbitrary tag “somewhat likely” to this). Then someone else comes along with highly convincing evidence that in fact evil can’t exist. You weigh the factors you now have and change the scales, applying a “very likely” for "evil can’t exist" and downgrade the “men are evil” viewpoint to “fairly unlikely.” You need not deny either one of those items and yet you can still hold a legitimate opinion that "According to my present data, evil is unlikely to exist."  The tricky thing is, English does not make it easy to give these sorts of answers.


"Nonsense is so good only because common sense is so limited." - George Santayana


In a brief change of topic, I want to stop and address what some readers might be thinking at this point.  I am sure some people see this as non-sense.  So I want to take a brief aside to consider the possibility that this idea doesn't matter.
"In the West, we take the “law of the excluded middle” as self-evident  and  as fundamental  to  logic.  Under  this  rule,  a proposition can have two states: something “is X” or “is not X”. There is no middle ground. In contrast, several Indian logicians assumed four states, “is X”, “is not X”, “is both X and not X” and “is neither X nor not X”. I puzzled over this for many years. I am not confident that my understanding of how this could be meaningful is the same as their understanding. But in human affairs, in interpreting  people,  how  they  interact,  what  situations  they  are  in,  and  how  to negotiate  with  them,  I  came  to  view  the exclusion of the middle as a heuristic rather than a law or a rule. It is often useful to assume that “X or not X” are the only two possibilities, but when that is not productive, it is time to look for a third way." - Dr. Cem Kaner, Tea-time with Testers, June 2015, Pg 61
I could go on, as Dr. Kaner certainly has more around this.  The idea that perception is in fact reality is often skewed.  In the same magazine issue, Jerry Weinberg tells a story in which the three team members whom did all the talking were perceived as the leaders performing actions but the the one with an effective action was not the team member being defined as the leader.  The team member with the affective action was another employee whom only did one thing, silently solve the problem.  Perceiving motion for action is not an uncommon occurrence.

The point I'm trying to make is those whom see belief as a solid, absolute thing probably have a more difficult time seeing that 'reality' is more difficult to pin down.  It seems likely to me, that for most people, as they gain more experience, they too will reform their view of belief into something more generalized, such as a probability.

"The unexamined life is not worth living." - Socrates


Briefly, I want to address another sticky subject, faith.  Faith is a belief with only external sources for justification. You might be able to cite examples where the external source was previously correct, but the external source is still your only justification. Faith is often replaceable with “trust” or “hope”. You can say I have faith that login is broken because my manager is telling me it is so. The justification is based upon hope or trust in some external or internal force that your belief is true. If instead we consider faith a probability based upon your level of trust in a given oracle, you can in fact see faith is just a specialized description of a belief.

One problem with this is our language does not naturally imply that our statements are guess work, nor easily allow one to explain what the most likely evaluation you have thus far computed. People say “I know ...”, or “It must be ...”. We assert our knowledge because people are not actually comfortable with “I don’t know, but I think ...” It is in our nature to say we are right. We even prefer people in which we perceive confidence, even though it maybe a lie.  It is also convenient:

"A little inaccuracy sometimes saves a ton of explanation." - Saki
Compare "According to my present data, evil is unlikely to exist." and the shorten "evil does not exist."  In my view, these absolutes often don't represent what people really think, but I admit it's only a probability.

Another problem is that people will claim this is a flip-flop philosophy that does not in fact hold a person’s “feet to the fire”. While this is true to a certain extent, it is designed to actually be more in line with how people and our estimation of reality actually work. People are rarely able to be self-consistent in what they say, much less what they do. How often do people say “Do as I say, not as I do.”?

In recent years, people have been able to pull out long and specific quotes from others because of how our technology records everything.  We are very much able to examine other people's lives.  Yet, in looking for this consistency, we fail to examine our own lives and the changes we make.  If other people are often unable to be consistent, it's likely you too will likely fail at consistency (Consider: did you know every time you recall something, the memory is re-written and thus modified?).  Yet we like consistency.  There are still two obvious options.  Either we are really failures in consistency but pretend we are not or we are constantly evolving our understanding and are designed to evolve our thinking.  I'd give it about a 20% / 80% probability for anyone individual in circumstances similar to mine when I consider....Oh you get the idea.

Ultimately this is a small chunk of analysis attempting to describe my view of how people work.  It is hard, if not impossible, to write out a full view due to the fact no one else has experienced my life and the context that brings.  I also recognize other views exist and I think they have some probability of them being correct and am open to replacing my own view with a more probable one.  So please feel free to share any insights you have, even if they contradict my own view.  I encourage you to sit down and attempt to write out your views on how people work.  It can be a useful exercise, as it will give your opinions a more solid foundation.  But if you have done all that hard work, why not post it as a comment?

Wednesday, July 15, 2015

My Many Years in Testing and Automation Development

I have heard the question expressed in multiple different ways.  "How do I get into QA?"  "How do I progress my career?"  "How do I level up my skill?"  "You read whitepapers?!?"  "What type of automation should I write?"  I have written various articles around this broad range of topics, but I wanted to take a slightly different tact around one of the implied questions.  The implied question I'm considering is how do I continue down the path from where I am to become a better tester and what skills should I spend time working on improving?  Instead of trying to explore the topic in a general sense, I am going to go through a few interesting experiences I have had in both "Testing" and "Automation Development".  While I will be talking about automation, I do not mean to imply that is where you should spend your time learning, just that I will discuss how I made the choice.  I will try to concentrate on interesting stories and lessons learned.

I have been writing code for nearly half my life, but I started working in the development of test automation somewhere between 2004 and 2005.  I started professionally testing in 2002, although I had been writing my own code and testing it earlier.  I also did some testing for others before 2002, but I was not paid for that work.  I will try to describe the companies and context a little, but I will not use company names, to protect the innocent.  But first, a small introduction to perhaps explain how early on I was interested in QA.

When I started my life in QA, I was roughly 6 years old and curious about this clock in a video game called "The Last Ninja" for the NES.  It kept a timer of how long you had been playing the game.  It gave hours, minutes and seconds.  There was a place where no bad guys were with a stream and a park bench.  It was perfect for leaving my ninja to sit and just enjoy the scenery.  In only 99 hours 59 minutes and 59 seconds I could find out what happened when the clock ran out.  I knew Sonic would run off the screen after 3 minutes, so I conducted some smaller tests.  In my smaller tests, I discovered, unlike Sonic, The Last Ninja did not seem to mind waiting for short periods of time, so it seemed easy enough to test the longer case.  The only problem was I didn't want to lose my Nintendo for a week while I waited for the result.  So when I went on vacation, I left my system running.  When I came back, what I found was "0A" and the game was frozen up.  Now, many years later, I know 0A is hex for 10, likely meaning the clock had run over to hex and then the game had crashed.  I didn't understand the result, but I was curious about it.  I was later grounded for leaving my system on for that long.  You've been warned kids!

Years later, I was looking to get a job out of High School.  I had been programming text adventures for some years, but never had a 'real' programming job.  I enjoyed programming, I enjoyed the experience of creating, but still didn't feel like I knew that much.  Still I kept at it.  I had worked at the Post Office for 3 weeks in a Christmas rush season, but had little other professional work under my belt.  I was going to school and studying Computer Science, but that was going to take years and I needed money sooner.  So I started looking for a computer job.  I interviewed once, but didn't know what I was doing and did not get the job.  College and High School don't seriously prepare you for interviewing.  In my second job interview, I took a computerized test and passed all but one question (I tried typing in a search but they expected me to use the dropdown...  No, I'm not still bitter.).

I got the job and started working as a managed contractor for a overly large company who worked on specialized hardware.  I was a tester who did everything from competitive intel to stress testing to environment testing to configuration testing.  However, I had no training and really no idea that testing was a general skill set.  I incidentally learned a few simple QA skills, but nearly all of the learning I did was around social skills in my first year.  Perhaps one the most important skills I learned was to apologize is important and doubly so when you are a manager and have made a mistake.  I moved to a more complex set of testing, but it was a script based.  I learned more about networking and security testing.  I almost got a job in a team that was inspired by James Bach's work around exploratory testing (in one of the places that James and his brother did some of the development around the idea of exploratory testing), but I missed out to someone with a lot more experience.

Then, in a sad set of events a former lead who changed shifts passed away suddenly.  I took his place, an awkward thing for sure.  I learned about this idea that you could 'automate' tests.  I began to apply my existing programming skills in order to test various UI elements.  It worked well in some cases, but we struggled to handle descriptions (a QTP phrase for describing a way of IDing elements in a UI system).  A co-worker of mine had more difficulty picking up the programming as his specialty was in hardware, not software.  He came into work more than once complaining of nightmares around programming.  My coworker did eventually get better at automation, but that was never his strength (Aside: He's now a big-wig manager, likely making way more than me!).

I bounced around from place to place, with a rough total of nine different jobs (not all jobs were title changes).  One thing looking back is how I was always trying to go meta, not just writing good automation but improving the entire process.  In a large company, that can be very difficult.  I ultimately had a conflict with one of the managers and felt the need to leave.  Before I left, I was told to quit trying to improve the organization (by providing useful tools) but instead work on personal projects!  I did and learned SQL and web development while looking for a job.  I also had multiple open source projects under my belt by this point.

If a big company didn't work, I reasoned that a smaller company would be better!  I applied to one company and studied up.  Rather than applying to many companies, I reasoned it was better to be prepared.  This from having done so poorly years ago seemed to work reasonably well.  I got a job offer. I tried to negotiate for more money, but was declined.  I declined the job but was later called back and given a slightly higher offer.  I concluded that because of the recession that a much bigger offer was not going to happen.

I took a job at a 50 person company, and within 2 weeks of me joining the QA manager had quit.  This left the QA group answering to the VP of Engineering and it remained that way for nearly 2 years.  I worked for longer hours at this company than anywhere else.  I lost another co-worker who I was training, and whom I may have been the last person he talked to.  He was hit by a bus.  I learned the 'hit by a bus' scenario is both real and the worry about how the business knowledge loss will be much less impactful than the emotional impact from the loss.  I learned what the term 'death march' meant as I worked 80+ hour work weeks.  I learned how a recession can affect you in spite of the fact that your field is in demand.  I learned the dangers of saying that you won't ship with any [known] bugs.  I learned how working more hours did not equal more productivity.  I wrote nearly 1300 automated UI tests.  I found the most bugs at the company, in part because of the automation I wrote, and because all bugs must be fixed, it must logically be concluded I found the most important ones.  (Yes, that is sarcasm.)  Yet, I was not promoted nor paid more for nearly the entire 3 years I was there.  Only after 50% turn over and the firing of upper management did things change even a little, and by then I was too tired to care.  I left.  In case you want more details, Isaac also detailed his own personal journey at this same company.

Now we are getting into more recent events.  To be perfectly honest, I don't plan on telling anything more about the various company cultures because of how it might affect me.  However, I still learned social and technical lessons.

I learned that 'manual' testers could appreciate automation and automation results.  I, for the first time, got to work with other people with equal or better coding skills than myself as we developed automation.  I got to learn how to play a technical leader, even when I was not promoted into a leadership position nor had any authority over others.  While I had dealt with databases and did a little database testing before, I got to do development in which nearly all the code was  around database usage, including communication between database systems.  For the first time I had to deal with the integration of multiple team's automation efforts which were written in different languages.  By this point in my career I had written test code in C, Java, JavaScript, SQL (various flavors), C#, J#, VB.NET and VB Script.

On the social front, I got to work with both one of the best teams and one of the most difficult teams I ever worked with.  In dealing with the difficult team, I got to see how personality clashes worked and how to work around them.  To be fair, I have my own personality quirks too, but when these hit extremes it is imperative that you figure out how to be a positive force for the team rather than yet another point of contention.  Perhaps one of the biggest lessons I learned was that it is important to have a solid social connection and understanding of an individual before giving a completely honest evaluation of someone's skill set.  In fact, I took the lesson so much to heart that I try to be more careful with my day-to-day speech as a whole.

Finally, I want to describe a little bit of my attempts at professional growth.  I spent a good amount of time embracing various avenues for learning.  I took BBST, and while I only took the class for the first BBST course, I watched all the videos and read all the material.  I have written on this blog for a few years now.  I have spent time giving talks at various venues.  I have written letters, written for other professional venues, testing non-work apps and taken serious amounts of time trying to understand the systems under which work is performed.

So to sum this all up, I think I have just two simple suggestion for you:

Learn

Making mistakes is fine.  Just learn from them.  Learn how you learn.

Don't just do work

We all are making a journey, and at some point it will stop.  Making an impact is important.  But don't make work the only impact in life.