Link Mania: 75 Links with Commentary on Testing, Security, Tech and Life

I have a friend who wants to get into security testing and asked for my help in broadening their education.  I assumed that my friend would have a good number of links on security, and so I would try to start with a broader set of testing knowledge.  They have a degree and know basic computer science, but in discussing needs, I decided that my primary goal would be to provide a broader understanding around testing.  I started by emailing some links I thought might be useful... but that quickly became clear that it was less manageable, so over a few months I put together groups of links I thought might be of value into a Google Doc.

I also wrote some notes on why I thought they were of value for my friend.  It should be noted that I cite myself frequently.  This is not because I'm full of myself but because I addressed specific questions my friend had and felt that my research + links I had in my posts would be of some benefit.  I did do some light editing of my notes, removing personal comments for privacy reasons, but this is roughly what I came up with.

One limitation to this sort of document is I could go on forever.  I certainly could have written several dozen posts around these links.  Before I turn you loose to all these links, let me give some advice.  I look in detail for things I need to know soon and skim general topics for data I might value in the future.  It’s a method that allows me to work well with a large set of disparate data.  It’s why I can talk about all sorts of subjects at some level, but am not an expert in any.  Knowing who actually is worth reading helps me filter the 'must reads' from the 'skims'.  So as you look at these links, look for what you want to learn about, and then look at the author.  If the data is useful, look for more of that author, even links I did not suggest. If it wasn't helpful, try a different author.  If you try about 3 different links in a row in this set, and all are not useful, either you are not primed to learn about the subject, or the type of data/authors I have gathered are not optimal for how you learn.  You will have to decide which.

Happy reading!

Links I Came Up With Before I Started Organizing [Mostly Test]

Link	JCD's notes
http://about98percentdone.blogspot.com/2015/01/leveling-up-your-testing-skills.html	I literally wrote this for people trying to improve themselves.
http://bbst.info/?page_id=23	Lectures 2,3,5,6 in particular.  They are about 30 mins per lecture.  Great testing stuff. The lecture has power point slides, which can be downloaded and are useful by themselves, HOWEVER, he talks about different points not just captured in the slides.
http://about98percentdone.blogspot.com/2013/09/where-cdt-fails-rebuttal.html	Isaac’s effort to describe what testers should learn in ‘levels’. Good stuff.
http://www.testingreferences.com/software_testing_bookstore.php	The books I read that they recommend are good, thus I trust their recommendation enough to suggest taking a look at the list.
http://www.developsense.com/blog/	An interesting thinker in the testing space.
http://www.satisfice.com/blog/archives/1346	James Bach, one of a hand full of people to affect the modern software testing world greatly with his words.  In one of his various attempt to define testing.  This one is of particular interesting, because it is about the art and act of doing testing.
http://about98percentdone.blogspot.com/2013/10/reflections.html http://about98percentdone.blogspot.com/2013/11/testing-with-reflections-part-ii.html http://about98percentdone.blogspot.com/2014/06/my-current-test-framework-testing-large.html	Reflections are a mind-expanding technique to make code think about itself.
https://sites.google.com/site/steveyegge2/godel-escher-blog https://sites.google.com/site/steveyegge2/is-weak-typing-strong-enough http://steve-yegge.blogspot.com/2008/05/dynamic-languages-strike-back.html http://steve-yegge.blogspot.com/2006/10/egomania-itself.html	I really love Steve Yegge, he maybe one of the best writer-programmer combinations I know of.  I come back to his work frequently.  But he is a programmer, and as such, the one that really matters is the top link.  The rest are very programmer centric.
http://blog.codinghorror.com/	Heard of Stack Overflow?  This guy made that (along with Joel Spolsky of Joel On Software, but Joel’s stuff is getting old).  He writes on a variety of topics and on his good days is really good.  That said, he’s still a developer, thinking like a developer.

Research in Life / Happiness / Living / The Mind / General Career Advice

Link	JCD's notes
https://www.youtube.com/watch?v=iCvmsMzlF7o http://www.ted.com/talks/brene_brown_listening_to_shame?language=en	These have made me think about how I live my life.
http://about98percentdone.blogspot.com/2014/10/noise-to-signal-attempt-to.html http://about98percentdone.blogspot.com/2014/06/what-is-highest-level-of-skill-in.html	Learn about what you should do to make yourself more valuable in life.  Learning about what matters in learning.
http://about98percentdone.blogspot.com/2014/02/being-fraud-and-failure.html	How to deal with feeling like you don’t know enough, when in fact you’re driven to know more. I have referenced this exact blog post more in my comments to other people’s blog entries than any other.  Often the people who care the most feel this way.
http://about98percentdone.blogspot.com/2013/12/book-consideration-introduction-to.html	Read the bullet points at the bottom.  In particular, “The Answer”.  The book reviewed is a little bit of a personal and spiritual look at science.  Somewhat like Sagan, but revolved around thinking.  You might like the book.
http://about98percentdone.blogspot.com/2013/09/testing-hiring-process-for-testers.html http://about98percentdone.blogspot.com/2013/09/a-subject-of-hiring-process.html http://about98percentdone.blogspot.com/2013/09/my-interviewing-start-and-changes-ive.html	Isaac and I wrote a little bit about getting hired. These are some of my early blog entries, but you also get both the view of a person being hired by Isaac as well as, Isaac, the hiring manager's method of choosing to hire people like me.
https://sites.google.com/site/steveyegge2/what-you-need-to-know https://sites.google.com/site/steveyegge2/miracle-interview	More interview related thoughts, but from a developer side.
https://sites.google.com/site/steveyegge2/age-racecar-driver	(same guy as above, but more philosophical questions in specialization.)
http://about98percentdone.blogspot.com/2013/10/interviewing-post-3.html http://www.stickyminds.com/article/developing-team-testers http://www.stickyminds.com/article/helpful-tips-hiring-better-testers	The last set of articles Isaac and I have on interviewing.
http://www.moserware.com/2009/01/wetware-refactorings.html	Fairly good.  Almost all true.  Interesting ideas and good set of resources.
http://www.moserware.com/2008/03/what-does-it-take-to-become-grandmaster.html	I would give this a lower priority because it’s long and this is not the first time I have found someone who says this and put it in the list.  Hopefully you see the patterns… and will learn.
http://breakingsmart.com/	An interesting description of the tug between the past and the future.  It seems to be a little dismissive of some problems, but the general picture is not wrong.
http://blog.codinghorror.com/level-5-means-never-having-to-say-youre-sorry/	I hate the title, but the content is pretty good. It provides some insights into why generating reactionary, scripted systems does not scale well in creative work.
http://www.paulgraham.com/usa.html	Paul Graham is fantastically interesting in general.  This is an interesting attempt to correlate technological outcomes to culture.  I recently heard from a friend who lived in Japan for a year about these Japanese workers at this company, whose company was bought out by an American company after it started to fail, had a real difficult time letting go of the quality of a product in order to get the product out the door. That story feels like a sort of quick CRC check for me, meaning this article probably does have some veracity.
http://www.ribbonfarm.com/the-gervais-principle/	I know I have talked about this before, but it is an interesting model of human behavior.  While I’m not actually a fan of The Office, I found it was mostly ‘translatable’.  It also offers small notes on the Peter and Dilbert principles which are also worth looking at.

Testing

Link	JCD's notes
https://www.youtube.com/watch?v=j_JviA5nvS0&list=PLSIUOFhnxEiDFckNDSjKWqOCtd8ksJrh4	GTAC is fairly good.  Some of the content might be on security. I have not watched all the 2014 (and soon 2015) videos, only the 2013 videos which were often very good.
http://www.associationforsoftwaretesting.org/conference/cast-2015/	CAST 2015 is probably worth watching. I have yet to watch it myself, but I attended last year.
http://www.testingreferences.com/testingnews.php	I use this as a tool, looking for things of interesting, read them, keeping note of who wrote it and if I decide I don’t like someone (for any reason), I mentally filter them out of the list.  It’s the firehose method, good for going in all different places, but you never know what you might get.
http://blog.codinghorror.com/a-visual-explanation-of-sql-joins/	How SQL joins work. They are often used both for getting data for testing as well as part of how many reports are generated.
http://angryweasel.com/blog/	In broad strokes, I agree with Alan and appreciate his perspective with about 20 years at Microsoft.
http://oredev.org/2013/wed-fri-conference/balancing-atdd-gui-automation-and-exploratory-testing	A video I watched long ago that is still in my notes.  Might be interesting from an automation vs exploratory testing perspective.  I have little memory of it though. Most people only teach the basics :(
http://www.huibschoots.nl/wordpress/	Like Alan, I agree with Huib broadly, and I find his view very interesting as he comes from a European background.

General Tech

Link	JCD's notes
https://howdns.works/episodes/	How DNS works.
http://blog.codinghorror.com/on-software-engineering/	Why software engineering is so difficult, and why consultants are sometimes looked down upon.
http://www.dreamincode.net/forums/topic/223324-an-interesting-interview-with-steve-yegge-and-james-duncan-about-java/	Interesting set of interviews, the biggest point is that you should know your tools well enough you know what is wrong with them.
http://www.moserware.com/2009/03/how-net-regular-expressions-really-work.html	I think the data here is interesting, however, the meta is important too, even at a security level.  A DDOS attack is a security attack, but if I can DDOS you with one click, it is a security issue too.  Knowing how regular expressions work tells you about how a lot of systems work inside, which is how samy was able to defeat the defenses of MySpace ( http://namb.la/popular/ ).  It’s also a huge part of getting things done.
http://edge.org/annual-question/what-do-you-think-about-machines-that-think	An interesting set of speakers, most very smart, all on the same topic.  The 2014 topic was interesting too. This might generate some interesting questions, such as: What would happen if/when the #1 job goes away (Driving, at least in the USA; currently biggest economy in the world; last count ~ 3 million souls in the US do this)?   What do we do with these people?   What of those that aren’t capable or interested in more mind-oriented work or who are too old to change careers?   What of my grandmother who doesn’t interact well with online work, in part because her hands are too crippled to do much with keyboards and did not grow up with this world? What happens when machines categorize someone as an outlaw in some countries which others counties would recognize as moral and legal (such as being gay, or a political dissidents)? Who is legally and morally responsible for a 'thinking' machine?
https://sites.google.com/site/steveyegge2/nonesuch-beast https://sites.google.com/site/steveyegge2/math-every-day	Here is a mission in life -- how do you solve problems in which there is no solution?  (My only hint is find another problem.)  However, the point is, solving one of these non-trivial problems is a lifetime’s worth of work, if you want it.  See the intro of the second piece to see what I mean.
http://www.amazon.com/review/RUGSCP3XBNBUV	Suggested by Yegge’s post… it’s an interesting read in and of itself.  It’s got some ideas I’ve not really researched.

Security

Link	JCD's notes
https://www.youtube.com/watch?v=n9-Gz1U87CI&index=9&list=PLQB4l9iafcelXpJnK6IyDsoFeEb1icqrl	Brilliant! I attend this live and you’ll see me in the end asking a question.  All the rest of the talks (I kept the list) are test related, but this is security related.
http://krebsonsecurity.com/ https://www.schneier.com http://taosecurity.blogspot.ro/ (Others…) https://heimdalsecurity.com/blog/best-internet-security-blogs/	Famous security researchers blog.
https://www.schneier.com/blog/archives/2012/07/how_to_become_a_1.html https://krebsonsecurity.com/category/how-to-break-into-security/	How to learn to be a security professional.
https://www.schneier.com/blog/archives/2013/04/nice_security_m.html	Why we are all frauds and failures sometimes (As address in a link above).
http://www.moserware.com/2009/06/first-few-milliseconds-of-https.html http://www.moserware.com/2009/09/stick-figure-guide-to-advanced.html	A *practical* piece of tech that really applies around security of the web, he has other good posts too.
http://www.guru99.com/what-is-security-testing.html http://www.guru99.com/learn-penetration-testing.html	I liked the graph, purely from a ‘what do you mean by “security testing”’ question. NOTE: My friend suggested http://www.techrepublic.com/blog/it-security/the-five-phases-of-a-successful-network-penetration/ might also be useful.
https://www.cs.purdue.edu/homes/xyzhang/fall07/Papers/sw-test.pdf	Read this a long time ago and it was a little helpful.
http://www.veracode.com/security/software-security-testing	Happened upon this, no idea if it is any good. I did not get chance to really review it.
http://blogs.msdn.com/b/oldnewthing/archive/2013/12/24/10484402.aspx	Might be interesting, taken from a comment from...
http://blog.codinghorror.com/welcome-to-the-internet-of-compromised-things/	here.  Also interesting.
https://www.grc.com/fingerprints.htm	They have some interesting stuff regarding SSL but more importantly they might be a generally useful resource.
https://www.youtube.com/watch?v=wKDE_upBlfc	Complex example of a hack and how it can be done by defeating multiple layers and understanding history.

Other

Link	JCD's notes
http://blog.incubaid.com/2012/03/28/the-game-of-distributed-systems-programming-which-level-are-you	The technical subject is interesting and it provides insight into architecture. It ALSO gives a nice idea of levels and how you can’t really know what level you are at until you see the next level up. I actually have a more complex theory on this, but I have yet to write the blog post.
http://www.moserware.com/2009/07/just-enough-mba-to-be-programmer.html	Just plain useful in a practical sort of way.
http://archive.wired.com/wired/archive/4.12/ffglass_pr.html	I have not yet completely read it but it was recommended to me by a trusted source.  It’s long.
http://archive.wired.com/wired/archive/8.04/joy.html	Very long indeed, but worth reading.

You made it to the bottom?  And you counted them?  Only 74 links and you want your money back? Alright! Fine. Well I'm sure I can come up with a 75th link just for you, my bean counting friend.  How about something interesting?  Like really interesting.  Here you go: http://www.damninteresting.com/the-zero-armed-bandit/

Resumes, Lying and Embellishment

I was reading how Raji Bhamidipati had noted how many people with little experience were using fake experience on their CVs (I will use the term resume).  They had several questions on why people choose to be unethical and fraudulent on their resume.  Furthermore, she states that many of these people end up being hired using their fake resume.  These hollow resumes, were not caught, but leave the person in question of being caught for the rest of their time at their job.  This entire article is speculative based upon my experience and research, but should not be taken as a sure thing.

History and Brains

I won't pretend this is a new issue, or something that 'our generation' started, but I suspect it has gotten worse over time.  First of all, economics and social history is a part of this.  Speaking specifically of the United States, after the 1950s, most of Europe was in a economic ditch due to the destruction from World War 2.  People could easily get jobs and brought up the 'baby boomer' generation.  The US invested a lot into tech, and made a large number of advances.  Without digging deep into the history, the boomer generation slowly amassed wealth and was able to assist their children to go to college and unlike any other time in history, the US population mostly has college education.  Unlike many areas in the world, in the United States it is often the case that you take out what is effectively a long-term loan for education.  While this is overly broad, many people excepted quick success or ignore the bill until after they graduated. The US slowly moved away from manufacturing jobs, leaving us with many 'service' jobs, like changing oil and child care.  One of the few other sectors with significant growth was tech.  Also part of the cultural zeitgeist was that the boomer generation encouraged a belief in their children of being special.  With large debts from school, few places to work (that pay well) and unreasonably high expectations in life, much of the younger generation has been forced to compete in an environment they were not prepared for.  Add in pressure from social networks where your successful friends are highly visible, and lying to get a job might seem like a way out and up.

There is another real set of reasons involving the brain.  The first one is, many of those who lie on their resume are young.  They have little life experience and don't have a sense of the possible consequences.  They also don't have the brain to handle the risks.  Literally, those who are young enough don't have as many connections to the executive part of the brain and make dumber choices.  It is also why crime trends towards youth as well.  Notice how the drop off age is about 25.  About the same time the brain develops.

Another factor involves the Dunning-Kruger effect, people don't know when they aren't very good and when they are very good they think less of themselves.  Those who don't know there lack of skill think they are amazing, so they tend to inflate their resume with less truthful statements.  Worse yet, since they are lying to themselves, they are in some ways more capable of lying to others.  Those who are more knowledgeable tend to know how little they really know.  They therefore seem less capable, which helps bolster the case of the less skilled.

Employers Worst Enemy: Themselves

The final factor is the automated checking of resumes that looks for keywords in a resume.  Sometimes playing alphabet soup can help you get a job.  If you have little to no experience, alphabet soup is about the best bet you can make.  If you can back up the claims even a little it might make sense to do so.  Sometimes this is referred to as 'Embellishment'.  However, that can become a slippery slope.  In what might have been the worst interview I ever observed, in looking to find an upside, my coworker choose a technology the candidate had listed and asked him to talk about it.  The resume had categories, like 'familiar' and 'expert', with this technology being marked as expert.  I believe it was Tomcat servers, something I think most of us were not expert in and was not used in that company, so we were not evaluating his actual knowledge.  His first comment was that he only used it in a testing context, and that was how he was an expert.  So we asked how he'd find a bug in said system, or how he'd get updates deployed.  The best answer we eventually got after digging into his background was other people did everything for him.

But what did we do?  If he was smart, we taught him perhaps how to answer that form of question better without ever learning any of the tech he claimed.  In essence, even though we didn't hire him, he could learn the wrong things from that interview.  Worse yet, while he didn't get our job, we had no way of notifying other employers of his behavior.  We couldn't say how we caught him in several lies because his former employers included employers we had, so we knew some of the work was not his.

From an employers point of view, when they are first hiring, it is not unlikely they don't know what a good tester looks like.  They do some web research and come up with a list of properties that sound reasonable to them and come up with a job description.  If you follow the CDT view where things depend on context, it will be hard for an employer to ask smart questions.  Worse yet, ISTQB is about the only thing an employer might be able to use to filter with.  Ignoring my personal opinion on the value of such a cert, since an employer probably doesn't know much about testing or  ISTQB, they are not likely to go doing the validation of such a certificate.  If someone feels ISTQB is of little value, they might justify in their head, "My employer won't know the difference."

The employer is more likely to offer on the low end of the pay scale, even if they want experienced people.  Sometimes this is because they can't afford anyone more advanced, else they would already have a test team.  Other times it is because they are small and thus don't have enough cash to afford senior people.  Even if they have a test team, not everyone treats this as more than a job, and might not take their interviewing seriously.  This allows beginners in, as well as those whom lie on their resume.  In my experience, often these sorts of companies also have death-march-like development processes because the company has not matured.  This makes it so that many newer recruits end up leaving.  Another area where this happens is in contracting.  When you need 30 testers in seats and you are contracted to have them, or your pay is tied to the number of people, you have a conflict of interest between hiring just anyone and keeping the customer happy.  You might hire a few 'iffy' people because you can hide them in the middle of some better employees.

 What Do We Do?

The truth of the matter is I don't know that we can as a society eliminate this sort of low-level corruption.  I worry as automation gets better and less high paying jobs come available what sort of world we will live in.  One likely result is with increased competition we will see an upswing in this sort of falsification.

In trying to evaluate my own predictions, I tried to find any data around trending, to see if this problem is in fact getting worse.  I couldn't find anything substantial, but a variety of different percentages were thrown around and nothing that felt authoritative.  The one thing I did notice was the numbers were typically large, like 40% of resumes have some form of fraud in them.  Okay, you caught me, it was 39% with other numbers on that page lower.  Oh, and that was percentage of organizations, not resumes.  Darn, you're reading up on my data now?  I just wanted to make the number seem more round and dramatic.  What's wrong with that?  The point is, we all have to be careful with our language and clear on what we expect.  I imagine you expect that I try to get my facts straight and while some rounding might be implied, that the unit of measure would remain consistent.  For example, culturally, Latin Americas commonly include pictures with their resumes, while in the United States, that is unusual.  So if you send a resume to Latin America, you better include a picture, as that is the norm.  I know no one of us can set such expectations, but you can set them for yourself and ask about unclear parts of resumes that don't seem to follow your standard.  We have to make honesty the norm by being honest ourselves.

Obviously if you are having people ask you about advice regarding lying on your resume, the answer should be no.  I have given recommendations for people, and I have had one friend list work he did for me in his resume when I was about 16-18 years old.  He actually did do some testing, but it was not for a formal company.  I said he could because he did the work, formal organization or not.  Some people might see that sort of buddy system as flawed because you can use contacts to get jobs, but those without contacts are disadvantaged.  I have even heard claims that this can be accidentally racist/sexist, based upon who you know and trust.  The fact of the matter is, that hiring people you know and trust has always been a thing.  The reality is that hiring based upon personal knowledge of someone else's previous work is one of the best filtering mechanisms.  Hiring based in part upon the recommendation of someone you know is probably helpful.  Dr. Kaner suggested a entire packet of stuff be used to track someone's career, but that doesn't help much for those starting off.  Perhaps a broader network, like that of Dr. Kaner or LinkedIn is also of value, but that only helps vet people you know.  I recognize that some people may disagree with the idea of hiring people you have formerly worked with, and I welcome discussion around that.  I think it is better to have an open discussion than for me to stand on my soap box and expect you to just accept what I say.

There are two things we can do today for every candidate.  The one easy thing we can do is be careful with our hiring.  Get to know the candidate, ask lots of questions, and in the end, if you aren't sure, don't hire them.  I know that can hurt candidates, but I think it is important to choose the right person rather than getting the wrong person but moving faster.  The second piece is to investigate the claims of a candidate.  While I think it is better to do so in an interview, even after an interview, if a candidate makes a claim, it isn't too late to go look into it.  Better to spend ten minutes now than waste 6 months with them and another 3 months trying to get rid of them.  If that means calling the candidate to ask them a question, get on the phone and call.

While most of this has been about the interviewer insuring they are not being lied to, the candidate too should be concerned about the company.  From a candidate side, ask detailed questions, and dig into the details.  What is your bug count?  How much does it vary?  What flaws are there at the company?  What strengths?  And again, don't assume the interviewer isn't embellishing in their answers.  I don't think any interviewer would say "It's hell here.  Go somewhere else."  So investigate the company and the workplace environment.

Ultimately, you have to work with either the candidate or the people interviewing you for 8 hours a day, for a long time, so choose wisely.

Heroes and Villains

Heroes

I have a host of people that I read and read and read.  These are my literary Heroes.  They constantly give me new gems and bright insights.  I pale in my work before these folks.  The sad news is that my heroes have slowly slipped away and all I have is their tremendous bodies of work.  In the blog-arena, I really appreciate Jeff Atwood, Joel Spolsky and Steve Yegge (who has multiple blogs).  Just as an example of my reading, I am all the back to 2005 in Jeff's blog, reading it backwards entry by entry.

Now two things you'll note from that.  One, I really enjoy what developers have to say and I really don't have a lot of test-related bloggers I hit on a regular basis.  Even if I start widening my knowledge net, Martin Fowler, Scott Hanselman and those Dot Net Rocks podcast guys probably has beaten out other test-related heroes I have read.  Now I do tend to focus on what they have to say on test when they have something on test, but I spend way more time on the development process than on capital-T Test.  These guys are brilliant and have wonderful things to say.  They come up with all sorts of clever coding patterns and practices.  They consider the process as a whole and care about the craft.  They are Heroes.

Then I should consider my list of more generic Heroes.  Heroes of science like Richard Feynman, Heroes of science fiction like Robert Heinlein.  Heroes of thought like Rene Des Cartes, Heroes of humanity like George Bernard Shaw.  These are all men I have read and found to have enlightening things to say, although some are rather obscure.

Right now, this is how I feel about test and "Heroes".  There are no Heroes.  There are a few knowledgeable people, but the fractured nature makes it hard to pin down.  Most people who come into test come into it by 'accident'.  We are still forging paths and hitting dead ends.  Yet we do have another aspect.

Villains

Perhaps our lack of Heroes is the nature of our business, so at best we get Villains.  Well, Villains are cool, right?  Who doesn't like a good Super Villain?  The Riddler or Magneto come to mind.  They have super powers, they do cool things and in the end, they often feel justified in their deeds.  I hesitate to call anyone a "Villain" in a community I work in.  Worse yet, CDT might be so accepting that Villains are considered good guys in their own way.  Now if I was James Bach, I would have "“depraved” enemies".  Now does that make someone like Rex Black a Villain in Bach's eyes?  Is the visa-versa true for Black?  Am I a sort of Villain of the CDT community for asking questions?  In talking with one senior-level tester, I could be seen as a Villain.  The all-mighty dollar of consultancy demands for true purity of our testing Scotsmen and my questions for CDT is dangerous for those dollars. I won't answer these questions regarding villainy for you, but I do want to be clear, I am not calling any of these gentlemen Villains.

The downfall of our method is that we don't have any golden boys of testing.  Instead, we challenge each other to get better but don't have an easy way of showing off our skills.  I can't even personally say if Kaner, Bach or Black are good testers.  I can look at some of Atwood, Spolsky and Yegges code and judge them, as that is often visible.  Testing is often an invisible task, not one with an output easily examined.  I think some of the information of the test community can be useful, but I don't use all of any practitioner's 'methods'.

I am left with two things.  Are there any real Villains of testing and can we have Heroes in testing?  How can I evaluate a given person's skill in "Test" compare to writing or management or bug writing or all the other skills that make up "Testing"?  Even with testing an application, it often requires parameters that are hard to control, like the versions of software, builds, time, memory, threading and all the other 'impossibility of complete testing' pieces.  So even with bug reports, test cases, etc. it might be impossible for me to truly evaluate the output.  If I can't know (there are plenty of heuristics, but actually knowing is much harder) someone is a good tester, then I guess I'm just left with is a Villain just a perspective or are their absolutes?

I don't know that I can answer that, but maybe I'll look into getting an outfit, just in case.

Interviewing, post 3

TLDR: brain dump.

This is my internal thoughts, presented for you to look at. (cause I want to remove my internal stigma of having to have perfect / point-filled blogs before I post.)

Little background: I used to interview people at a much larger company, we had a team of roughly 50 testers / SDETs. This allowed me a certain amount of freedom in hiring people that might not perfectly fit any particular profile. Aka they could be higher risk (in my mind) in certain key areas, cause they were offset but a large group of other people. Or we could move them to a team that offset any other risky areas while we saw if they panned out over a 2-3 month period. Or were able to learn what we thought was necessary.
I now work for a company with a 4 person QA team, hiring a final person for this year to make it a team of 5. Personally I think this gives me significantly less leeway on what it is that we can hire. I have less wiggle room for potential issues. I don't have the cycles I would like to, to devote to training someone with only some potential. Basically what I think I'm looking for and what I need to fill are more tightly coupled for this current position.

So, in interviewing people, I look for certain key talents. One of those talents is self-motivation / drive / passion. In an effort to attempt to figure out what I'm looking for in good people to hire, one of my employees asked me to define what I meant by motivation or drive.

Before looking it up in a dictionary:
Motivation: A reason to do a task.
Drive: Internal motivation, requiring little to no outside force(s). This can be synonomous with Self-motivated.
Passion: A strong drive that allows one to singularly focus on a task or set of tasks. Passion can be good and bad. Not allowing oneself to defocus when necessary (OCD).

Dictionary (the definition I thought the most pertinent):
Motivation: a motivating force, stimulus, or influence : incentive
Drive: to press or force into an activity, course, or direction
Passion: a strong liking or desire for or devotion to some activity, object, or concept

What does all this mean to me? I think my ideas of what motivation / drive are…are pretty reasonable (perhaps self-motivated is the more appropriate word). Now the real question comes down to HOW do you find out if someone is self-motivated or driven?
I've been reading Hiring with your Head (1) recently to see how someone regarded as a great interviewer goes about it. Adler likes the idea of past proof of having done it. He asks, "What is your most significant accomplishment related to X?" 
Personally, I'm not sure I want to just ask someone directly, "Are you motived? Prove it." People can make up anything if they know what you are looking for.

Lately I've been saying something to the point of:
One of the key objectives for this position is the ability to write bug reports that are clear, concise, accurate and relevant. At a high level, what do you think about this and how would you go about this? What have you accomplished that's most similar to this?
One of the key objectives for this position is the conversion of component, functional, system and regression tests into automation. At a high level, what do you think about this and how would you go about this? What have you accomplished that's most similar to this?

And then digging into the details of the given answers to get specific details. I have found this seems to give me the data I need to determine "is this person self-motivated". But since I just changed up the interview questions I use to start taking this into account, I'll reserve judgment till later.

-------------------

(1) Adler, Lou. Hire with Your Head: Using Performance-based Hiring to Build Great Teams. 3rd ed. Hoboken, N.J: John Wiley & Sons, 2007. Print.
http://www.amazon.com/Hire-Your-Head-Performance-Based-ebook/dp/B000SEOVH6

My interviewing start and the changes I've made.

tldr: Changes in how you interview is inevitable.

I started interviewing at my first job. It was a gig with a small company (<20 people) and I was the Engineering Intern. No, I still don't know what that meant, I just did everything anyone asked me to from hardware to software to fetching donuts. I was included in interviews there, but not as a participant, I was asked to only be an observer. They were very traditional. What tech do you know? Do you know this specific tool? And not much more then that. If everyone got along with the person, and they had decent answers, they were hired.

I'm moved through several companies now and moved from observing to influencing to being the final say about who gets hired. Initially, I started with the traditional questions, but they resulted in people that didn't always work out. Then I started to experiment with hiring. What questions should we ask? What answers should we expect? I'm grateful to the employer at the time who allowed me the freedom to experiment with how to judge what people to hire.

While this data doesn't cover all of my hiring, it is an 18 month period where I kept fairly detailed records while I was experimenting heavily with interviewing.

  - 250-300 phone screenings (don't remember an exact number, as there were untold tons of them)
  - 50 people made it to a physical interview
  - 1 person disappeared during the interview process (moved to Texas I found out)
  - 25 people made it to a second interview, 2 of those people jumped straight to 2nd interviews as they came from recommends of people I trusted, or I had already directly worked with them.
  - 19 people got offers
  - 3 people declined
  - which means after interviewing ~300 people, we hired 16 people.
  - 4 of those people didn't work out.

I'm not giving you this data to tell you how bad or good I was at interviewing, but to allow you to see what the process was like for me. I had to phone screen every resume (300, ~150 hours), talk with 50 people once (minimum 25 hours), setup a team of 3 people to talk with another 25 (minimum 150 hours, I count prep time and debriefing time), then extend offers, then worry and wait. 325 hours to get 12 people that worked out. That's three quarters of a week to find each person. 

What is the point of an interview? To find the most qualified candidate for a given job. Or at least for now that's what I'm going to use.

So at first I started with question like I had observed. Why do you want to leave your current employer? Do you know Agile? Do you know C#? Write me an algorithm in some language. Plus a host of other tech skill questions. Over the course of an interview I'd get a feel about someone from body language, terms and sentences they used, and then extrapolate information from the way they answered.  This system worked semi-okay for 1-2 years, but about that time I noticed that I really didn't care 'what' they answered, I wanted to hear 'how' they answered. I was just asking questions to get them talking, trying to get them to talk long enough to understand how they thought.

That's roughly when I started to change up the game. It took about 6 months of on and off introspection to come to the next level. Why was I using these questions? What was I really looking for in these people? After reading several books (Strengths Finder and Good to Great) I had a more solid idea of the traits I wanted to see in people: character, work ethic, intelligence, responsibility and values. Observing people I knew that were good testers, and people I knew that were bad, I clarified my definitions of those traits. Then I tried to design questions about those traits that I wanted to investigate in each interviewee. 

It's not that I don't ask tech questions anymore. Merely that tech questions are not my first level weed-out questions. I have found over and over again that personality and mentality are the most important parts. It's nice if you have tech skills, but I can judge your ability to learn tech skills independent of your personality and mentality at a second interview. And I'm more likely to hire someone with the right mindset and little skills, than I am someone with the wrong mindset but the the right tech skills.

*****************
As an example of how I would change up interviews (occurred just today):

As I was sitting in the bathroom, I noticed that this plastic piece had shimmied down the hinge pin and the lock was wiggly. I pulled out my swiss army knife and moved the plastic piece back into position and tightened the 2 screws holding the lock. After I did this it occurred to me, is that normal behavior? Do people just fix things when they notice them broken, or do they just complain to someone else?

So my new thought on how to change my current test interview. Put something in the room that anyone would want to fix. Something easy to fix, but not so weird that someone in an interview wouldn't want to do it for fear of looking strange. A marker with a cap off is my first thought. Do they notice it? Do they fix it? Does that make them proactive?


So I brought this test to my crew of testers. They of course shot me down, like all good testers would. What does being proactive prove? That they have OCD?
Probably the best rebuttal of this was Wayne, "talents and(sic) patterns of thought ... you(sic) can't determine a pattern from one event". 
Then JCD piped in with "Is that as much cultural behavior(sic) as otherwise? Some people think it is better to be polite, or maybe that is some sort of test tool you will use later?"


I'll still be using the marker test in my next couple of interviews. However, I won't use the data I collect from it. It might end up being useless, it might be an indicator of something I wasn't looking for, who knows.

Lastly, I'd like to point out, this interview thing is a constantly evolving subject. Finding good people is difficult and making it easier is a goal of mine (short of just opening a training school). Lately, I've dived into blogs about hiring and have 2 new books, by Lou Adler, I want to read on the subject. Stay tuned...

Testing the hiring process for testers...

 tldr: Hiring is hard.

  Having now interviewed hundreds of people for testing positions. I can say, "hire the people you, or someone you trust, knows." Learn to adjust who you trust based on interviews you do, and the people they recommend. Occasionally you'll get a diamond in the rough and you'll interview someone who you walk out saying "please stay here, I'll be right back" and off you run to get an offer letter made quickly. More often than not you will walk out saying "another 30 minutes lost." It's all the people in between that are the hardest pieces to sort, people that are pluses on some things but minuses on others. Which pieces are more important? Can someone be a plus here and does that offset the minus over there enough to hire this person? If you're on the fence about a hire, say no, trust your instincts. I think this one piece has saved me more then anything else.

A good testing person is hard to identify.
   I've always been a 2-stage interview person. During the first interview ask some personality questions, some company culture questions. See if they are even going to fit within your company / team. Then I dive into tech questions until I get 2-3 "I don't know" answers, this is where I end most first interviews. Second interview I open with the 2-3 questions they missed, if they didn't research them or have a decent answer the second time that's a major strike against them.
  I've also tried the more overt tactic of telling an interviewee who I thought was good enough for a second, "Here are two topics, I would like you to research one and we'll talk at your second interview next week." The first person I tried this on blew my mind with what she was capable of learning (over a weekend due to a miscommunication with HR). She learned both topics and to a depth that I'll admit I wasn't able to verify if she was correct. We hired her immediately, which is what might of set me up for a serious failure of the next person to get this same test. The second person came back the next week and gave me nothing. They didn't even have excuses for why they didn't learn anything. I was more then disappointed, I was dumbstruck. Here I had given this individual the perfect chance to shine and they had slapped me in the face. I stopped performing this overt tactic as it caused me too much stress to think that people like that existed.
  Over time the questions have evolved from technology to one of personality, intelligence, self-motivation and open ended questions. I have written across the top of my interview sheet "Ask the question, then shut-up Isaac". I've had so many people impress me with their answers to simple questions, and just as many burn themselves when I just listen. Questions like: 
  "Tell me about the most memorable bug you've found."
  "Describe a difficult co-worker and how you handled it."
  "What causes bugs?"
  "How have you improved your testing skills in the last 3-6 months?"
  "How do you prefer to communicate to developers / Product Owners / management?"
  "Describe a tool you use, and how you would improve that tool." 
  Now none of these questions are better than others, they all fit different people and different situations. sometimes I have to ask a variety because someone is clearly not interested in one of two of them.

Hire for the position:
  If you are hiring for a senior level position the person who is going to fill that better be able to be shown a problem, left alone for 1-3 months and when you come back *poof* results. If you are hiring for a junior level position remember YOU are the one responsible for showing this person the ropes, and  responsibilities, through a mentor; and that's you or someone you assign to do that. 
  Don't forget individual team dynamics. Some teams are uber-quiet get-work-done types, bringing in a loud obnoxious likes-to-talk person probably isn't the best idea.

Test the process:
   Keep the resumes of everyone you hire, along with your notes on what they said when you interviewed them. That way, as time progresses and you find out who the really great testers are, you can look back and see what they answered for questions you asked. Same for people who turned out poorly, you can look for patterns in how they answered. If your people really, really trust you re-interview them with new questions and get a feel for how your rock-stars answer these types of questions. Do this with caution as most people will freak out if their boss asked them to re-interview.

A team of interviewers is required:
  While I generally perform the first interview, no one gets through the gate without at least a second set of eyes. I try and get at least three other sets of eyes on a candidate, depending on position, seniority, skill set and any other number of attributes. 
  This was sunk into me on the first time I passed someone through a first interview, just to have a squad of people I trusted walk out of the second interview, and none of them would look me in the eye. I knew immediately that they wanted to know, "How the hell did this guy get this far?." Truth be told though, I still think the guy was a decent candidate. However, with three current employees I trusted saying "No", I couldn't ignore them because I asked for their opinion.
  With a team of interviewers it's also handy to have a code phrase, that instantly ends an interview. When the phrase is said, you allow the candidate to ask some questions so they feel comfortable then end the interview. That way no one wastes time on someone that is a hell no. This practice is mainly for those who are uncomfortable saying, "Look, clearly you don't fit, thanks for coming in, goodbye".

Know when to break the rules:
  Yes, I've ignored my own rules as I've laid them out, but I've hired over 50 people now. I've learned when a position I'm filling can take a certain amount of risk on a new tester that is going to require lots of training. But that was always explained to the candidate while making them an offer. Yes, some of them got offended cause they had "5 years experience" which amounted to nothing for me. Not because 5 years is nothing, but clearly their 5 years was only not spent on what I needed them to know.

  Apologies to all those people who have endured my interviews. Mainly to those who I have hired and then used as guinea pigs to find the next level of great testers.