Link Mania: 75 Links with Commentary on Testing, Security, Tech and Life

I have a friend who wants to get into security testing and asked for my help in broadening their education.  I assumed that my friend would have a good number of links on security, and so I would try to start with a broader set of testing knowledge.  They have a degree and know basic computer science, but in discussing needs, I decided that my primary goal would be to provide a broader understanding around testing.  I started by emailing some links I thought might be useful... but that quickly became clear that it was less manageable, so over a few months I put together groups of links I thought might be of value into a Google Doc.

I also wrote some notes on why I thought they were of value for my friend.  It should be noted that I cite myself frequently.  This is not because I'm full of myself but because I addressed specific questions my friend had and felt that my research + links I had in my posts would be of some benefit.  I did do some light editing of my notes, removing personal comments for privacy reasons, but this is roughly what I came up with.

One limitation to this sort of document is I could go on forever.  I certainly could have written several dozen posts around these links.  Before I turn you loose to all these links, let me give some advice.  I look in detail for things I need to know soon and skim general topics for data I might value in the future.  It’s a method that allows me to work well with a large set of disparate data.  It’s why I can talk about all sorts of subjects at some level, but am not an expert in any.  Knowing who actually is worth reading helps me filter the 'must reads' from the 'skims'.  So as you look at these links, look for what you want to learn about, and then look at the author.  If the data is useful, look for more of that author, even links I did not suggest. If it wasn't helpful, try a different author.  If you try about 3 different links in a row in this set, and all are not useful, either you are not primed to learn about the subject, or the type of data/authors I have gathered are not optimal for how you learn.  You will have to decide which.

Happy reading!

Links I Came Up With Before I Started Organizing [Mostly Test]

Link	JCD's notes
http://about98percentdone.blogspot.com/2015/01/leveling-up-your-testing-skills.html	I literally wrote this for people trying to improve themselves.
http://bbst.info/?page_id=23	Lectures 2,3,5,6 in particular.  They are about 30 mins per lecture.  Great testing stuff. The lecture has power point slides, which can be downloaded and are useful by themselves, HOWEVER, he talks about different points not just captured in the slides.
http://about98percentdone.blogspot.com/2013/09/where-cdt-fails-rebuttal.html	Isaac’s effort to describe what testers should learn in ‘levels’. Good stuff.
http://www.testingreferences.com/software_testing_bookstore.php	The books I read that they recommend are good, thus I trust their recommendation enough to suggest taking a look at the list.
http://www.developsense.com/blog/	An interesting thinker in the testing space.
http://www.satisfice.com/blog/archives/1346	James Bach, one of a hand full of people to affect the modern software testing world greatly with his words.  In one of his various attempt to define testing.  This one is of particular interesting, because it is about the art and act of doing testing.
http://about98percentdone.blogspot.com/2013/10/reflections.html http://about98percentdone.blogspot.com/2013/11/testing-with-reflections-part-ii.html http://about98percentdone.blogspot.com/2014/06/my-current-test-framework-testing-large.html	Reflections are a mind-expanding technique to make code think about itself.
https://sites.google.com/site/steveyegge2/godel-escher-blog https://sites.google.com/site/steveyegge2/is-weak-typing-strong-enough http://steve-yegge.blogspot.com/2008/05/dynamic-languages-strike-back.html http://steve-yegge.blogspot.com/2006/10/egomania-itself.html	I really love Steve Yegge, he maybe one of the best writer-programmer combinations I know of.  I come back to his work frequently.  But he is a programmer, and as such, the one that really matters is the top link.  The rest are very programmer centric.
http://blog.codinghorror.com/	Heard of Stack Overflow?  This guy made that (along with Joel Spolsky of Joel On Software, but Joel’s stuff is getting old).  He writes on a variety of topics and on his good days is really good.  That said, he’s still a developer, thinking like a developer.

Research in Life / Happiness / Living / The Mind / General Career Advice

Link	JCD's notes
https://www.youtube.com/watch?v=iCvmsMzlF7o http://www.ted.com/talks/brene_brown_listening_to_shame?language=en	These have made me think about how I live my life.
http://about98percentdone.blogspot.com/2014/10/noise-to-signal-attempt-to.html http://about98percentdone.blogspot.com/2014/06/what-is-highest-level-of-skill-in.html	Learn about what you should do to make yourself more valuable in life.  Learning about what matters in learning.
http://about98percentdone.blogspot.com/2014/02/being-fraud-and-failure.html	How to deal with feeling like you don’t know enough, when in fact you’re driven to know more. I have referenced this exact blog post more in my comments to other people’s blog entries than any other.  Often the people who care the most feel this way.
http://about98percentdone.blogspot.com/2013/12/book-consideration-introduction-to.html	Read the bullet points at the bottom.  In particular, “The Answer”.  The book reviewed is a little bit of a personal and spiritual look at science.  Somewhat like Sagan, but revolved around thinking.  You might like the book.
http://about98percentdone.blogspot.com/2013/09/testing-hiring-process-for-testers.html http://about98percentdone.blogspot.com/2013/09/a-subject-of-hiring-process.html http://about98percentdone.blogspot.com/2013/09/my-interviewing-start-and-changes-ive.html	Isaac and I wrote a little bit about getting hired. These are some of my early blog entries, but you also get both the view of a person being hired by Isaac as well as, Isaac, the hiring manager's method of choosing to hire people like me.
https://sites.google.com/site/steveyegge2/what-you-need-to-know https://sites.google.com/site/steveyegge2/miracle-interview	More interview related thoughts, but from a developer side.
https://sites.google.com/site/steveyegge2/age-racecar-driver	(same guy as above, but more philosophical questions in specialization.)
http://about98percentdone.blogspot.com/2013/10/interviewing-post-3.html http://www.stickyminds.com/article/developing-team-testers http://www.stickyminds.com/article/helpful-tips-hiring-better-testers	The last set of articles Isaac and I have on interviewing.
http://www.moserware.com/2009/01/wetware-refactorings.html	Fairly good.  Almost all true.  Interesting ideas and good set of resources.
http://www.moserware.com/2008/03/what-does-it-take-to-become-grandmaster.html	I would give this a lower priority because it’s long and this is not the first time I have found someone who says this and put it in the list.  Hopefully you see the patterns… and will learn.
http://breakingsmart.com/	An interesting description of the tug between the past and the future.  It seems to be a little dismissive of some problems, but the general picture is not wrong.
http://blog.codinghorror.com/level-5-means-never-having-to-say-youre-sorry/	I hate the title, but the content is pretty good. It provides some insights into why generating reactionary, scripted systems does not scale well in creative work.
http://www.paulgraham.com/usa.html	Paul Graham is fantastically interesting in general.  This is an interesting attempt to correlate technological outcomes to culture.  I recently heard from a friend who lived in Japan for a year about these Japanese workers at this company, whose company was bought out by an American company after it started to fail, had a real difficult time letting go of the quality of a product in order to get the product out the door. That story feels like a sort of quick CRC check for me, meaning this article probably does have some veracity.
http://www.ribbonfarm.com/the-gervais-principle/	I know I have talked about this before, but it is an interesting model of human behavior.  While I’m not actually a fan of The Office, I found it was mostly ‘translatable’.  It also offers small notes on the Peter and Dilbert principles which are also worth looking at.

Testing

Link	JCD's notes
https://www.youtube.com/watch?v=j_JviA5nvS0&list=PLSIUOFhnxEiDFckNDSjKWqOCtd8ksJrh4	GTAC is fairly good.  Some of the content might be on security. I have not watched all the 2014 (and soon 2015) videos, only the 2013 videos which were often very good.
http://www.associationforsoftwaretesting.org/conference/cast-2015/	CAST 2015 is probably worth watching. I have yet to watch it myself, but I attended last year.
http://www.testingreferences.com/testingnews.php	I use this as a tool, looking for things of interesting, read them, keeping note of who wrote it and if I decide I don’t like someone (for any reason), I mentally filter them out of the list.  It’s the firehose method, good for going in all different places, but you never know what you might get.
http://blog.codinghorror.com/a-visual-explanation-of-sql-joins/	How SQL joins work. They are often used both for getting data for testing as well as part of how many reports are generated.
http://angryweasel.com/blog/	In broad strokes, I agree with Alan and appreciate his perspective with about 20 years at Microsoft.
http://oredev.org/2013/wed-fri-conference/balancing-atdd-gui-automation-and-exploratory-testing	A video I watched long ago that is still in my notes.  Might be interesting from an automation vs exploratory testing perspective.  I have little memory of it though. Most people only teach the basics :(
http://www.huibschoots.nl/wordpress/	Like Alan, I agree with Huib broadly, and I find his view very interesting as he comes from a European background.

General Tech

Link	JCD's notes
https://howdns.works/episodes/	How DNS works.
http://blog.codinghorror.com/on-software-engineering/	Why software engineering is so difficult, and why consultants are sometimes looked down upon.
http://www.dreamincode.net/forums/topic/223324-an-interesting-interview-with-steve-yegge-and-james-duncan-about-java/	Interesting set of interviews, the biggest point is that you should know your tools well enough you know what is wrong with them.
http://www.moserware.com/2009/03/how-net-regular-expressions-really-work.html	I think the data here is interesting, however, the meta is important too, even at a security level.  A DDOS attack is a security attack, but if I can DDOS you with one click, it is a security issue too.  Knowing how regular expressions work tells you about how a lot of systems work inside, which is how samy was able to defeat the defenses of MySpace ( http://namb.la/popular/ ).  It’s also a huge part of getting things done.
http://edge.org/annual-question/what-do-you-think-about-machines-that-think	An interesting set of speakers, most very smart, all on the same topic.  The 2014 topic was interesting too. This might generate some interesting questions, such as: What would happen if/when the #1 job goes away (Driving, at least in the USA; currently biggest economy in the world; last count ~ 3 million souls in the US do this)?   What do we do with these people?   What of those that aren’t capable or interested in more mind-oriented work or who are too old to change careers?   What of my grandmother who doesn’t interact well with online work, in part because her hands are too crippled to do much with keyboards and did not grow up with this world? What happens when machines categorize someone as an outlaw in some countries which others counties would recognize as moral and legal (such as being gay, or a political dissidents)? Who is legally and morally responsible for a 'thinking' machine?
https://sites.google.com/site/steveyegge2/nonesuch-beast https://sites.google.com/site/steveyegge2/math-every-day	Here is a mission in life -- how do you solve problems in which there is no solution?  (My only hint is find another problem.)  However, the point is, solving one of these non-trivial problems is a lifetime’s worth of work, if you want it.  See the intro of the second piece to see what I mean.
http://www.amazon.com/review/RUGSCP3XBNBUV	Suggested by Yegge’s post… it’s an interesting read in and of itself.  It’s got some ideas I’ve not really researched.

Security

Link	JCD's notes
https://www.youtube.com/watch?v=n9-Gz1U87CI&index=9&list=PLQB4l9iafcelXpJnK6IyDsoFeEb1icqrl	Brilliant! I attend this live and you’ll see me in the end asking a question.  All the rest of the talks (I kept the list) are test related, but this is security related.
http://krebsonsecurity.com/ https://www.schneier.com http://taosecurity.blogspot.ro/ (Others…) https://heimdalsecurity.com/blog/best-internet-security-blogs/	Famous security researchers blog.
https://www.schneier.com/blog/archives/2012/07/how_to_become_a_1.html https://krebsonsecurity.com/category/how-to-break-into-security/	How to learn to be a security professional.
https://www.schneier.com/blog/archives/2013/04/nice_security_m.html	Why we are all frauds and failures sometimes (As address in a link above).
http://www.moserware.com/2009/06/first-few-milliseconds-of-https.html http://www.moserware.com/2009/09/stick-figure-guide-to-advanced.html	A *practical* piece of tech that really applies around security of the web, he has other good posts too.
http://www.guru99.com/what-is-security-testing.html http://www.guru99.com/learn-penetration-testing.html	I liked the graph, purely from a ‘what do you mean by “security testing”’ question. NOTE: My friend suggested http://www.techrepublic.com/blog/it-security/the-five-phases-of-a-successful-network-penetration/ might also be useful.
https://www.cs.purdue.edu/homes/xyzhang/fall07/Papers/sw-test.pdf	Read this a long time ago and it was a little helpful.
http://www.veracode.com/security/software-security-testing	Happened upon this, no idea if it is any good. I did not get chance to really review it.
http://blogs.msdn.com/b/oldnewthing/archive/2013/12/24/10484402.aspx	Might be interesting, taken from a comment from...
http://blog.codinghorror.com/welcome-to-the-internet-of-compromised-things/	here.  Also interesting.
https://www.grc.com/fingerprints.htm	They have some interesting stuff regarding SSL but more importantly they might be a generally useful resource.
https://www.youtube.com/watch?v=wKDE_upBlfc	Complex example of a hack and how it can be done by defeating multiple layers and understanding history.

Other

Link	JCD's notes
http://blog.incubaid.com/2012/03/28/the-game-of-distributed-systems-programming-which-level-are-you	The technical subject is interesting and it provides insight into architecture. It ALSO gives a nice idea of levels and how you can’t really know what level you are at until you see the next level up. I actually have a more complex theory on this, but I have yet to write the blog post.
http://www.moserware.com/2009/07/just-enough-mba-to-be-programmer.html	Just plain useful in a practical sort of way.
http://archive.wired.com/wired/archive/4.12/ffglass_pr.html	I have not yet completely read it but it was recommended to me by a trusted source.  It’s long.
http://archive.wired.com/wired/archive/8.04/joy.html	Very long indeed, but worth reading.

You made it to the bottom?  And you counted them?  Only 74 links and you want your money back? Alright! Fine. Well I'm sure I can come up with a 75th link just for you, my bean counting friend.  How about something interesting?  Like really interesting.  Here you go: http://www.damninteresting.com/the-zero-armed-bandit/

Expectations...again...

This was written a long time ago...but finally edited to readability.

So I just finished the AST BBST Instructors course. In it, we started by introducing ourselves. I of course summarized my introduction from this blog cause I was being lazy, and it said what I thought I wanted to say.

In response to that I was asked:

"It sounds like you have the confidence and experience to be a manager, but I wonder if this might make it hard for you to teach? What do you think?" - Another Student

I responded with:

"Yes I find that my mental model has a hard time bonding with people who aren't also self-starters and/or newbies. However I've been specifically working on how do we 'as a testing community' get the first level of people into testing. I still have a problem dealing with people who just want to be spoon-feed information.
My experience is mainly in taking people with 3-5 years of experience and tuning (yes tuning) them into the next skill they need to be the best at their current position. However so far this has been limited to where I work, as I usually understand the context enough to say "if you could do this" it's the next skill you should learn.
So part of my goal is to be able to teach less experienced people. But I still struggle with if I should TRY to deal with non-self-starters…"

I’d like to take a more detailed look at my personal thoughts on the matter. Specifically around, “Does your own attitude and personality cause you difficulty in teaching people?” I’m going to try and detail an internal dialogue I’ve been having about this lately. I first tried to sum up some feelings in an older blog about Expectations.

This is what I mean when I say, "not-self starters". There have always been people at places I've worked that are comfortable doing just the basics, show up at work, do their job, go home. These are the people that when asked in an interview "What do you do to improve your skills", have an answer similar to "On the job training". These are the people that don't seem to want to take time outside of work to improve their skills (which to me translates to "I don't consider this job a career"). Some of these people seem bright, others are merely content at getting along and that's enough for them.

Then there are the self-starter people who will go out of their way to learn new things. These people ask you questions, and when you say, "tell me what you know about it so far", they blow your mind with the research they have done. Or alter your understanding of the subject with some new pieces that you haven't heard or seen of yet. Sometimes at a minimum they have clearly researched and/or understood the material, but don't change your level of knowledge.

I know mentors / leaders want to use terms like taught, educated or instructed. But when you have self-starters, that isn't the appropriate wording. The closest I've come is tuning them. When I chat with a self-starter cause they are asking "What should I learn next?" (and them asking is one of the key points). There are a couple of ways to encourage them. And generally that is all you can do. Point in a direction, say 'that way', and then get out of their way.

• Have them learn a new skill. Sometimes these are easy for beginning self-starters. Don't know SQL, yep I have yet to meet any tester that doesn't benefit from knowing it. As the self-starters get more and more experience, this can get harder and harder to find for each individual. Currently I've had serious success with just allowing them the freedom to find new things. This is the real power of the self-starter, they aren't okay with sitting idly by and surfing reddit, they WANT to provide value, they WANT to solve problems.
• Have them level up a skill they are already strong in. This helps those who have just finished something rigorous, demanding or seriously mentally intensive. (This can be work related like finishing a project or a mentally challenging class, or a non work related life changing event.) It allows them to learn something, but with the permission to not be as intensive with it. An example would be having someone learn the singleton pattern, instead of a whole new programming paradigm like LISP.
• Have them up a skill they are weak in. I don't normally recommend this for very many people. I tend to follow the Good to Great idea, that weakness isn't inherently bad. But there are genuinely some weaknesses you need to either compensate for, or bring up to a minimum bar.

Sometimes even that isn't enough. I recall one situation where I was attempting to tune someone I classify as a full-blown self-starter. We were going over the OSI model, and I was attempting to explain how each layer could be an attack point for testing. Unfortunately we were not speaking the same language, after about 30 minutes, they were frustrated and on the verge of crying. I was frustrated and not understanding why they couldn't get the 'simple concepts'. We parted that day, and I don’t recall us ever trying to train together again. I took this as a personal failure…how could I not explain such simple concepts (which OSI is and is not) to someone who tries really hard when they self-educate and generally succeeds? It took me a long time to understand that I had a base of knowledge about computers that they had never had exposure to. Bad on me for assuming what they knew. The problem was that I know stuff they didn't. They also know stuff I don't. It takes time and effort to truly get to know people. I hadn't taken the time or maybe didn't have the perception to understand that they had no idea what I was talking about. They didn't have the trust to tell me 'What the hell are you talking about'.

S'long as you learn to temper what you tell people you want, with realistic acceptance of what people can really accomplish. It all comes down to do you trust your people to work hard, do they trust you to not hose them with unreasonable expectations. It takes some serious time to build that trust. It's why you see people follow a strong leader around to companies.

If you don't fall into the category of self-starter as I've laid out here. There is still hope, you can become one. Start today, motivation breeds motivation, find something interesting, and learn it. When you're done learning that, find another. Then repeat ad nauseum.

I've going to wrap up and caveat this entire article with a thinking exercise for you.
Can your expectations be too high? Is this a bad thing?
How else would we ever achieve something new?

Introspection on why I left...

This is a story about a job long, long ago. 
In having gone through this, I developed a system of heuristic flags to look at an organization that you work for. While these are context sensitive, I tried to keep a little of the context in the flags to give the reader some idea of what I saw. Seeing these things as they happen is extremely difficult. When you are in the moment they can look like a compromise, yet remove yourself from the situation for 2-3 months and suddenly they become “OH MY GOD, I can’t believe they even asked us to do that!”
The reason that some of us persevered through lots and lots of these yellow and red flags is many. People don't like change, interviewing is change. People like to see things through to the end, quitting feels like you failed. I genuinely liked the people I worked with, or at least lots of them, but that doesn’t change the culture of the sociopaths in charge (Gervais Principle). Hindsight is 20/20, looking back, I should have quit way earlier, but recognizing the signs in real time is much more difficult then it would seem when writing this down years later.

(the stuff in italics are the warning signs)

It all started slowly. That’s the best way to disguise failure. It’s how we can convince ourselves, “we're just learning"…
I had been working for this company about three months when it was decided that we needed to go Agile. We hired 2 product owners did a couple of days of training. Made a backlog, estimated, planned a sprint and brought new team members on board. We didn’t have a scrum master but we ‘were’ looking for one. Can’t have everything at once we told ourselves.

Yellow Flag 1: Starting a process by ignoring pieces of the process before trying them is folly.

We start this grand adventure with a meeting where it was deemed, the biggest oldest crappiest hard-to-read, difficult-to-modify code would be the starting point for the new hotness. Not lets rewrite it, but take the ‘scroll of insanity’ as it was called, and use it to build on for the new hotness. But we sallied-forth, new process, one bad decision won’t kill us, and we plodded along for a couple of months. 
Then the QA manager had a life event and never recovered, he left the company and the rest of this story is sans QA having a voice. We told ourselves we can work without a manager ‘for the team’.

Yellow Flag 2: Working without a direct supervisor for more then 3 months. 

Don't get me wrong I don't want to work for just any manager, and neither should you. But if a decent (notice NOT GOOD) manager doesn't show up within 3 months, I can think of 2 things. The company doesn't attract quality people. Or they aren't making it their highest priority.

Soon our VP was walking through the building explaining to other people / board members the agile process in exquisite detail. Basically lying to their face, in front of ours.

Red Flag 1: When management tells everyone including you, we are Agile. Then proceeds to prevent you from being a self-directed team, doing backlog estimations, working on a single sprint at a time or defining done. Now that I see this, I wonder if it was a sadistic way of daring us to challenge him.

At this point we had a new development architect team. They had a working pre-alpha. As a concurrent B2B web application the VP decides we should load test this. I can handle a new challenge and was excited to get started.
First round of testing showed two concurrent users crash the new hotness. Spent a week tweaking on load testing cause clearly I’m missing something with the load test. After showing the development manager and VP the results. I was called a liar, and the load part of testing was given to someone else. Two weeks later he has the same results I do, and the development manager admits maybe there is an issue in our session management.
Meanwhile I'm off to functional automation. Automation is brittle and the 3rd party GUI libraries aren't helping. No hooks, dynamic id’s, I can’t work with this.  I of course had no manager to explain this to, but was informed, “It helps the developers move faster.” Which just puts QA, without a voice, further behind…

At this point the only thing keeping me at the company is the Warhammer Fantasy board between a developer and myself.
A co-worker and I start to share any job openings we find in an effort to get out, but due to the recession, there isn’t much.

As all this is happening, we move to production with two active sprints. In and of itself wasn’t insane until you note that shortly we had 3 active sprints in production with only two sprint teams. At one point we had three active sprints in production being worked on with bugs, an active sprint in stage, one in QA and a sixth in development.

Yellow Flag 3: Is having multiple sprints active per sprint team. 
Red Flag 2: Is having six different sprints being actively worked on, by two sprint teams.

We had a sprint board which was viewed as sacred ground, only the PO could touch, move, write, etc. on or with it. At one point one of the testers called the board the joke that it was, and said that without the ability to move, edit or touch things, it was stupid. He said this to the POs face. Within 10 minutes he was in the VP’s office being told to apologize or he was fired.

Red Flag 3: Anyone who criticizes the process, being threatened with their job. 

So, we were fast approaching an imaginary deadline of an alpha version of our software, and we were behind, working OT and generally trying to get 'er done. This particular company prided itself on the 2 weeks over christmas only requiring a skeleton crew, so most of us didn’t mind the OT, in a couple of weeks most of us would get a respite and then come back refreshed. When the VP came out and announced, “ very one will be working over christmas, no PTO except on the holidays themselves, and mandatory 50 hours every other week“, yes they had canceled christmas. For a date that only they believed in, never mind that most people were doing 60-80’s.

Yellow Flag 4:(approaching red) Is when management changes work schedules for you.  

Projects dead. QA ignored. Automation maintenance nightmare. VP unwilling to compromise. 5 active sprints. 

The day I gave my two week notice I had to write a letter to quit, cause I couldn’t find the VP (who was my manager) for the entire day. He never did show up.

So he politely asked me to not inform people of my two week deadline, he’d like to handle it. Of course I was more them willing to allow him to  handle it in his own way.  A week rolled by, with no announcement, so I was unable to tell anyone. As a result no one thought it important to learn and take over the tasks that I was doing. Week two started with a scum where people started to assign me work, and at this point I was tired of waiting and kinda upset. I told the entire team that most of that work needed to be someone else’s cause Friday was my last day. Everyone was upset, and yet officially my leaving still wasn’t announced till one day before my last.

To top off a wonderful last week of trying hard to make sure people are prepped for me being gone. My exit interview consisted of HR telling me over the phone, “No, that’s not why you’re leaving.” What do you say to that?

Red Flag 4: The process of handing off your knowledge to someone is stymied, and no one in management cares why you are leaving. (Albeit this flag is more for people still there.)

Software Testing is in my Genes (maybe).

In possibly good news, we may now hire based upon a genetic test! I wonder how that will go wrong as I'm sure it will.  As a personal aside, I find that the glass contains what appears to be roughly 50% atmosphere and 50% liquid, but I have not tested it to validate those observations.

Interviewing, post 3

TLDR: brain dump.

This is my internal thoughts, presented for you to look at. (cause I want to remove my internal stigma of having to have perfect / point-filled blogs before I post.)

Little background: I used to interview people at a much larger company, we had a team of roughly 50 testers / SDETs. This allowed me a certain amount of freedom in hiring people that might not perfectly fit any particular profile. Aka they could be higher risk (in my mind) in certain key areas, cause they were offset but a large group of other people. Or we could move them to a team that offset any other risky areas while we saw if they panned out over a 2-3 month period. Or were able to learn what we thought was necessary.
I now work for a company with a 4 person QA team, hiring a final person for this year to make it a team of 5. Personally I think this gives me significantly less leeway on what it is that we can hire. I have less wiggle room for potential issues. I don't have the cycles I would like to, to devote to training someone with only some potential. Basically what I think I'm looking for and what I need to fill are more tightly coupled for this current position.

So, in interviewing people, I look for certain key talents. One of those talents is self-motivation / drive / passion. In an effort to attempt to figure out what I'm looking for in good people to hire, one of my employees asked me to define what I meant by motivation or drive.

Before looking it up in a dictionary:
Motivation: A reason to do a task.
Drive: Internal motivation, requiring little to no outside force(s). This can be synonomous with Self-motivated.
Passion: A strong drive that allows one to singularly focus on a task or set of tasks. Passion can be good and bad. Not allowing oneself to defocus when necessary (OCD).

Dictionary (the definition I thought the most pertinent):
Motivation: a motivating force, stimulus, or influence : incentive
Drive: to press or force into an activity, course, or direction
Passion: a strong liking or desire for or devotion to some activity, object, or concept

What does all this mean to me? I think my ideas of what motivation / drive are…are pretty reasonable (perhaps self-motivated is the more appropriate word). Now the real question comes down to HOW do you find out if someone is self-motivated or driven?
I've been reading Hiring with your Head (1) recently to see how someone regarded as a great interviewer goes about it. Adler likes the idea of past proof of having done it. He asks, "What is your most significant accomplishment related to X?" 
Personally, I'm not sure I want to just ask someone directly, "Are you motived? Prove it." People can make up anything if they know what you are looking for.

Lately I've been saying something to the point of:
One of the key objectives for this position is the ability to write bug reports that are clear, concise, accurate and relevant. At a high level, what do you think about this and how would you go about this? What have you accomplished that's most similar to this?
One of the key objectives for this position is the conversion of component, functional, system and regression tests into automation. At a high level, what do you think about this and how would you go about this? What have you accomplished that's most similar to this?

And then digging into the details of the given answers to get specific details. I have found this seems to give me the data I need to determine "is this person self-motivated". But since I just changed up the interview questions I use to start taking this into account, I'll reserve judgment till later.

-------------------

(1) Adler, Lou. Hire with Your Head: Using Performance-based Hiring to Build Great Teams. 3rd ed. Hoboken, N.J: John Wiley & Sons, 2007. Print.
http://www.amazon.com/Hire-Your-Head-Performance-Based-ebook/dp/B000SEOVH6

My interviewing start and the changes I've made.

tldr: Changes in how you interview is inevitable.

I started interviewing at my first job. It was a gig with a small company (<20 people) and I was the Engineering Intern. No, I still don't know what that meant, I just did everything anyone asked me to from hardware to software to fetching donuts. I was included in interviews there, but not as a participant, I was asked to only be an observer. They were very traditional. What tech do you know? Do you know this specific tool? And not much more then that. If everyone got along with the person, and they had decent answers, they were hired.

I'm moved through several companies now and moved from observing to influencing to being the final say about who gets hired. Initially, I started with the traditional questions, but they resulted in people that didn't always work out. Then I started to experiment with hiring. What questions should we ask? What answers should we expect? I'm grateful to the employer at the time who allowed me the freedom to experiment with how to judge what people to hire.

While this data doesn't cover all of my hiring, it is an 18 month period where I kept fairly detailed records while I was experimenting heavily with interviewing.

  - 250-300 phone screenings (don't remember an exact number, as there were untold tons of them)
  - 50 people made it to a physical interview
  - 1 person disappeared during the interview process (moved to Texas I found out)
  - 25 people made it to a second interview, 2 of those people jumped straight to 2nd interviews as they came from recommends of people I trusted, or I had already directly worked with them.
  - 19 people got offers
  - 3 people declined
  - which means after interviewing ~300 people, we hired 16 people.
  - 4 of those people didn't work out.

I'm not giving you this data to tell you how bad or good I was at interviewing, but to allow you to see what the process was like for me. I had to phone screen every resume (300, ~150 hours), talk with 50 people once (minimum 25 hours), setup a team of 3 people to talk with another 25 (minimum 150 hours, I count prep time and debriefing time), then extend offers, then worry and wait. 325 hours to get 12 people that worked out. That's three quarters of a week to find each person. 

What is the point of an interview? To find the most qualified candidate for a given job. Or at least for now that's what I'm going to use.

So at first I started with question like I had observed. Why do you want to leave your current employer? Do you know Agile? Do you know C#? Write me an algorithm in some language. Plus a host of other tech skill questions. Over the course of an interview I'd get a feel about someone from body language, terms and sentences they used, and then extrapolate information from the way they answered.  This system worked semi-okay for 1-2 years, but about that time I noticed that I really didn't care 'what' they answered, I wanted to hear 'how' they answered. I was just asking questions to get them talking, trying to get them to talk long enough to understand how they thought.

That's roughly when I started to change up the game. It took about 6 months of on and off introspection to come to the next level. Why was I using these questions? What was I really looking for in these people? After reading several books (Strengths Finder and Good to Great) I had a more solid idea of the traits I wanted to see in people: character, work ethic, intelligence, responsibility and values. Observing people I knew that were good testers, and people I knew that were bad, I clarified my definitions of those traits. Then I tried to design questions about those traits that I wanted to investigate in each interviewee. 

It's not that I don't ask tech questions anymore. Merely that tech questions are not my first level weed-out questions. I have found over and over again that personality and mentality are the most important parts. It's nice if you have tech skills, but I can judge your ability to learn tech skills independent of your personality and mentality at a second interview. And I'm more likely to hire someone with the right mindset and little skills, than I am someone with the wrong mindset but the the right tech skills.

*****************
As an example of how I would change up interviews (occurred just today):

As I was sitting in the bathroom, I noticed that this plastic piece had shimmied down the hinge pin and the lock was wiggly. I pulled out my swiss army knife and moved the plastic piece back into position and tightened the 2 screws holding the lock. After I did this it occurred to me, is that normal behavior? Do people just fix things when they notice them broken, or do they just complain to someone else?

So my new thought on how to change my current test interview. Put something in the room that anyone would want to fix. Something easy to fix, but not so weird that someone in an interview wouldn't want to do it for fear of looking strange. A marker with a cap off is my first thought. Do they notice it? Do they fix it? Does that make them proactive?


So I brought this test to my crew of testers. They of course shot me down, like all good testers would. What does being proactive prove? That they have OCD?
Probably the best rebuttal of this was Wayne, "talents and(sic) patterns of thought ... you(sic) can't determine a pattern from one event". 
Then JCD piped in with "Is that as much cultural behavior(sic) as otherwise? Some people think it is better to be polite, or maybe that is some sort of test tool you will use later?"


I'll still be using the marker test in my next couple of interviews. However, I won't use the data I collect from it. It might end up being useless, it might be an indicator of something I wasn't looking for, who knows.

Lastly, I'd like to point out, this interview thing is a constantly evolving subject. Finding good people is difficult and making it easier is a goal of mine (short of just opening a training school). Lately, I've dived into blogs about hiring and have 2 new books, by Lou Adler, I want to read on the subject. Stay tuned...

A subject of the hiring process...

Independently of Isaac (as in he didn't know I'd do this), I was writing a blog on the hiring process.  However, now that he's done so first, I'm going to use it as a thing to reply to, as I feel I have something to add.  In particular, I'm going to talk about the experiences I have had hiring, being hired and being semi-reinterviewed in the 10 years I worked near/with/for/against/whatever with Isaac.  I'm editing out some of the experiences to protect other people (who interviewed really badly, and making this as 'my personal view' as I can).  So let's start at the beginning...

My first time I meet Isaac, he was leading a different team, in the same lab, working very different hours (testing in shifts), but I saw him once a week or so.  Now I don't know who came up with the idea, but they decided to do 'testing' for the testers in the lab to see where their strengths were and where their weaknesses were.  Each test lead wrote questions and tests were given to about 100 employees.  I believe I lost points on 3 questions.  The one question I recall being bitter about was "You have a screen with 8 widgets, 2 buttons (cancel, save), and each widget has 4 states.  How many tests would you run?"  I worked for a small look and feel group and so I answered "1".  I backed this up by the fact that I'd run one automated test that checks all of the UI positions of the elements, that all the elements were standard controls and that no text was truncated in English.  That is exactly how we were writing tests at the time.  I also noted when more languages showed up, I'd write more tests, one additional for each language.  Needless to say, I was marked down on that answer because they wanted a total number of states you could test.  I note this because I think it is a mistake to attempt to "Grade" your employees without really considering what they do and that context.  In other contexts, my answer would have been different.  If I recall right, Isaac specifically said it was a stupid idea, and that if the leads didn't know their own people's strengths, that was a problem unto itself.

A few years later, I am looking for a new job and I interview at the first place I had gotten an interview opportunity with, and there sitting on the other side is Isaac.  We chatted about the old days for a few mins and what I had been doing since.  He mentioned another guy I had worked with at HP was working there.  At some point, he started in on the interview.  He asked me to test a login page, which was odd considering most people asked about testing a pen.  Isaac did take notes, and I did try to look at them, but I don't recall actually seeing anything but sort of set of check boxes with skills.  SQL came up at some point, and keeping in mind I had mostly been doing black box hardware/software testing, SQL was not something I had done day to day.  I explained this but noted I had played with a very light version of SQL for an open source project.  I noted it didn't support stored procedures, so I didn't know how they worked, etc., but I could do a simple select/insert/delete/update.  He had me demonstrate it and I think that was the end of the interview.

I won't go into the second interview other than to say it was a round robin with 5 guys in a 10x10 room with a table in it.  Yeah, not the most fun I've had and maybe worth a different blog post.  Oh, and I was hired.

Isaac continued to do all the first interviews, and so I didn't get to experience much of his technique or the people he said no to.  However, I can speak to the people we had a second interview with.  Keeping in mind, the average time between hires was 9-16 months, we did a lot of interviews.  I will mention only the style and some highlights.  We used a round robin technique and each of the QA Engineers would ask a question, with occasional follow up questions.  Some people had signature questions, like "You have a 2 gallon bucket and a 5 gallon bucket and you need to get 3 gallons of water..." or Randomly pick a phrase from the resume and ask about it in detail.  Isaac did not do those things as best I can recall.  Isaac mostly asked questions to see if he could find any value the person might be able to give.  Ask an automation question.  They fail.  Okay, what about critical thinking...?  What sorts of learning do they do?

We had almost all duds, and on occasion I do recall that "Why did you okay this person?" looks.  More often than not, the candidate was meh enough that I can get that Isaac hoped they might study up, but they didn't.  On rare occasion, we found someone who just bombed the second interview after doing well on the first with no reason for the change.  As interviews are just as much 'keeping the wrong people out' as they are about 'getting the right person', we might have missed a person or two who was good but had a bad day.

I should note, I write this with little fear that someone will use these notes in an interview as I would thinking it a good thing they read blogs if they prepped that way.  I see all research valid for preparing for an interview up to asking someone internally about the position.  That being said, I don't plan on giving all of our specific interview questions here, in part because I change up my questions based upon the candidate and how well they do on some area.  Sorry for the digression.

After the round robin, we would all think a little about it and sort of talk through the good and bad points.  If anyone said no, it was a no.  There were only one or two times when only one of us said no.  We did all take context of the position in mind, but we also all were aware that times change and the person we interviewed had to be able to adapt some.

Since then, I think the interview process has changed a little based upon the company and how practical it is.  That is to say, in a company with 30 testers, only 5 or so might actually interview the person.

I do want to make some additional comments, things that don't fit well into the narrative, but are equally useful.  I have on occasion look at Isaac's interview notes, and sometimes I find interesting little bits I miss.  One of the big ones is 'temporal' observations.  Like "Hesitation", or "Long Pause" that he will note, showing that someone thought about their answer.  He also would write down weaknesses to observe if they improve in the next interview.  I never get to see if that happens, so often when we consult about a person, we depend on Isaac's input of 'what improved'.  Sometimes we would consult Isaac before an interview about the person, but not always.  I'm not sure if the data is worth the bias or not.  Last but not least, sometimes LinkedIn or Google is consulted about a person to see what they want to share.  I have never used Facebook or any social data for an interview and as far as I know, nor has Isaac.

In perhaps a part 2, I may go over how I conduct a second interview rather than the structure.