Friday, September 6, 2013

A subject of the hiring process...

Independently of Isaac (as in he didn't know I'd do this), I was writing a blog on the hiring process.  However, now that he's done so first, I'm going to use it as a thing to reply to, as I feel I have something to add.  In particular, I'm going to talk about the experiences I have had hiring, being hired and being semi-reinterviewed in the 10 years I worked near/with/for/against/whatever with Isaac.  I'm editing out some of the experiences to protect other people (who interviewed really badly, and making this as 'my personal view' as I can).  So let's start at the beginning...

My first time I meet Isaac, he was leading a different team, in the same lab, working very different hours (testing in shifts), but I saw him once a week or so.  Now I don't know who came up with the idea, but they decided to do 'testing' for the testers in the lab to see where their strengths were and where their weaknesses were.  Each test lead wrote questions and tests were given to about 100 employees.  I believe I lost points on 3 questions.  The one question I recall being bitter about was "You have a screen with 8 widgets, 2 buttons (cancel, save), and each widget has 4 states.  How many tests would you run?"  I worked for a small look and feel group and so I answered "1".  I backed this up by the fact that I'd run one automated test that checks all of the UI positions of the elements, that all the elements were standard controls and that no text was truncated in English.  That is exactly how we were writing tests at the time.  I also noted when more languages showed up, I'd write more tests, one additional for each language.  Needless to say, I was marked down on that answer because they wanted a total number of states you could test.  I note this because I think it is a mistake to attempt to "Grade" your employees without really considering what they do and that context.  In other contexts, my answer would have been different.  If I recall right, Isaac specifically said it was a stupid idea, and that if the leads didn't know their own people's strengths, that was a problem unto itself.

A few years later, I am looking for a new job and I interview at the first place I had gotten an interview opportunity with, and there sitting on the other side is Isaac.  We chatted about the old days for a few mins and what I had been doing since.  He mentioned another guy I had worked with at HP was working there.  At some point, he started in on the interview.  He asked me to test a login page, which was odd considering most people asked about testing a pen.  Isaac did take notes, and I did try to look at them, but I don't recall actually seeing anything but sort of set of check boxes with skills.  SQL came up at some point, and keeping in mind I had mostly been doing black box hardware/software testing, SQL was not something I had done day to day.  I explained this but noted I had played with a very light version of SQL for an open source project.  I noted it didn't support stored procedures, so I didn't know how they worked, etc., but I could do a simple select/insert/delete/update.  He had me demonstrate it and I think that was the end of the interview.

I won't go into the second interview other than to say it was a round robin with 5 guys in a 10x10 room with a table in it.  Yeah, not the most fun I've had and maybe worth a different blog post.  Oh, and I was hired.

Isaac continued to do all the first interviews, and so I didn't get to experience much of his technique or the people he said no to.  However, I can speak to the people we had a second interview with.  Keeping in mind, the average time between hires was 9-16 months, we did a lot of interviews.  I will mention only the style and some highlights.  We used a round robin technique and each of the QA Engineers would ask a question, with occasional follow up questions.  Some people had signature questions, like "You have a 2 gallon bucket and a 5 gallon bucket and you need to get 3 gallons of water..." or Randomly pick a phrase from the resume and ask about it in detail.  Isaac did not do those things as best I can recall.  Isaac mostly asked questions to see if he could find any value the person might be able to give.  Ask an automation question.  They fail.  Okay, what about critical thinking...?  What sorts of learning do they do?

We had almost all duds, and on occasion I do recall that "Why did you okay this person?" looks.  More often than not, the candidate was meh enough that I can get that Isaac hoped they might study up, but they didn't.  On rare occasion, we found someone who just bombed the second interview after doing well on the first with no reason for the change.  As interviews are just as much 'keeping the wrong people out' as they are about 'getting the right person', we might have missed a person or two who was good but had a bad day.

I should note, I write this with little fear that someone will use these notes in an interview as I would thinking it a good thing they read blogs if they prepped that way.  I see all research valid for preparing for an interview up to asking someone internally about the position.  That being said, I don't plan on giving all of our specific interview questions here, in part because I change up my questions based upon the candidate and how well they do on some area.  Sorry for the digression.

After the round robin, we would all think a little about it and sort of talk through the good and bad points.  If anyone said no, it was a no.  There were only one or two times when only one of us said no.  We did all take context of the position in mind, but we also all were aware that times change and the person we interviewed had to be able to adapt some.

Since then, I think the interview process has changed a little based upon the company and how practical it is.  That is to say, in a company with 30 testers, only 5 or so might actually interview the person.

I do want to make some additional comments, things that don't fit well into the narrative, but are equally useful.  I have on occasion look at Isaac's interview notes, and sometimes I find interesting little bits I miss.  One of the big ones is 'temporal' observations.  Like "Hesitation", or "Long Pause" that he will note, showing that someone thought about their answer.  He also would write down weaknesses to observe if they improve in the next interview.  I never get to see if that happens, so often when we consult about a person, we depend on Isaac's input of 'what improved'.  Sometimes we would consult Isaac before an interview about the person, but not always.  I'm not sure if the data is worth the bias or not.  Last but not least, sometimes LinkedIn or Google is consulted about a person to see what they want to share.  I have never used Facebook or any social data for an interview and as far as I know, nor has Isaac.

In perhaps a part 2, I may go over how I conduct a second interview rather than the structure.


  1. Thanks for the post, Jeremy. Even in just two posts, I can see how you and Isaac have complementary styles - it's nice. Would you mind a couple questions?

    1) You said Issac might "Ask an automation question" - Could you elaborate on what these sound like? I can think of some very different automation questions. Mine would typically be "I see you have (tool) on your resume. Tell me about a time you used (tool)." and a roving conversation.

    2) I think it'd be nice to elaborate on interviews and tests for physical objects vs. how would you test the actual work. I'm a fan of both.

    3) One common line I see in interviews is that keeping the bad hires out is critical - that if a company wanted to make a mistake, it would fail be by hiring too few, instead of fail by hiring too many. I wonder if there is a middle course - 3-month contract-to-hire choices? It seems to me that many companies that choose this route end up hiring the person on day 90 and regretting that two years later, mostly because of fear of conflict at go-no-go decision time. I wonder if we could decrease the stigma on both sides of the table from a no-go decision. (Having worked in not-great-fit environments myself, today, a 3-month contract followed by starting fresh is better, in my mind, than a temp-to-hire at a place I don't want to be.)

    Anyway, nice post, thanks.

    1. Thanks, Isaac and I have worked together for a long time, so I'd like to think we know how to work with each other.

      1. I would love to... that is actually a part of that 'part 2' I want to do. I don't want to talk of Isaac's questions, as he will probably reply too, however, I will talk about my own questions. I typically look to see if the person understands development and if they understand test more than how to write automation. Since I think automation is a subsection of development, I care more about that than the details of automation.

      I might ask, "Here is some code: . Tell me what bugs are in it." I am looking for their ability to understand the syntax, their ability to note issues in code and I want to hear them talk it out. I want to know if they can read other peoples code. I want to see both their testing technique (as they may have to write or at least read other people's unit tests) as well as their technical skills.

      If I ask a directly automation related question, it might be, "what sorts of things can you ask the developers to do in order to properly automate a project?" where might be , , , , etc. I am looking for their experience in automation by trying to determine what sorts of issues they see in automation.

      2. I'll keep that in mind for a future blog. :)

      3. I think the problem is that is a corp culture issue. Some places don't mind getting rid of someone bad, but the risk of lawsuits and the likes are great. I can't speak to the details, but I know HR in some organizations is and has to be very risk adverse because of that possibility.

      Three month contracts, while they sound good, can be bad for morale. Imagine hiring someone who you like working with simply disappearing one day, with HR cleaning out their desk. Now imagine two employees leaving on the same week and the rumor mill will create a mess. And yes, that is a true story. Oh, and I was one of the two people leaving, so it was interesting having people ask me about an employee who was let go on his 90th day. Needless to say, I don't know if that was a one off or typical as I had not heard of many people being told 'we don't want you' after 90 days but also was not a manager who was directly involved in the choices.

      Could it work? Sure, but not in the environments I've been in.

    2. 1 - I try and focus on the development side as well. I have found that the specific technology that someone uses is less important then their ability to understand the underlying development practices.

      I have people write me some brief code snippet, Fizz Buzz, Some Math function or something else. After they complete the code, I then ask them to test it. Most people fail at step one, most that make it past step one fail at step two. But it's the thought process they describe to me that really make me say yes or no to a candidate on 'automation'.

      2 - At previous places I prefer to ask a question that directly relates "How would you test a login page" as that was one of our pages. However, depending on how complex the SUT is, I can also deal with asking a physical object test. But I don't think the questions are equivalent. (BLOG TBD…)

      3 - Personally I would love to see 3 month probation periods in tech. However, there is lots of morale and potential for employer abuse there (we'll hire them for 3 months, then say they don't work out, then let them go). You'd really have to trust an organization to go in with that option (but I have done it). That being said, I'm gone to a company that had that policy, it worked out great, they didn't know me, I didn't know them. After three months both of us were happy, and we decided to continue our relationship.

      Because most places don't use that policy I error on the side of don't hire. I really don't like to fire people, nothing is more stressful (at least that I've experienced). Plus most people don't like to be fired :)