Interviewing Tip: Stress Test Critical Thinking. Please.
Interviewing is art, and perhaps the most important thing you could do for your company.
My post with tips on hiring web analysts (Hiring? What Works: Fresh blood or old hands? Experience or Novicity?) generated lots of comments [25!] and emails. The most common request was for tips on how to pick the right employer, I think that's a reflection that this is a buyers market [I promise to write that post just as soon as I can].
But is also generated this very interesting question via email:
I read your "Hiring? What Works" blog this morning and I like what you say about looking for the critical thinker. I agree that you can find this person "flipping burger" as much as anywhere else. Here's my question “how do you identify the critical thinker?" I can some gut feels on candidates, but it's really hard to say. Hire with a trial period? Doesn't necessarily seem to be the solution. Do you have some pointers/hints on this?
It is a tough question.
There are several ways to do it.
Microsoft was famous for giving out puzzles and riddles that help show the way of thinking.
Here's one: "You've got someone working for you for seven days and a gold bar to pay them. The gold bar is segmented into seven connected pieces. You must give them a piece of gold at the end of every day. If you are only allowed to make two breaks in the gold bar, how do you pay your worker?"
Get it? You are hired! [If you know the answer post it in comments!]
I have to admit that I am not the biggest of fans of this approach, and to do full disclosure I stink at them. :)
The model suggested in the email of, what I call, temp to perm is not a bad idea because it both allows you to test the person out but more importantly you can check other attributes like leadership and team fit etc. I have done that a lot.
Sometimes though you don't have that option. So what can you do?
My personal favorite is to give the candidate a real business problem (something that needs critical and analytical thinking) and then ask them to solve it.
It could be a numbers (web analytics) problem. It could be a competitive problem. It could be a page level problem. It could be a acquisition problem. Anything really, but a real problem for my company or a company I know. You'll see later why the reality of the problem is important.
I am watching for:
1) what they come up with as a solution and
2) how they think
The latter is perhaps even more important than the former.
I am looking for creativity and atleast some facet of the solution to surprise me (in a good or a bad way).
If you are a solid critical thinker then you'll think of something really interesting. Something I can't think about (if for no other reason than because I am too close to the problem and you are not).
I am also looking for, because I am in Analytics after all, a part of the answer to involve numbers (how to look at data to solve the problem, qualitative or quantitative – so it does not have to involve hard core numbers). Something innovative.
When Interviewee provides the answer I will push back a bit, regardless of the answer (right or wrong).
I'll probably throw a curve ball, try to say something totally silly, try to throw a spanner into the logic, point out a flaw, share a sub-plot etc.
I am trying to see if I can influence you to change your answer. I am trying to see if you can defend your original answer (most people fail here). I am trying to see what your rationale is for whatever you say / do. I am trying to see if you ask clarifying questions. I am trying to see how fast you think, and what happens, if you are backed into a corner.
If you are a critical thinker you'll think uniquely and be able to, for the lack of a better word, survive.
In fact you'll love the challenge and you'll reply with specifics and not ambiguous FUD. It is amazing how quickly one can cut through the Interviewee's BS or lack of thinking capability or actual experience. No matter how many books they have read / written :) and how long and impressive the resume was.
Frame a open ended real problem, wait for the answer, then push back politely.
There is one more reason for pushing back politely. I am also looking for a certain personality type.
Most interviews are stacked against a Interviewee. By pushing you a bit, very respectfully and in a very dignified non-challenging way, I am trying to assess your personality type.
Can't be a push over. Can't be a "report writer mindset". Can't be "pushed" too easily to change opinions. Can stand up and defend (it is amazing how many people fall at this). Can't make stuff up. Can't get flustered. Can say "I have no idea", at the right moment. Can give a complete answer (so many people fail at this), even if a wrong one.
I have used the word amazing in this post a few times. But here is the most amazing thing: even if you know precisely what I am going to do, above, it is nearly impossible for you to "prepare" for it. :)
There are always exceptions of course, there are some great FUD creators out there, but for the most part the reality of the problem offered, the Interviewee's answer, the data involved and the nature of the discussion all ensure that reality will most likely come out.
Interviewing is the hardest thing you'll ever do. Hope this post helps a bit.
Now your turn.
What do you think of this method? What works for you? Any favorite tips / best practices you have developed over the years? How did you identify the last fantastic employee you hired? How did you know?
Please share your perspectives via comments.
P S : Here are two of my favorite posts on this blog, and they are totally non web analytics related.
I wanted to highlight them for you, one about how to be a great leader, IMHO, and the second about my own rules when it comes to managing careers / jobs / work life:
I think you'll really enjoy them as well.
Subscribe 
September 14th, 2007 at 00:43
Give the worker any piece and he has to break it himself and give you the other pieces back. Done :-)
September 14th, 2007 at 00:44
The answer to the puzzle
Break the gold bar int to 3 pieces and the propotion should be 1:2:4 (we can name them as A:B:C) 1st day, you pay him with A. 2nd day, you pay him with B,and he returns you with A. 3rd day, you pay him with A. 4th day, you pay him with C,and he gives back A and B to you. 5th day, you pay him with A. 6th day, you pay him with B,and he returns back A. 7th day, you know what to do.
September 14th, 2007 at 05:11
break off one single piece, and then break off two pieces together as a chunk.
At the end of the first day, pay them the one single piece.
At the end of the second day, ask for the single piece back in exchange for the two-piece chunk.
At the end of the third day, give them the single piece (they now have three pieces).
At the end of the fourth day, ask for the single piece and the two-piece chunk back in exchange for the remaining part of the bar – which is a four-piece chunk.
At the end of the fifth day, give them the single piece (now they have five pieces).
At the end of the sixth day, ask for the single piece back in exchange for the two-piece chunk.
And at the end of the seventh day, give them the single piece yet again – they now have seven pieces.
Please send my job as soon as possible, because I clearly have way too much time on my hands.
September 14th, 2007 at 05:19
Having spent 15 years in the staffing industry and now being directly involved in teaching critical thinking to companies I found your comments insightful. I agree that mind puzzles show how good someone is at solving puzzles–but they don't tell you much about a person's ability to "think logically and comprehensively abotu a situation and identify an appropriate course of action" which is what critical thinking is all about.
A well-defined set of behavior-based questions is a much more reliable way to identify whether or not someone really knows how to think through a problem to a workable solution.
September 14th, 2007 at 05:51
Break into pieces of 1, 2, 4. Give him/her one piece day 1, exchange for the two day two, give the 1 back to them day 3, exchange all that for the 4 on day 4, etc on up to 7.
On the interview issue, we usually mix in a business case that's pretty much as you describe. We weight the thought process more highly, but the answer does have to be plausible, as any good analyst always looks at the end product and reassesses if it doesn't seem to make sense.
I also like to have them talk me through an important analysis that they led. By that I mean what problem they were trying to solve, what data they decided to assess, what the data showed, and what recommendations they made off it.
You'd think that a person could bluff their way through it even if their boss guided them heavily at the time, but if you're really digging into why they did this or that along the way, it's incredibly hard to do. If a person isn't strong at analysis, they have trouble talking through that beginning-to-end case coherently. Nothing's perfect, but I've had very little in the way of "false positives" from this method.
September 14th, 2007 at 06:13
I think that the answer is that the employer should just find an better and easier way to pay their employees.
September 14th, 2007 at 07:35
Brillian answer Chuck :-)
Avinash-
You have company (in me) about not being a fan of "gold bar" type problems to hire a critical thinker (however, I must also admit that I love puzzles and riddles on the side:-) ). Talking about a puzzle or riddle over a mug of beer with friends is different than addressing it in a panel with 2 or 3 minutes to think [plus the added pressure] — sometimes even the best of candidates can falter on this type of questions.
One of the things I always try to fathom during the interveiw process is the way the candidate thinks and put less importance on the outcome itself. In this day of googlism, I also weigh down the knowledge of tactical procedures and heuristics that a person can always pick up and learn later. The key I am looking for is how open and how capable that individual is on ramping up to a situation once he/she gets hired.
Also, I am not looking for 'canned' correct answers; on the other hand, I would rather hire a person with an original answer that is slightly offbase. Like you, I almost always have follow-up questions or like Chuck said, ask them to walk through the process– more to again differentiate those who came 'prepared' :-) and those who can think through a dynamic situation.
But net net, I have to admit that personality and attitude are the #1 criteria that I use to make a hiring decision. In the end, he/she has to work with my team and with other folks — and so given a choice, I would rather hire a personable decent critical thinker than a brilliant but morose critical thinker :-))
September 14th, 2007 at 07:49
Nothing is a better indicator of a person's critical-thinking skills than the questions he or she asks. Rather than presenting someone with a puzzle, give them a work product and ask them, "What else would you like to ask about this? What questions does it make you want to ask, and what do you think the answers will tell you?" Pay attention to any question the person asks. Something as simple as a question about whether the workload varies regularly during most days can indicate that a person is putting facts together and trying to draw insights from them.
September 14th, 2007 at 08:41
I would say you need to ask "How are the pieces connect?" People shouldn't assume the connections are even. I like the ability to look at the thought process of the employee. I would want an employee to ask questions and tell me what they are thinking and assuming. I know a manager who asked "How many women shoes were sold in China?" It is not the question but the thought process
September 14th, 2007 at 09:04
Everyone : You'll see "duplicate" answers to the riddle. Just to stress that most were held in moderation (Brian's was the first public on). I was asleep, literally (!), and approved them all this morning. So each person who provided the answer here did so without reading anyone else's. :)
I suspect Ziggy would get hired by any company for his answer. Brilliant!
Kenny does need a job, so anyone who is looking for a great employee please email me so I can email him. :)
Chuck & Richard, great points. I love the stress on the thought process.
Thanks everyone.
-Avinash.
September 14th, 2007 at 10:04
When interviewing for IA / UX design positions, I like to ask:
Lets say you're looking at a mock up of a page. How would you tell where the user is likely to get confused or make errors?
Then I listen for whether they give an a) seat of the pants type answer, or b) use of guidelines type answer.
If they give a seat of the pants type answer, I'll probe to see if they have any principles or models of user behavior and errors that they can use.
I might ask: what types of user error are there?
None of the people I've interviewed have given a satisfactory answer to this – not even mentioning material I know they studied in their Masters degree – like Norman's list of error types, or the gulf of evaluation or execution.
Not having these more incisive frameworks at hand suggests they wouldn't be able to critically analyze a web page to the degree we want.
Sure its not a fool proof method, but it does reveal people's approaches especially if its more a) seat of pants or b) reasoned.
September 14th, 2007 at 13:13
I'm generally not involved in hiring people, but from the flip side I can tell you that I am much more likely to choose an employer who is giving me realistic problem solving situations over one that gives me puzzles.
Maybe that's because I'm more interested in strategic action. The discussion of the overall problem and how to solve it seems much more interesting.
Of course, either option is so much more attractive than neither.
September 14th, 2007 at 16:07
I find it fascinating that people like Joe the Staffing Specialist are so enamored with Behavioral Interviewing (BI).
I have two problems with BI:
1) It is easy to prep for these. There are just not that many different topics. If you rehearse 10 situations, you can stear almost all BI questions to one of those.
2) BI questions don't tell me if you did it or if your team did it. Furthermore, they don't tell me how FAST you did it. "Tell me about a time you…" doesn't tell me if it took two days or two months.
September 14th, 2007 at 22:34
Here is another way.
Fold bar in half so one side is three segments and other is four segements.
Next, take this shape and fold it upon itself as if rolling it up, starting from the fold. Do this until have it one bar wide and fold the fourth piece over.
Then cut once along the seam where the fourth piece folds over. When you do this, the 7th piece is removed and you will have three two pieces. Keep the pieces stacked and cut down the middle and you will have six pieces in two cuts.
September 14th, 2007 at 23:00
I forgot to mention on thing regarding the exchange model : What would happen if the new hiree went to the local pub after work to celebrate his new job and paid for the round of beer for the house with his new shiny gold bar?
September 15th, 2007 at 03:48
On the back of an envelope with a pencil, I sketched the 'golden chocolate bar' and then figured 1, for day one, 2 for day two in exchange for 1, return 1 on day three, and using the remaining 4 for day four. This works as long as a. the hiree doesn't need to liquidate the asset to live and b. you have explained how payment is to be made in advance! Then I started thinking about other things like getting out and enjoying the sunshine and why I visited this page in the first place, ahh yes….
September 15th, 2007 at 04:49
[...] Posted by abstracts on September 15th, 2007 Hiring in today's knowledge driven economy is the toughest task for an organizations and specially startup companies. Avinash in his blog posting on hiring hits the bullet. Key take aways are [...]
September 15th, 2007 at 12:09
Why not melt the goldbar? Solves the whole breaking problem. :)
September 17th, 2007 at 09:20
As for the goldbar riddle: The answers I read are no doubt amazing, but I immediately took a different approach b/c I read "critical thinking" in the post title and the "normal solution" doesnt seems to have any critical thinking involved imho? (it requires great analytical thinking and admittedly I doubt I would have solved it)
"You’ve got someone working for you for seven days and a gold bar to pay them. The gold bar is segmented into seven connected pieces. You must give them a piece of gold at the end of every day. If you are only allowed to make two breaks in the gold bar, how do you pay your worker?"
How about making a quick contract (Im from Germany, bureaucracy rules here ;) – or just trust them of course): You give the person the gold bar at the end of every day, but have it handed back to you before the next evening..and you repeat the process, but at the end of the 7th day you give him the gold bar and he gets to keep it. That day he actually gets paid. (Nobody says he can't give it back to you)
If I mine the information from this riddle critically (this post is about critical thinking right ?;)) I think I don't even have to care whether the gold bar is segmented into seven connected pieces nor do I have to break it.
It only reads "you have to give him a piece of gold" (it doesnt say how big it has to be or that you have to break it).
Also it doesn't say when you have to pay him only that you have to "give" him a piece of gold at the end of every day (it doesn't say how big that piece of gold must be).
My second solution would be taking the gold bar to a bank or something and trading it for gold coins ;).
Or maybe I would tell the person (or somebody else) that he has to break one piece off at the end of each day (and then give it to me so I can give it to him). It only says "*you* are only allowed to break it twice, etc.", right?;)
[b]as for the post itself[/b]
I really like your idea of giving the person a practical test instead of abstract riddles.
I think a cool idea would be to give them problems you encountered in past projects (or very similar to those) and ask them what tweaks they would have done to those websites based on the data they see (if they have web analytics experience, of course) and look how the decisions they would have made overlapped with the decisions that really paid off.
I also think that creativity, practical & emotional intelligence are extremely underrated (probably b/c they aren't very important in school or college though very important on the job later-on):
Most people I know think of creativity as something..well cool…artists, musicians, etc., but wouldn't ever imagine how important it can be in SEO or Analytics:
One of the best SEOs I know of has stated before that he thought creativity was more important than analytical intelligence for SEO (though he's obviously is great at both). The things about SEO that require analytical thinking are things most people can learn. For example everybody will be able to learn how Google evaluates links by reading about it. However what decides if somebody is a great SEO and can rank sites for competitive phrases at the top of the engines all comes down to if he can then get tons of links. Most people can't, but the great people in this field come up with creative, outside-the-box ideas for unique content that haven't been done before and are able to attract links in saturated market places (otherwise nobody will want to link to it).
In the NBA some teams have statisticians/analysts employed…and being able to use statistics and do analysis is of course very important there (though I guess it doesnt take a genius to learn statistical tools if given enough time). But what it all comes down to is having good ideas (creativity is also about the speed at which one generates ideas and how unique they are) that nobody had before/that other people can't think of..that are actionable and give the team an edge over other teams.
(I bet most people who dont deal with analytics would laugh at somebody saying it takes "creativity" to analyze basketball or web data as it seems like the most uncreative thing in the world LOL).
I think the same is true for emotional intelligence: If you hear that term you probably think of ability to socialize, not get into arguments..stuff that makes a good diplomat, right? But people with great emotional intelligence also have the ability to…hate to say it, but "manipulate" other people, because they know what makes them tick….This type of intelligence is leveraged in branding (Manipulating sounds so negative, but if were honest why do people buy one brand over another if the product is of the same quality, but only perceived as more valuable due to the brand's image?) or in word-of-mouth / viral marketing (how can I influence people to spread my idea for me? extremely important in SEO, too btw).
I think such segments of intelligence which vary from one person to another and are extremely important when finding the right fit for a job don't really show (even if they may correlate in some way) in school or university grades and can only be found by an employer who really knows what type of skills/traits are crucial to have for the job he's hiring for (and knows enough about different forms of intelligence and traits that help predict them (for example neutral/critical thinkers are often good at analytics and there are many more such traits).
This reminds me pretty much of Avinash's "segmenting like crazy" advice: Segmenting intelligence (as observed) to find somebody who's the right fit for the job ;-)
September 17th, 2007 at 20:33
I learned a lot about the interview process from reading "How Would You Move Mount Fuji?". The book is a collection of puzzles that are used by Microsoft during interviews to gain insight into the thinking skills of the interviewee. One of the key points made in the book is that it is very expensive when an organization hires the wrong person. Following each interview, a hiring team should evaluate whether the candidate answered any questions in a way that would cause the team to immediately reject the candidate. Likewise, the team should evaluate whether the candidate provided any answers which gives the hiring team confidence in the candidate's ability. I often see hiring teams rush to fill a position by choosing someone they think might work out rather than taking the time and waiting for the right candidate to submit an application and interview. If the right person does not show up for an interview, the team should reinvestigate the job description and possibly their interviewing techniques.
September 19th, 2007 at 09:42
[...] Part of the answer may lie in using good hiring practices. Part may be in seeking employees who are freaks of nature and, like entrepreneurs, have a higher risk tolerance. The environment and situation is probably relevant. Most importantly and most difficult, the reward structure has to shift to reward quality creativity more than profitability. Unfortunately it’s very hard to tell a quality creative idea. It’s much easier to use profit as a measure of performance because it is an actual number. Most people would probably argue that it is the right measure also, because a business is supposed to make money. [...]
September 21st, 2007 at 08:31
I think that you need to give some kind of problem to an interviewee to find out there critical thinking and problem-solving skills.
But I think the challenge with asking them to solve a "realistic" problem is that you often (even without realizing it) frame the problem with the advantage of the context and knowledge you possess, that an interviewee cannot. Also, you probably have directions, or leads, that you would like them to pursue, and once they miss those, they are toast. I've been on the other side of one of these questions, when I know that the inteviewer is fishing for a specific angle or solution type, and I know I don't have the context to make them happy.
I like to find out, during the small talk, things that interest them. Then, during the problem solving phase, I ask them to identify a couple of problems in that space. I want to see if they are willing to be strategic, if they can think long term, and then – I either pick one or let them choose, based on how fair I think I can be as an evaluator. If you can think further out of the box, and come up with more opportunities than they can in one of their "comfort zone" topic areas, that is a red flag.
Then I try to see if they are willing to roll up their sleeves and get to work, and display a resonable, tactical approach to coming up with a solution to their "think big" strategy. Can they clearly identify the risks, the most difficult pieces? Do they see opportunity for areas that are the most difficult, but are just waiting for an innovation? Can they identify some key assumptions, propose a couple of alternate solutions, based on tangible margins of error inherent in their assumptions?
If so, we've got a winner. They can think big, they can execute on an idea, but they also don't do it with blinders on. They acknowledge practicality, but don't shy away from a challenge.
September 25th, 2007 at 17:11
hi Avinash,
long time reader of your blog, but got to comment only now. this post was way too tempting not to comment :)
I've interviewed / reviewed at least 20+ people in the past few years and have been experimenting with various models. the one that has worked best is analysis of actual data. I give candidates an excel sheet that has few ten thousand records and ask them to summarize, analyze and recommend.
I get to see a lot of things this way.
1. can this person handle lots of data?
2. the presentation skills of the candidate
3. Analytical ability
4. Ability to express thoughts and opinions in a written form.
(apologies for the typos. I'm on my phone typing away furiously)
September 25th, 2007 at 22:35
Just read this post, found it interesting how many solutions were taking for granted that the employee would bring back the pay (piece(s) of the gold bar) you gave him the day(s) before. In my experience, once you get paid you usually spend the money :)
I always value the simplicity of a solution over a complex and over-engineered one. In this case I would prefer to break the rules but have a motivational answer:
I only break the gold one bar once: I give him 1/7th of the gold bar at the end of the first day, and the remaining 6/7th when his work is completed (even if it is before the end of the 7th day). This way I might even have the job I need done before the deadline ;)
September 28th, 2007 at 07:42
Ooops! I hit submit before I finished. . . As I was saying–example questions might be
Tell me about a time when you were working on a project and the people working with you didn't give the project the same effort as you. How did you handle the situation and what was the result?
When you encounter problems, unexpected challenges, or delays in completing a goal what do you do and how do tour actions help you achieve your objective?
What is the most difficult thing you've accomplished professionally? What factors were critical to your success?
Basically, a behavioral questions asks for a situation, how the person acted/responded, and the outcome that followed. (Tell me about a time when. . . how did you respond. . . what was the outcome?) The key issue with behavioral questions is to not be hypothetical. If you ask a hypothetical question (what would you do if or what would you do when) you're inviting a hypothetical answer where the person can tell you whatever he/she wants.
Can someone lie during a behavioral interview? Sure–but it can get difficult. If you have a set of questions for several traits you want to evaluate and you mix those questions into an interview without telling the person what you're looking for, it becomes very difficult to lie beacause the candidate doesn't know what the questions are evaluating. Getting several questions about the same behavior spread out over an hour makes it challenging to keep the stories straight.
October 1st, 2007 at 05:49
We use puzzles sometimes, just to see how a candidate handles stress. They're just a small part of the process, and not the most important.
Joel Spolsky advises against them, arguing they are about "aha!" more than thinking:
That from http://www.joelonsoftware.com/articles/GuerrillaInterviewing3.html.
While Joel's writing about programmers, I'd highly recommend his whole series on hiring.
November 10th, 2007 at 07:53
[...] We found an excellent post while searching the Blogosphere today! Here’s a quick excerpt :
The model suggested in the email of, what I call, temp to perm is not a bad idea because it both allows you to test the person out but more importantly you can check other attributes like leadership and team fit etc. …
You can read the rest of this blog post by going to the original source, here [...]
August 20th, 2008 at 14:40
I would ignore the part about making only two breaks in the gold bar and give my employee a piece every day. If there are any ramifications as a result of my actions, then I'll deal with them when the time comes. Bottom line is, my employee was paid, consequently he was happy and, as a result, he produced good work.
January 25th, 2009 at 11:55
The 1:2:4 answer seems like the optimal one (given the
circumstances) but if the worker can't spend the pay until
after day four then you're not really "paying" him at all til
then. I considered that and then figured an employment contract to pay him weekly would suffice.
It's still a good test to see if the applicant is mentally
awake though. Also it may develop a two way trust to have him return with the money, trusting he could cover food and
transportation to return. :)
August 13th, 2009 at 00:58
I just found this blog.. and found interesting to answer the question
Per the question, The gold bar will have seven connected pieces.
I can break them into seven equal pieces. So, no need to bother about the equal payment (similarity in each piece, if I brake the piece). But the given condition is I have to break it only twice.
Here it goes.
1 2 3 4 5 6 7
These are seven pieces connected together.
1st day – I will break the 7th piece and pay the worker.
2nd day – I will break at 5th piece so that I will get two pieces connected (5th and 6th), I will ask the worker to give me 7th piece and pay him with 5th and 6th
3rd day – He already has 2 pieces (5th and 6th) now I will give him 7th piece, so that he will have three pieces
4th day – I will ask the worker to return all the pieces he have (5th, 6th and 7th) and I will give him the other piece which i have (1st to 4th connected pieces), so that the worker will have 4 pieces with him
5th day – I will give him the 7th piece, he will have five pieces with him
6th day – I will ask the worker to return the 7th piece and I will give him 5th and 6th pieces bar, so that he will have 6 pieces (1st to 4th and 5th to 6th)
7th day – I will give him 7th piece, so that he will have seven pieces with him.
Thanks
A
August 25th, 2009 at 14:11
Interesting read, thanks!