When Should You Stop Using an Assessment?

I’m guessing that you would not expect me to write about discontinuing the use of a pre-employment or pre-promotional assessment.  But, there are instances when it is appropriate to do so.

For instance, the National Football League (NFL) has decided to stop using an intelligence test that they had been using for years to evaluate new players.  I have written about the league’s use of the test before, so I won’t rehash the arguments about it here.  However, its reasoning for not using it any more really comes down to:

  1. They did not feel it was predictive.
  2. It led to a poor candidate experience (which, to the NFL means bad publicity).
  3. And those are two very good reasons not to use a test.

Another reason to discontinue the use of a test is when knowledge, skills, abilities, or personal characteristics (KSAPs) required of a job change.  At some point, administrative assistants stopped typing pages of documents, so a test of how quickly someone could manipulate a keyboard no longer made sense.  Changes in customer dynamics can impact KSAPs as well.  When working with a call center client, our validation data showed that personality tests that predicted performance for those taking phone calls were not effective for those who took customer inquiries via e-mail or chat.  This led to a change to how the tests were scored depending on the open position.

This does not mean you should automatically drop using assessments because a job changes or has converted to WFH from an office position.  However, knowing that for many people WFH is the new normal, it may be time to see if the work has really changed and the if that impacts the KSAPs.  If the status quo has held, you have your answer.  If there are some changes, then another validation study is likely in order.

The use of assessments, like many HR procedures, tends to take on a life of its own.  Once they are in place, there is a lot inertia (we have always done it this way) keeping them there.  It does not have to be that way.  A good job analysis and validation study can help you modify your testing tools so that you get high value from them.

What Are Your Company’s Selection Myths?

For North American sports fans, there is not a more public selection process than that National Football League (NFL) preparing for the annual talent draft.  This is the process for allocating new players (rookies) who have finished their college careers to the teams.  Players cannot sign a contract with any team they choose until after they complete their rookie contract with the team that drafts them.  Players not chosen in the draft can sign with any team.

Besides evaluating players based on their college games, the NFL teams also invite the top players to be evaluated at what they call a combine.  At the combine, players get interviewed by teams and are put through a variety of physical and medical tests.  Teams use all of this information to compare players against each other (by position) so they can make the best choices during the draft.

Of course, in reality, the top draft choices are made mostly based on the players performance in college.  Players at the best schools compete with and against other players who are likely to be drafted, so watching them perform in a game tells teams pretty much what they need to know.  And, as I wrote about last week, there is a big bias towards players who went to the “best” schools.  But, the teams do use information at the combine to inform them about players who they don’t feel they have good data on.  For instance, those who are recovering from injuries or played at schools that don’t compete against the top schools.

There’s only one problem:  There is very little data that supports that the “tests” given at the combine of predictive of success in the NFL.  This article about the problems in measuring hand size in quarterbacks provides just one example of that.

One can see how this all got started.  Quarterbacks need to be able to throw a ball well (with a lot of speed and accuracy) and to be able to hold on to it under pressure and having a large hand (as measured from the tip of the thumb to tip of the pinkie) would seemingly be related to both of those.  But, it’s not.  All quarterbacks grip the ball a little bit differently, regardless of hand size, to get the best results.  The article suggests that hand strength is the better predictor of quarterback performance and that it is unrelated to size.  But, those who evaluate quarterbacks just cannot let the size measurement go.

I am guessing that most of your organizations have an unproven selection myth, such as, “Our best managers have gotten their MBAs from one of 10 schools” or “Good supervisors are those who have worked their way up in our industry” or “Our most successful programmers had previous experience at specific companies.”  I used to hear, “Our best call center agents have previous experience before coming here” all of the time.  But, when I conducted validation studies in contact centers, it was rare that previous experience was a good predictor of future performance. These myths are easy to evaluate, but changing HR practices is harder.  It often requires good data and a shift in culture to change thinking.  However, moving on from myths is often required to make better talent decisions.

Adapting Selection Systems After the Robots Take Over

I am not sure that any HR futurist can tell us how many jobs will be displaced by automation over the next 5, 10, or 20 years. The answer is clearly more than zero. The latest example of this can be read here. The theme of the article is, “Really, a formula can make predictions better than a person’s intuition?” In psychology (well, industrial psychology), we have only known this since the mid-1950s (see this book), so I can see why the idea is just catching on.

Any kind of judgment that is made based on accumulating data will ALWAYS be more accurate over time when done by a machine than a person. This is because the machine is not biased by what has happened most recently, how impacted it is by the decision, how attractive the others who are involved are, etc. While this type of analysis is somewhat difficult for people to do consistently well, it is simple math for a computer. There is really no reason, besides stroking someone’s ego, to have humans do it.

As computers continue to remove the computational portions of jobs, such as analyzing trends, making buying decisions, they will impact HR in the following ways:

• Fewer customer facing jobs to manage, but more IT related ones.

• Many of the remaining jobs will require less cognitive ability and more interpersonal skills. This is because these employees could potentially spend more time meeting specific customer needs and being the interface between end users and the algorithms.

• The key predictors of job success would potentially become conscientiousness, agreeableness, and customer service orientation rather than problem solving ability.

• Developing a validating a different set of pre-employment tests.

• Recruiters will need to source people with very specific skills (cognitive ability for programmers and willingness to get along with others for many other jobs).

The challenge to industrial psychology continues to be developing more valid measures of personality. Tests of cognitive ability predict job performance about twice as well as those of “soft” skills, even in those that already have a high personality component (such as customer service). This also means developing better measures of performance (e.g., how interpersonal skills impact business outcomes).

Or, maybe the robots will do it for us.

Thanks for coming by!

Please provide this information so we can stay in touch.

CLOSE