The Task of Predicting Performance for Complex Jobs

Pre-employment tests are used for a lot of different jobs—everything from entry level clerks to executives.  One of the best known users of tests is a consortium of 32 companies that uses tests for their 1760 “front of the house” employees.  I am speaking of the National Football League (NFL).

NFL teams have a very in depth selection process.  They hire scouts (think of them as recruiters) who are in charge of finding the best college football players in the country (high school players cannot go directly to the NFL, per the collective bargaining agreement between the players and the owners).  Sometimes the players are easy to find as they play for teams in conferences that show all of their games somewhere on TV.  Or, they travel to some remote outposts to see players who are on teams with less notoriety.

There is one thing you should know about the NFL owners.  For being a small group of very rich men (with a couple of exceptions) who made their fortunes in the free market system, they’ve adopted a very socialistic model for the league.  This model includes a provision where they choose talent from the labor poor in reverse order of how good the team is (called the draft).  So, instead of competing for the best talent, the team that did the worst the previous year gets to choose first, then the next worst team, all the way to the winner of the previous season’s Super Bowl (championship game).  This means that mistakes are very costly as 31 more players get picked before a team can make another choice.

The league also puts together days where all of the teams can watch the players run, throw, jump, etc.  College teams have “pro days” where their players train in front of the NFL scouts (all are invited).

As you can see, the NFL teams have the opportunity to observe the players perform the job they are likely to be hired for over at least a 3 year period (maybe longer in some cases).  They also get to put them through laboratory situations to measure specific abilities.  And, since the 1970s, the league has administered the Wonderlic Test to the players as well.

The Wonderlic is a pretty straight forward measure of intelligence.  Not surprisingly, players from colleges (e.g., Ivy League) that are pickier about admissions do better than those who will take pretty much all comers.  Also, offensive players tend to score higher than defensive ones.  The league doesn’t have any hard and fast rules on using the results, so teams could choose to use it (nor not) as they please.  Personally, my big concerns about the testing program are:

1)    There’s not any data supporting that it’s predictive (see Lyons, et al, 2009)

2)    I realize that pro football players are in the public view, so they don’t have the same level of privacy that the rest of us do.  But, do they really need their scores on an intelligence test made public?

3)    If we assume that the cognitive abilities required of a quarterback are different than a middle linebacker, wouldn’t we need separate validation studies/cut-off scores for each?

News now comes that the NFL has paid for the development of a new test (read about it here).  I find it amusing the NY Times shows it with a graphic which refers to a Rorschach test, which is nothing like an intelligence test.  But, that’s just lazy editing.

The NFL is generally keeping the test under wraps.  It was funny how the agents for the players complained that their clients couldn’t practice the test.  Imagine a friend of a job applicant being upset because his/her friend couldn’t see your pre-employment selection instruments before hand.  It is good to know that the developer got some input from subject matter experts.  There are a lot of test security issues, so I can how validating the test on current players would be tough.  A study with college players would also compromise the test in that they will be the ones taking it soon.

There’s a lot of temptation for using any pre-employment test in this kind of recruiting situation.  There are only so many good players that can help each team, and you may not get the chance to draft the ones that you really want.  Any edge in evaluating players under these rules is a huge competitive advantage.  I think the only question is whether there is something on a test that tells the team something unavailable through all of the game film.  The answer is probably yes–The ability to perform at a higher level when there is more pressure and the game is played by faster and more talented people.  Each subsequent performance level comes with a new set of challenges, and professional football is no exception.  If a team can use any test to figure that out better than the other teams, they will be more successful in selecting players.

Predicting human performance is an inexact science.  But, as the NFL shows, there are significant competitive advantages to doing it better than your competitors.  What are you doing to ensure you have this advantage?

For more information on pre-employment testing, test validation, skills assessment, and talent management, please contact Warren at 310 670-4175 or [email protected]

Really, We Can Do That

The Sunday NY Times had an interesting interview with Daniel Hendrix of Interface, Inc.  The primary focus of the discussion was on Hendrix’s views on work-life balance and delegation.  That message is important to leaders who are working way too much.  Unless you are about to find a cure for cancer, work can wait as there will always be more.  Spend time with your family (or, at least away from work).

The conversation also goes into hiring and interviewing.  While some of the things that Hendrix says he likes to interview for will make your legal antennae go up (asking about family?), others are potentially quite insightful (Does the person have work-life balance?  Does s/he understand principles of servant leadership?).  This is particularly true when he addresses candidates’ ability to see and address the “big picture” rather than doing things as they appear.  Hendrix admits that these are hard things to get at in an interview.  Is interviewing the best way to measure these kinds of complex attributes?  I don’t think so.

You can measure the attributes he is talking about in assessment centers.  This process puts leaders in realistic work situations and their behavior is recorded and assessed using objective scoring methods.  It’s much more powerful (and valid) to watch someone generate work product and/or behave in a situation than asking them to talk about past experiences in a behavioral interviewing setting.  This is because recalling what we’ve done is not objective.  This is not to bash behavioral interviewing, which when used well can yield valid and valuable information about candidates.  However, I’d much rather watch a person do something than tell me about it.  It’s more valid and I can provide them with specific behavioral feedback.

It’s good to see an executive thinking seriously about which attributes should be used in selecting future leaders.  It also shows that there are opportunities for HR to introduce innovative tools that executives don’t know about.  It helps us demonstrate our value in a language they can understand.  Don’t miss things kinds of opportunities!

For more information on pre-employment testing, assessment centers, skills assessment, and talent management, please contact Warren at 310 670-4175 or [email protected]

Another Example From California Law as to Why You Need Honest Performance Appraisals

I am more than open to the argument that most performance appraisal systems are mean, misguided, and do not improve performance.  Many are designed to be punitive rather than part of a coaching process.  Supervisors hate doing them because they are time consuming and they are ill-trained to provide developmental feedback.  Employees despise them because, well, who wants to hear that they are not perfect?  And don’t get me started on forced rankings, everyone being rated as acceptable to avoid conflict, and other potential problems with the process.

Ideally, HR participates in designing a process that truly reflects a person’s performance.  Managers are trained to share the information, good and not so good, in a way that is honest and promotes improvement and appropriate rewards.  The organization has the resources and opportunities available to those who are willing to put in the effort to add more value to the organization.  As a quick aside, the people who take the time to take these steps are already your top performers.

Yet, there comes a time where you need to demonstrate that a person is not pulling his or her weight and needs to be let go.  This may be to show (either to you or to them) that everything had been done to improve performance.  Or, to demonstrate that performance had been documented in the event of a legal case.

For the latter, here’s another example.  It contains the typical tortured logic of California labor law.  The company claims that the plaintiff was fired for poor performance.  She says it was because she was pregnant.  As the company documented that the action would have been taken regardless of the pregnancy, an initial verdict was overturned, which saved the company a lot of money.  This only happened because they were willing to track poor performance.  However, since the court also found that the timing reflected bias, the plaintiff could recover legal fees, but not collect damages.  Something tells me this will affect how lawyers argue these kinds of cases (and calculate damages/fees in the future).

The point isn’t whether this kind of baby-splitting (sorry for the pun) will ever make it to your state.  Rather, if a company is sued for wrongful termination, it will be asked to justify the action.  I do not believe that HR’s primary role should be compliance.  It is not a value-added activity and it gets in the way of productive activities.  However, performance evaluation is an opportunity for us to bring measurement, concepts of successful change, and an eye towards improvement to a compliance activity.

For more information on employee engagement, performance, and talent management, please contact Warren at 310 670-4175 or [email protected]

Thanks for coming by!

Please provide this information so we can stay in touch.