Assessing Talent is HARD—Ask the NFL

The National Football League (NFL) is holding its annual talent draft tonight (as I write this).  So, yes, you’re getting two sports related blog posts this week.  What’s fascinating about this is how the 32 teams evaluate a large pool of potential players and how wrong they are in the most critical evaluations.  If you’re familiar with the NFL draft, you can skip ahead past the bullets.  If not, here’s a quick primer:

  • Each year, the NFL forces the top players who have not yet played in the NFL (think of guys who played college football) to participate in a process where teams select which players that they want (the draft).  The order of the draft is determined by the teams’ records the year before, with the worst team (Jacksonville) going first and the team that won the championship (Tampa Bay) going last.  There are 7 rounds in the draft.
  • If a team wants a particular player, but thinks that he will be taken before it is their turn, they can trade players and/or their draft picks for a better pick.  This is called trading up and can be very expensive.
  • The most important employee (player) on an NFL team is the quarterback.  Teams that do not have an “elite” player at this position will do almost anything to draft one.  There are economic incentives for drafting a quarterback rather than trying to get an experienced one from another team.
  • Each team has an entire department of people whose sole job is to evaluate the skills and abilities of college players, determine the needs of the team, and recommend players to draft and in which order.  The NFL even sets up what are essentially job fairs where the teams can look at players up close, interview them, and they even give a league-wide cognitive ability test.  These players really get put under the microscope.

So, we have a situation where the skill and ability levels of the players are well known.  Given that, you would think that teams can select players, especially quarterbacks, with uncanny accuracy.  And you’d be wrong.

If we look at the quarterbacks drafted in the 1st round (63) since 2000, it’s pretty fair to say that (not including those who are too early in their career to judge):

  • 15 turned out to be very good (consistently led their team to a winning record and won playoff games)
  • 13 were/are good (there teams won more than they lost, but were nothing fabulous)
  • 30 were not good (losing record, didn’t stay with the team long, etc.). 

Those of you who are football fans may quibble with the categories where I put the players, but even if we move players around a bit between the categories, teams have about a 25% success rate and about a 50% miss rate.

In roughly the same time period, teams have “traded up” 25 times to draft a quarterback in the 1st round.  In other words, they paid a premium to “get their guy.”  14 of them were/are not very good, 3 were/are good, and 6 were/are very good.  Again, we see this 50% miss rate, though slightly higher hit rate.

How can this be?

Most obviously, football is a game where all of the players on the field need to do their job well for the team to be successful.  Like it or not, the quarterback gets outsized credit and blame for the team’s success and failures.  But, regardless of his skill level, he cannot control a lot of what goes on around him.  This makes for a very difficult selection situation.

There are also some confounding factors.  The first being that the worst teams pick before the good ones and they are more likely to choose a quarterback than good teams.  And, one of the reasons these teams are not very good is that they don’t judge talent well and do a poor job of developing and retaining the good players that they have.  Related to this, they often take a quarterback before they have good supporting players around him.  It’s entirely possible that several of the quarterbacks who were drafted early, but performed poorly, would have been successful with other teams.  And, a few of them, but not a lot, were after they changed teams.

To make the success rate seem even worse, teams choose to ignore players are other positions that are more likely (historically) to be successful in order to make the sexier, and riskier, play for a quarterback.  And I get that—there are enough examples of a highly drafted quarterback completely changing a team’s fortunes that teams are willing to miss 2 out of 3 times to get one.  Any team would be happy to draft 2 not so good quarterbacks to get to the one that leads them to a championship, even if it takes them several years to do it.

What we do know is that new quarterbacks tend to succeed where the team has a sustainable plan.  Often times, teams change their playing philosophy if they don’t have a successful year so they fail to implement one that fits the quarterback’s skills.  Successful quarterbacks rarely play for multiple head coaches (managers).  The lesson here is that good plan that senior management sticks with is more likely to be one where your most expensive talent can succeed.

It used to be that newly drafted quarterbacks would have a more experienced one ahead of him to learn from for a year or two before being elevated to leading the team.  There are many reasons why that is not the case anymore, but it does limit the opportunity for some drafted players to develop into good quarterbacks. Players drafted at other positions tend to have the luxury.  Organizations that realize which key players can contribute immediately and which ones need some seasoning are also more likely to get the most out of their talent.

The NFL (and other professional sports) draft is a unique selection system.  In some ways it allows talent evaluators to get a great deal of information, but in other ways there is a lot of mystery around what allows players to translate their college success into the pros.  And, in looking at success vs. failure of the quarterback, sports fans tend to ignore the organizational issues that allow some players to shine rather than flounder.  All good lessons to keep in mind as companies begin to hire recent college graduates.

And good luck to Trevor Lawrence.

A View From the Other Side

As long as there are pre-employment tests there will be people who do not like taking them. That is fair—most people would like to get the job of their choosing without jumping through hoops or by going through some process which perfectly recognizes their unique (and superior) skills compared to other applicants. But, that is not the reality that we live in (or one that is fair to all applicants). But, we must be mindful of how those who take tests and go through interviews see them. We want their experience to be one that would encourage them to recommend others to apply at a particular company or use them when they are in a position to hire/promote others.

This article is not atypical of what industrial psychologists hear about tests. 

“The test was stupid.” 

“It did not measure skills or abilities that I would actually use on the job.” 

“I did not have an opportunity to show my true skills during the process.”

But, the author does more than complain.  He offers suggestions that he (and the singular is important, because the all of the comments on the article do not support his positions) thinks would improve the hiring process.  Listening to test takers who want to improve the process, and not just get a free pass, can lead to valuable improvements in your systems.

In my experience, the top 3 things that candidates want from a testing experience are:

  1. Convenience.  The industry has gone a long way towards this by adapting to mobile technology and shortening personality and aptitude tests.
  2. Something that looks like the job or their expectations of it.  Sometimes this means interacting with others rather than just solving a problem individually.  Or, answering questions where the process is as important as the answer (since many real life work problems have more than one solution).  When a portion of the assessment does not feel like the job, candidates are more likely to exit the process.
  3. Not feeling as if they are being “tricked.”  This can range from being asked (seemingly) the same question more than once on a personality test to impossible brain teasers.  While the former can have some statistical value, Google and others have found that the latter does not. 

Pre-employment and promotional testing is a zero-sum game.  Many people, due to the fundamental attribution error, are more than willing to fault the process than themselves.  That is fine and as assessment and interview developers and users, we should listen to them.

Can We Accurately Evaluate Leadership Before Someone Has a Chance to Lead?

In general, our personalities are pretty stable over our adulthood. Yes, we mature and big life events can alter us, but the building blocks of who we are as people are closer in stability to our eye color than our hair color.

This stability has been important to the science of employee selection. The underlying idea of giving any type of pre-employment or promotional test is that the knowledge, skill, ability, or characteristic being measured is stable in that person for a given period of time so that it can be used to predict future performance. With skills, we assume that they will improve over time, so we look for those that a person has right now. For personality and cognitive abilities, we assume that a person will have those at a consistent level for many years and that these aptitudes can be used to develop specific skills, such as leadership.

When I conduct leadership workshops, I typically ask participants if leaders are born (e.g., do some people just have what it takes) or made (e.g., pretty much anyone can be an effective leader if given the right opportunities to develop). The conversation gets people thinking about what behaviors are necessary to lead (good communication, willingness to direct others, attention to details, etc.), which of those can be taught, and which cannot. Put another way, to become a professional basketball player, I can improve how well I shoot, but I cannot do too much about how tall I am.

But, what if we have the direction of trait to leadership wrong? What if the traits to become a leader don’t blossom until someone is given the chance to lead?

This study suggests that being promoted into a leadership position does change the conscientiousness factor of personality. Conscientiousness has been found to be a significant predictor of overall manager effectiveness. It’s an interesting idea in that it suggests that, for some people, we do not know if they have a sufficient amount of a trait that contribute to leadership success until after they become leaders.

As with all good research, it poses as many new questions as answers. For instance, were there increases in conscientiousness across the spectrum or only among certain groups (e.g., were there gains for those who already showed relatively high levels of conscientiousness, so the rich got richer)? Or, does it take a leadership experience to bring out conscientiousness in people who typically do not show it? Or, is leadership a tide that raises everyone’s conscientiousness?

Practically speaking, this is where the study has me thinking about assessing leadership:

1)  Putting a re-emphasis on using performance on temporary assignments that involve leadership as part of the selection process in promoting people into supervisory positions. 

2)  Validating responses on personality tests that are taken after a person goes through a leadership role-play exercise or situational judgment test.

3)  Re-thinking what aspects of personality indicate leadership potential (e.g., willingness to direct others and resilience) and broaden our list of things that are leadership skills to include some other aspects of personality (e.g., conscientiousness). We can then focus on selecting based on the former and training on the latter.

Some people have the right mix of attributes that allow leadership to come easily to them. As it turns out, some of those things become more apparent after a person has a chance to lead. This should encourage us to think about how we choose to evaluate leadership potential.

Making Work From Home Work

OK, let’s all take a deep breath and say it out loud, “Work from home is not going to be temporary.”  There, we did it.  Now let’s talk about it.

We need to acknowledge that WFM is not going to be one size fits all from an organization’s standpoint.  Some companies will look for as many people to return to the office as soon as possible as their CEOs swear that productivity and creativity can only occur face-to-face.  Others, after realizing the cost savings of reducing their office footprint, reduced stress of employees not commuting, and maintained productivity will encourage it going forward.

Eight or so months into WFM, we also need to acknowledge that it is not for everyone.  Some people (well organized introverts) thrive in the environment while others (people with a high need for affiliation) struggle in it.  And there are many who are in-between.  They would like to interact and work with others a few days a week, but appreciate those days where they can work with fewer distractions and in a safer environment.

Given these variables, how can we adapt to WFM or make it work for organizations and employees?  Let’s walk through some big steps and see where the key points are.

  1. Organization Design.  This is the time for companies to look at their WFM experiences and make policy decisions.  Some of this should be business driven (Are we productive with what we are doing now? Do COVID-19 impacts on our business support how much space we have?). There should be employee input via a survey, or some other method, that allows everyone to have his/her/their voice heard.  Don’t assume you know what everyone wants and be sure to stay away from just a couple of anecdotes.  From this info you can then develop go forward policies.  Oh, and don’t worry—not everyone is going to like the new policies.  Just like before COVID-19 when there were people who wanted to work from home but were not allowed to.
  2. Implementing Change.  Most WFM policies I’ve seen have been done on the fly—which is not a criticism.  Rather, most employees saw the impact of COVID-19 so there was not the resistance to change that we normally see when organizations choose to pivot.  However, as you transition from a reactive state to the new normal one, you’ll need to use your change management techniques, such as:

    a. Be transparent—describe to employees the data you gathered and why WFM policies are either going back to pre-COVID-19 or changing.

    b. Describe the benefits of the change from an employee’s perspective.

    c. Have resources (technical, informational, skill development, etc.) available that support the change.

    d. Measure the impact of the new policies so you can make future decisions based on data.
  3. Recruitment.  Once a WFM policy is in place, you’ll want to be able to describe it clearly when attracting new talent.  Your WFM requirements, or lack thereof, will be attractive to some and repellent to others.  And that’s OK.  Just be sure to let people know what they are in for.
  4. Selection.  Remember before when I talked about which people thrive or suffer in WFM?  This is important information to use when selecting new employees going forward.  You will want to review your job descriptions and competency models to be sure they include any changes that would come from your WFM policies.  For instance, if you are moving to a model that allows or mandates a lot of working from home, you will want to include characteristics like need for autonomy in your selection protocols.  Be sure to validate any tools you use to measure this and other skills/abilities/personal characteristics.
  5. Training and Development. Despite the occasional video conference faux pas, I think most of us were pleasantly surprised how quickly people gained skills at using this software.  Those in the learning and development area have also adapted their materials and approaches to video.  You should be providing training on how to get the most out of not only video conferencing technology, but other tools that allow people to collaborate across distances.  In a WFM environment, some people will benefit significantly from instruction on how to be productive in a home environment.  Also, continue to keep up with professional and leadership development so that people do not stagnate.  There are a lot of great online training experiences out there and without travel expenses the true cost has gone down.
  6. Performance Management.  An employee concern about WFM has been, “How will I get promoted?”  The manager’s concern is, “How will I get productivity?”  Of course, these two things go together.  WFM has exposed poorly designed performance goals and objectives.  It is likely that your business has changed post-COVID.  It is a good time to revisit your performance standards and measurements as well.  As before, develop S.M.A.R.T. goals, but now with an extra emphasis on measurable, especially if people are participating in WFM.  This helps to minimize lack of face time issues as employees can more easily demonstrate their performance. 
  7. Coaching and Feedback. WFM minimizes spontaneous interactions between employees and their managers.  This means that they both need to make additional efforts to schedule conversations to stay in touch.  This will help with the visibility employees want and the accountability managers are looking for. There should be some structure to these conversations so that they cover current performance as well as career development.
  8. Managing the Change.  Whether you think we are closer to the beginning or the end of the pandemic, we can all agree that there are going to be more changes to business and WFH before it is behind us (e.g., another wave of cases, kids going back to school or back to distance learning, etc.).  Just as you want to gather employee and other stakeholder information when forming policies, you will want keep track of how attitudes towards the policies, the pandemic, and the business shift.  Acting on these issues as necessary will help you maintain or increase employee engagement and make policy adjustments.

Our work lives have always involved managing change and COVID-19 has made this more pronounced.  It is now time to do so in a more thoughtful and forward looking way.  We can do this more effectively by recognizing that some changes, including WFH, are not temporary adjustments, but are permanent in one form or another.  HR benefits employees and organizations when it recognizes the wide-spread impact of WFH and adjusts its practices and manages the change effectively.

Silver Linings to Losing Some of Your Selection Processes

COVID-19 is, and will continue to, affect many parts of our work processes.  One of them is how we select new employees. Yes, even with layoffs some companies are hiring now and most will be again before the end of the year.  With social distancing and the acceptance of video-conferencing, we are beginning to accept that how we select candidates will change.

This does provide for a process improvement opportunity in what we do.  Are all of the current steps we use necessary or are some based on myth?  For instance, the National Football League is going forward with their big selection weekend at the end of the month, but there are concerns from those who evaluate the candidates that they do not have access to the tools that they normally would in doing their final rankings.  I am guessing that they will find that some of those tools are for making people feel important in the process and do not really add a lot of value in finding meaningful differences between players.  You may find that some aspects of your process are redundant or done for the sake of tradition rather than adding value.

Here are some selection traditions that we are going to have to let go of for a bit and the silver linings associate with the changes:

  1. Face to face interviews.  Whether social distancing is officially with us for four more weeks or four more months, the hesitancy to be physically close to others will likely be with us for a while.  People are becoming more comfortable and adept with video calls and we should continue to utilize them.  Silver lining:  In areas with heavy traffic, the video calls are easier to schedule for both parties.

  2. Virtual assessments.  Whether it is for skills and personality testing, or role-plays, assessments have been moving online for several years and the current situation will likely convert some who have not yet made the switch.  Silver lining: giving these assessments online is very efficient.  The reduced cost improves their business impact and will make it easier to process candidates when hiring picks up again.

  3. Being ultra-professional.  Being interviewed or assessed online was a way to put one’s best professional foot forward.  Doing so from home, with kids and pets around, is going to chip away at the veneer.  Silver lining: While I feel for the candidate who is trying to respond to a question with a barking dog in the background, I do think that interviewees will bring forth more of their authentic self.  Whether this leads to a more valid process is an open question.  But, hiring managers and HR will have a better idea of the “real” person being hired.

In HR we often talk about implementing change, but this is a time where we also need to be the leaders of it in our own areas.  Let’s skip the denial of what is happening and ditch the resistance to new ways of evaluating candidates.  I think we will be pleasantly surprised with the results.

What Are Your Company’s Selection Myths?

For North American sports fans, there is not a more public selection process than that National Football League (NFL) preparing for the annual talent draft.  This is the process for allocating new players (rookies) who have finished their college careers to the teams.  Players cannot sign a contract with any team they choose until after they complete their rookie contract with the team that drafts them.  Players not chosen in the draft can sign with any team.

Besides evaluating players based on their college games, the NFL teams also invite the top players to be evaluated at what they call a combine.  At the combine, players get interviewed by teams and are put through a variety of physical and medical tests.  Teams use all of this information to compare players against each other (by position) so they can make the best choices during the draft.

Of course, in reality, the top draft choices are made mostly based on the players performance in college.  Players at the best schools compete with and against other players who are likely to be drafted, so watching them perform in a game tells teams pretty much what they need to know.  And, as I wrote about last week, there is a big bias towards players who went to the “best” schools.  But, the teams do use information at the combine to inform them about players who they don’t feel they have good data on.  For instance, those who are recovering from injuries or played at schools that don’t compete against the top schools.

There’s only one problem:  There is very little data that supports that the “tests” given at the combine of predictive of success in the NFL.  This article about the problems in measuring hand size in quarterbacks provides just one example of that.

One can see how this all got started.  Quarterbacks need to be able to throw a ball well (with a lot of speed and accuracy) and to be able to hold on to it under pressure and having a large hand (as measured from the tip of the thumb to tip of the pinkie) would seemingly be related to both of those.  But, it’s not.  All quarterbacks grip the ball a little bit differently, regardless of hand size, to get the best results.  The article suggests that hand strength is the better predictor of quarterback performance and that it is unrelated to size.  But, those who evaluate quarterbacks just cannot let the size measurement go.

I am guessing that most of your organizations have an unproven selection myth, such as, “Our best managers have gotten their MBAs from one of 10 schools” or “Good supervisors are those who have worked their way up in our industry” or “Our most successful programmers had previous experience at specific companies.”  I used to hear, “Our best call center agents have previous experience before coming here” all of the time.  But, when I conducted validation studies in contact centers, it was rare that previous experience was a good predictor of future performance. These myths are easy to evaluate, but changing HR practices is harder.  It often requires good data and a shift in culture to change thinking.  However, moving on from myths is often required to make better talent decisions.

Who’s Next?

My process improvement friends like to say, “Improving the work is the work.”  There is some truth to that in HR as well.  But, I think that it is also fair to say that “Keeping the talent pipeline full is the work.”  I’ll admit that it’s less catchy.

Succession planning is a topic as old as business, so I will cut to the chase:  This may be an area where companies are getting better.  The data is interesting as well in that it shows (at least in this sample) that public companies are better at it than private ones.  I would be curious as to whether there is an additional split between family owned and other types of ownership among the private companies.

I think that transparency (welcomed or not) has a lot to do with boards (and, hopefully, HR) being more concerned about high level succession planning.  Part of what big investors are buying is the leadership team and the more focus there is on CEOs and their impact, the more concern investors will have in the less-than-famous leaders.

Good succession planning does not stop at the C-Suite.  It should be considered part of talent development for every position in the company.  Whether it is at the entry (where are we going to find new employees?) or management (how can we identify leadership potential?) levels, the work is ensuring that there is a strategy for identifying talent.

This process involves both valid assessment (who is interested and capable of doing what we need?) and development (what are the experiences that a person needs to be prepared for the next move?).  Keeping the pipeline full means focusing on both so that when a position comes open the question, “Who’s next?” can be answered quickly and reliably.

Are we Biased AGAINST Top Talent?

We all want to believe that we are looking to recruit, select, and develop top talent.  We spend lots of time reading and writing articles on the topic.  But, what if hiring managers are not interested?

This article throws a bit of cold water on the topic.  It documents a study where hiring managers were shown to doubt the organizational commitment of those deemed the most capable.  It was almost as if they were saying, “Why would someone really good want to work for us?”

There are several issues at work here.  But, what they boil down to is a bias among hiring mangers that negatively affects their selection processes.  Sure, I can imagine anecdotal evidence (“Yeah, we hired that one really bright person, but she jumped ship as soon as she got a better offer.”), but I don’t think that this is a data driven decision.

What this also underlines is the importance of developing a culture that encourages top talent to stay.  There’s no question that selecting the right people will drive business performance.  And having a culture that acknowledges and rewards high performance will do so as well.  When hiring managers feel that top talent will not stay, it is really an indictment of the culture rather than an accurate prediction of management’s view.  How can you fight this?

  1. If managers do not think top talent would be committed to your organization, they should NOT be involved in hiring. 
  2. Those who are doing the hiring should be able to provide a realistic preview of the organization, but should also be able to succinctly describe why people stay.  And I’m not just talking about a good cafeteria.  They should be able to provide examples of people who have found challenging work over time in the organization.
  3. If you are speaking with hiring managers who show an anti-talent bias, ask them what needs to be changed so they would believe that top talent would want to stay.
  4. The best way to fight bias is with data.  You should be able to study turnover rates by talent bands (contact me for tips on this).  This way you can either show people that top talent does not leave any faster than other employee groups or demonstrate to executives that this is a problem that needs to be addressed.

Organizations should strive for selection processes that identify top talent and cultures that nurture them.  Do not let bias against hiring top talent work against these two initiatives.

Selecting Managers Who Understand the Value of Praise

When I do leadership/management workshops, the first topic is always motivation.  While I am a big believer that motivation must come from within, managers can impact performance, in the short term, by effectively using rewards.

Years of research tells us that cash and other extrinsic rewards can be effective motivators for tasks where individual effort leads to individual results.  However, the bigger the distance between effort and results, the less value these incentives have.  Oh, and they also lose their effect over time.

The wise manager knows that recognition, praise, and other behaviors that lead to intrinsic rewards are much more powerful. This article provides a good synopsis on how to use a combination of intrinsic and extrinsic rewards.

While there tends to be a strong focus on rewards, something that gets overlooked is how to select managers who already have this insight.  Sure, most can learn it. But, I would think that there are traits that predict how well a person rewards employees.  Three of these would include:

  • A person with a high level of agreeableness is usually warm, friendly, and tactful. They generally have an optimistic view of human nature and get along well with others.  People high on this trait are likely to want to make others feel engaged in their work.
  • Generous people are the ones who give more than is expected of them.  Giving a reward to another person is an act that provides praise or a reward to another person when it could be kept to oneself.
  • View of Employees. Managers who have a “your paycheck is your reward” mentality are not likely to give out a lot of praise.  Those who recognize people as individuals, and learn what their needs are, will be much more likely to provide meaningful motivators.

By making motivational skills part of the valid selection process, we are more likely to hire managers who will seek out opportunities to reward results.  Appropriate use of such techniques will lead to more engaged and productive employees.  They are less likely to turnover, which is critical in our current low unemployment economy.

Adapting to Changes in Job Duties

I wrote a couple of months ago about how McDonald’s is changing the cognitive requirements of some of its jobs by adding channels for customers to order food. I argued that such a development should get them thinking about who they hire and how they train new employees.

If you have recently wandered into one of their stores, you probably noticed that, if it is not too busy, a McDonald’s employee may bring you your order. OK, this is not particularly revolutionary. But, to quote a franchisee in an article, “We’re bringing the employees from behind the counter out front to engage, in a more personal way, with our customers.” Maybe I am making more out of this particular example than it warrants, but this strikes me a really upping the customer service requirements of a McDonald’s employee. And I am guessing that a fair amount of the employees are not going to meet it. It’s just not what they signed up for.

This is not about whether McDonald’s employees are capable of providing the additional service or whether their ability to do it well affects the customer experience and/or sales. Rather, it appears to be an example of company changing job requirements and then assuming that people hired using a process that does not account for the new skills will be able to carry out the new duties.

Changing skills requirements is a good thing. It shows adaptation to technology and customer needs and makes the work experience more interesting for people in repetitive jobs. But, companies cannot assume that the incumbents can magically adapt without training and revised performance expectations.

This change also requires updating validation selection processes. Whether it means increasing the weight given to certain aspects or validating a new test, we must adapt our workforce to new job requirements on the front end. As jobs change, hiring practices should as well.

Technology and customers are big drivers of change in the skills, abilities, and personality characteristics required of employees. Smart companies not only redesign work to account for this, but they also update how they train and hire to help their workforce adapt.

Thanks for coming by!

Please provide this information so we can stay in touch.

CLOSE