Training Hiring AI Not to be Biased

Artificial Intelligence (AI) and Machine Learning (ML) play integral roles in our lives.  In fact, many of you probably came across this blog post due to a type of one of these systems.  AI is the idea that machines should be taught to do tasks (everything from search engines to driving cars).  ML is an application of AI where machines get to learn for themselves based on available data.

ML is gaining popularity in the evaluation of job candidates because, given large enough datasets, the process can find small, but predictive, bits of data and maximize their use.  This idea of letting the data guide decisions is not new.  I/O psychologists used this kind of process when developing work/life inventories (biodata) and examining response patterns of test items (item response theory—IRT).  The approaches have their advantages (being atheoretical, they are free from pre-conceptions) and problems (the number of people participating need to be very large so that results are not subject to peculiarities about the sample).  ML accelerated the ideas behind both biodata and IRT, which I think has led to solutions that don’t generalize well.  But, that’s for another blog post.

What is important here is the data made available and whether that data is biased.  For instance, if your hiring algorithm includes zipcodes or a classification of college/university attended, it has race baked in.  This article has several examples of how ML systems get well trained on only the data that goes in, leading to all kinds of biases (and not just human ones).  So, if your company wants to avoid bias based on race, sex, and age, it needs to dig into each element the ML is looking at to see if it is a proxy for something else (for instance, many hobbies are sex specific).  You then have to ask yourself whether the predictive value of that bit is worth the bias it has.

Systemic bias in hiring is insidious and we need to hunt it down.  It is not enough to say, “We have a data driven system” and presume that it is not discriminatory.  If the ML driving it was based on inadvertent bias, it will perpetuate it.  We need to check the elements that go into these systems to ensure that they are valid and fair to candidates.

I’d like to thank Dennis Adsit for recommending the article from The Economist to me.

Blacks Welcome to Apply

The aftermath of George Floyd’s murder has many of us asking, “What can I do better?” when it comes to ending racism.  This is critical in that racial bias in hiring have changed little in 30 years.  HR and I/O psychology play a unique role in that we create the processes that allow for equal employment.

None of the suggestions below require lowering of standards.  Rather, it provides a framework for applying standards in an equitable way.  Science and good sense points us in this direction with these actions:

  1. Widen your recruitment net.  If you recruit from the same places, your workforce will always look the same.  There is talent everywhere—go find it.  Whether from a high school in a different part of town or a historically black college/university.
  2. Make Resumes Anonymous.  The science is very clear that anonymous resumes reduce racial and gender bias.  It is not an expensive process to implement and works for all kinds of business.
  3. Examine minimum qualifications carefully.  Whether based on job experience or education, these can serve as barriers to black job candidates.  The ground breaking employment discrimination lawsuit, Griggs v. Duke Power, was based on an invalid requirement that supervisors needed a high school diploma.  Don’t get me wrong—I want my surgeon to be an M.D. But, do your entry level positions really need a college degree?  Do your managers really need to be MBAs?  If you analyze the relationships between education/experience and job performance, you are likely to find that they are not as strong as you think.
  4. Use validated pre-employment and promotional tests.  As a rule, validated pre-employment tests do not adversely affect blacks and are certainly less biased than interviews (see below).  This is particularly true for work sample tests (show me what you can do) and personality tests.  However, cognitive ability tests, especially speeded ones, may lead to discrimination.  If you use them, analyze your cutting score to ensure that it is not set so high that qualified candidates are being screened out.
  5. Reduce reliance on interviews.  Interviews can be biased by race and ethnicity.  And, more often than not, they are far less valid than tests.  We need to convince hiring managers that they are not good judges of talent—very few people are.  Remember, interviewing someone to see if s/he is a “good fit” is another way of saying, “this person is like me.” 

  6. Make your interviews more structured.  This can be achieved by asking candidates the same questions and using an objective scoring methodology.   Adding structure to the interview process can reduce bias (and improve validity).

You may already be doing some of the above.  I would encourage you to do all of them.  The outcome is fairness AND better hires.  What could be better than that?

Valid Virtual Employee Selection

I wrote a couple of weeks ago about how the National Football League (NFL) had to adapt their selection procedures to deal with the pandemic.  To recap, the NFL selects new players primarily through a draft of eligible college football players.  Leading up to the draft teams review the players’ performance in previous games, have them go through physical examinations, athletic drills, personality and cognitive tests, structured interviews, and background investigations.  However, with COVID-19, the NFL ruled out many of these things for health reasons.

It is much too early to tell if the slimming of the selection tools impacted the effectiveness of any team’s draft.  However, there are two observations that can be made:

  1. The order of the most talented players chosen was pretty much what was expected by experts back in January.  26 of the first 32 players drafted were predicted (by one expert), with 7 of the first 8 going to the predicted team as well.  This is pretty typical.

  2. The lack of some the selection tools appeared to hurt those who attended smaller and/or not as well-known schools.  Typically, about 18 players from such schools are taken in the draft.  This year, only 6 were.  With a lack of information, teams may not have known, or wanted to take a risk, on such players.

For the latter, this is not a case of re-arranging crumbs.  Some of the best players in the NFL have come from these schools, so the teams lose a competitive advantage when they don’t properly identify relatively unknown talent. 

What we saw is easily explained: Past performance is the best (but not perfect) predictor of future performance.  The teams could evaluate how well players from the bigger schools performed against similar talent in college.  The NFL did not have, and did not develop, tools to uncover the best players who did not have the opportunity to play against other very talented players.  So, they relied on what they knew best.  But, this resulted in opportunity costs for them and created a slew of players with chips on their shoulders.

Since this selection event takes place once a year, it is likely that the NFL draft will (largely) return to normal next year.  But, what if it doesn’t? Or, in the future there is another interruption?  The teams that find alternative (and equally valid) methods of evaluating talent will benefit.  Your company should be thinking in the same vein during COVID-19 and beyond.

Silver Linings to Losing Some of Your Selection Processes

COVID-19 is, and will continue to, affect many parts of our work processes.  One of them is how we select new employees. Yes, even with layoffs some companies are hiring now and most will be again before the end of the year.  With social distancing and the acceptance of video-conferencing, we are beginning to accept that how we select candidates will change.

This does provide for a process improvement opportunity in what we do.  Are all of the current steps we use necessary or are some based on myth?  For instance, the National Football League is going forward with their big selection weekend at the end of the month, but there are concerns from those who evaluate the candidates that they do not have access to the tools that they normally would in doing their final rankings.  I am guessing that they will find that some of those tools are for making people feel important in the process and do not really add a lot of value in finding meaningful differences between players.  You may find that some aspects of your process are redundant or done for the sake of tradition rather than adding value.

Here are some selection traditions that we are going to have to let go of for a bit and the silver linings associate with the changes:

  1. Face to face interviews.  Whether social distancing is officially with us for four more weeks or four more months, the hesitancy to be physically close to others will likely be with us for a while.  People are becoming more comfortable and adept with video calls and we should continue to utilize them.  Silver lining:  In areas with heavy traffic, the video calls are easier to schedule for both parties.

  2. Virtual assessments.  Whether it is for skills and personality testing, or role-plays, assessments have been moving online for several years and the current situation will likely convert some who have not yet made the switch.  Silver lining: giving these assessments online is very efficient.  The reduced cost improves their business impact and will make it easier to process candidates when hiring picks up again.

  3. Being ultra-professional.  Being interviewed or assessed online was a way to put one’s best professional foot forward.  Doing so from home, with kids and pets around, is going to chip away at the veneer.  Silver lining: While I feel for the candidate who is trying to respond to a question with a barking dog in the background, I do think that interviewees will bring forth more of their authentic self.  Whether this leads to a more valid process is an open question.  But, hiring managers and HR will have a better idea of the “real” person being hired.

In HR we often talk about implementing change, but this is a time where we also need to be the leaders of it in our own areas.  Let’s skip the denial of what is happening and ditch the resistance to new ways of evaluating candidates.  I think we will be pleasantly surprised with the results.

What Are Your Company’s Selection Myths?

For North American sports fans, there is not a more public selection process than that National Football League (NFL) preparing for the annual talent draft.  This is the process for allocating new players (rookies) who have finished their college careers to the teams.  Players cannot sign a contract with any team they choose until after they complete their rookie contract with the team that drafts them.  Players not chosen in the draft can sign with any team.

Besides evaluating players based on their college games, the NFL teams also invite the top players to be evaluated at what they call a combine.  At the combine, players get interviewed by teams and are put through a variety of physical and medical tests.  Teams use all of this information to compare players against each other (by position) so they can make the best choices during the draft.

Of course, in reality, the top draft choices are made mostly based on the players performance in college.  Players at the best schools compete with and against other players who are likely to be drafted, so watching them perform in a game tells teams pretty much what they need to know.  And, as I wrote about last week, there is a big bias towards players who went to the “best” schools.  But, the teams do use information at the combine to inform them about players who they don’t feel they have good data on.  For instance, those who are recovering from injuries or played at schools that don’t compete against the top schools.

There’s only one problem:  There is very little data that supports that the “tests” given at the combine of predictive of success in the NFL.  This article about the problems in measuring hand size in quarterbacks provides just one example of that.

One can see how this all got started.  Quarterbacks need to be able to throw a ball well (with a lot of speed and accuracy) and to be able to hold on to it under pressure and having a large hand (as measured from the tip of the thumb to tip of the pinkie) would seemingly be related to both of those.  But, it’s not.  All quarterbacks grip the ball a little bit differently, regardless of hand size, to get the best results.  The article suggests that hand strength is the better predictor of quarterback performance and that it is unrelated to size.  But, those who evaluate quarterbacks just cannot let the size measurement go.

I am guessing that most of your organizations have an unproven selection myth, such as, “Our best managers have gotten their MBAs from one of 10 schools” or “Good supervisors are those who have worked their way up in our industry” or “Our most successful programmers had previous experience at specific companies.”  I used to hear, “Our best call center agents have previous experience before coming here” all of the time.  But, when I conducted validation studies in contact centers, it was rare that previous experience was a good predictor of future performance. These myths are easy to evaluate, but changing HR practices is harder.  It often requires good data and a shift in culture to change thinking.  However, moving on from myths is often required to make better talent decisions.

Adjusting Your HR Strategy When Your Company Decides to Train For Basic Job Skills

There is a presumption that the US education system will provide employers with workers that possess requisite job skills.  Companies are then responsible for providing more advanced ones through apprenticeships, job training, and leadership development.  But, what if job seekers do not possess the skills for tech jobs?

This article describes what lengths some employers are going to get people in their talent pipeline.  In many ways, there is nothing new here.  It comes down to searching for talent where they previously hadn’t and providing training rather than expecting people to come with skills.  It’s the latter that I find most interesting.

When designing selection programs, particularly for entry level positions, we tend to focus on what knowledge or skills the candidates needs on the first day.  Those expectations are higher if we expect someone to come with experience than if we are going to be providing a lot of training.  This has important impacts on how we select candidates, including:

  1. Use of aptitude tests rather than knowledge tests.  Aptitude tests are terrific measures of basic skills and are quite valid.  However, speeded ones can lead to adverse impact, so they require good validation studies, meaningful passing scores, and adverse impact analyses.
  2. Alter interview questions so that a wide variety of experiences can be used to answer them.  If you are hiring people who don’t have experiences in your industry, you should be asking valid questions that people with little or no job experience can answer.  For instance, instead of, “Tell me about a time when you led a team project at work and…” use “Tell me about a time when you had to influence a group of friends and…”
  3. Focus on reducing turnover.  Training is EXPENSIVE, so hiring mistakes in a boot camp environment are very costly.  Take special care in developing realistic job previews and other ways that allow candidates to decide if they are not a good fit.  Collect information (previous experiences, referral sources, school majors, etc.) that may be indicative of future turnover and validate them.  These can be part of very useful pre-employment processes.

What this approach really presents is a change in HR strategy from one that relies on people to be able to start on day one to taking time to get them up to speed.  By having recruitment, selection, and development leaders involved in the execution, organizations can adapt their tactics for identifying and selecting talent and have a smoother transition.

How Often Should You Use Your Gut Instinct? How About Never?

Why do tests predict job performance better than interviews?  Because interviewers let their “gut instinct” cloud their judgment and introduce lots of related bias. 

This recent article suggests (without any data to back it up) that sometimes we should just trust our gut because it is better at predicting the future than our analytical mind, which is better at predicting the past.  Huh?  Our instant reactions to something make us psychic?

In Nobel Prize winner Daniel Kahneman’s book Thinking, Fast and Slow he summarizes decades of research on decision making.  He describes our fast, “gut instinct” thinking as System 1.  Let’s talk about a few of the reasons why this kind of decision making leads us to poorer decisions:

  1. System 1 thinking is highly influence by irrelevant numbers.  For instance, valuing something at a higher price if the first cost is presented at $50,000 than if the first cost is presented at $25,000.

  2. This level of thinking leads us to make judgments based on how easily we can think of examples.  When we can think of those instances, we give them higher probabilities of occurring.

  3. Our gut is overconfident—it assumes we have more control than we do.  Kahneman explains that System 1 decision making involves only our own experiences, which are a small and does not account for randomness.  Despite the article above saying that our gut instincts are forward thinking, it is just the opposite.  System 1 thinking assumes that what I experienced before is a far greater predictor of the future than it is.

If your instinct tells you that an upcoming decision is wrong, don’t just trust it.  Do some research and/or talk to others and see if you are falling into a System 1 pitfall. 

We rarely have 100% of the data we want before making business decisions.  But, throwing away what we have because going in another direction “feels” better is not a recipe for success. 

Let’s put this in a selection context.  Our gut tells us that people who are similar to ourselves in background and experience are the best hires.  Slower thinking tells us to look at other factors, such as skills and abilities before making such decisions.  And when we do so, we make better hiring choices.

Going with your gut instinct It may sound sexy and empowering, but it is not effective.  Our slower System 2 (per Kahneman) processing system, despite its own set of biases, is more likely to lead us in the right direction.

Are we Biased AGAINST Top Talent?

We all want to believe that we are looking to recruit, select, and develop top talent.  We spend lots of time reading and writing articles on the topic.  But, what if hiring managers are not interested?

This article throws a bit of cold water on the topic.  It documents a study where hiring managers were shown to doubt the organizational commitment of those deemed the most capable.  It was almost as if they were saying, “Why would someone really good want to work for us?”

There are several issues at work here.  But, what they boil down to is a bias among hiring mangers that negatively affects their selection processes.  Sure, I can imagine anecdotal evidence (“Yeah, we hired that one really bright person, but she jumped ship as soon as she got a better offer.”), but I don’t think that this is a data driven decision.

What this also underlines is the importance of developing a culture that encourages top talent to stay.  There’s no question that selecting the right people will drive business performance.  And having a culture that acknowledges and rewards high performance will do so as well.  When hiring managers feel that top talent will not stay, it is really an indictment of the culture rather than an accurate prediction of management’s view.  How can you fight this?

  1. If managers do not think top talent would be committed to your organization, they should NOT be involved in hiring. 
  2. Those who are doing the hiring should be able to provide a realistic preview of the organization, but should also be able to succinctly describe why people stay.  And I’m not just talking about a good cafeteria.  They should be able to provide examples of people who have found challenging work over time in the organization.
  3. If you are speaking with hiring managers who show an anti-talent bias, ask them what needs to be changed so they would believe that top talent would want to stay.
  4. The best way to fight bias is with data.  You should be able to study turnover rates by talent bands (contact me for tips on this).  This way you can either show people that top talent does not leave any faster than other employee groups or demonstrate to executives that this is a problem that needs to be addressed.

Organizations should strive for selection processes that identify top talent and cultures that nurture them.  Do not let bias against hiring top talent work against these two initiatives.

Shaping Skills to Your Work

It is important to use valid selection tools to hire people for the work you have for them.  But, what happens when technology changes the tasks or the jobs get replaced by automation?  You can let people go as their work becomes obsolete and hire new staff.  However, in times with low unemployment, this strategy will be difficult to execute.  Or, you can train people to acquire the new skills.  This has big implications in industries where tech is changing the nature of work, such as mining and warehousing, just to name two.  However, trying to train lots of people in new skills assumes that they have the interest and aptitude for learning them.  Remember, people chose to pursue their given job/career for a reason.

Amazon and Walmart provide an example of this type of investment.  Their programs include technical and college training.  What is telling about their plans is that neither company considers it a “nice to have.”  Rather, it is an acknowledgement that the skills it takes to run their businesses are changing and they don’t think they can find enough talent to meet future needs in the labor pool.  This may be because so many young people want a career that requires as little work as possible.

The situation also makes one think about selecting people for industries where the skills required change rapidly.  Instead of using tests or interviews that focus on specific abilities, perhaps addressing broader ones, such as openness to new experiences and general aptitude, will serve companies better.

Drilling for Good Candidates

The current low unemployment rates and data mining have led to companies tossing out wider and wider nets to fill positions.  But, is all of this confusing activity with productivity?

This article (thanks to Dennis Adsit for bringing it to my attention) brings up some great reminders about some very solid things that employers should be doing (valid testing, structured interviews, etc.) and avoiding others (tech is NOT a magic bullet for recruitment and selection).  However, it also does a good job of challenging some basic assumptions about hiring, all of which can be evaluated.  These include:

  • Unless you are adding positions, why are you looking for so many outside candidates? One reason people leave companies is because they do not feel they have promotional opportunities.  One reason you are looking for so many outside candidates is that people quit.  Chicken, meet egg.
  • Taken a step further, HR really needs to test the effectiveness of its processes on an ongoing basis. If there is data to support that, in general, outside candidates perform better than those promoted, then keep on searching for them.  And you should probably revamp your entry level recruitment and hiring processes.  If not, then career development and taking steps to increase internal mobility will be more effective actions than scouring the universe of passive candidates for new hires.
  • Develop measures to evaluate the success of what you are doing. Few things frustrate me more than a client saying, “We cannot measure someone’s individual performance.”  Really? Does that mean the cost of turnover is the only reason you keep people in their jobs? Granted, it can take some time to measure output, but you can typically find ways of evaluating a person’s contribution to a team.  If a manager says, “I like/don’t think this person is effective” she should be able to say why.
  • Related to the above, don’t assume that a good process will always have the same effectiveness. As your business changes, recruiting and selection systems need to adapt as well.

I do not think that HR has to constantly be reinvented.  But, basic assumptions should occasionally be challenged.  It is only by measuring and evaluating our processes that we can truly improve them.

Thanks for coming by!

Please provide this information so we can stay in touch.

CLOSE