Expediency vs. Effectiveness

I’ve blogged several times (here, here and here) about the City of LA’s firefighter selection process. More specifically, how factors besides the validity of the test, interviews, etc. are being used to cull applicants. Since my last post on the subject, RAND has completed a study of the city’s firefighter selection process. Full disclosure: my wife works at RAND, but she was not involved in this study.

The paper is a good read and provides a solid overview on conducting validation studies. As the title suggests, their task was to suggest to the city how to improve recruiting and hiring of firefighters. The study outlines how the city currently attracts applicants and screens them. The authors then provide recommendations on making these processes better in terms of streamlining and making them more valid.

What is clear throughout the study is that the city’s biggest issue in managing this process is the sheer number of applications they get. Several selection decisions are made based on reducing the number of people in the process. This was the driver behind the city stating that applications would be evaluated on a first come, first served basis, which lead to the application cutoff period being one minute after submission.

The city is between a rock and hard place when it comes to narrowing the applicant pool early in the process. The most pressing is that they do not have the budget to process as many applications as they receive. One would think that a solution to this would be to raise the passing score on the tests. However, based on the data presented, raising the passing scores on the written test will lead to adverse impact against African-Americans and Hispanics and doing so on the physical abilities test would negatively affect women. Interestingly, the city’s “first come, first served” policy led to even more adverse impact against racial minorities and women. Some will say this is because the policy was not well publicized outside of the fire department so this gave an advantage to friends and family members of existing firefighters (note that data shows that a very high percentage of new hires in the department are family members of current firefighters who tend to be white males). To its credit, the interview process, which can often lead to adverse impact, has been shown to be fair to racial minorities and provides an advantage to women over men.

 

I was pleased to see that RAND’s suggestions to reduce adverse impact were not to make the test(s) easier to pass, but to target recruiting efforts on minorities and women that would increase their passing rates. Specifically, the report suggested reaching out to female athletes (more likely to pass the physical abilities test) and minority valedictorians (more likely to pass the written test). The former is a solid idea. However, I’m thinking that school valedictorians (and their parents) are normally looking for a career path that includes a 4-year college and a job in a knowledge industry, but you never know.

Most interesting in the report is the city’s focus on managing the numbers rather than the quality of the process. The city insisted that RAND analyze the impact of randomly choosing people to continue in the process when the number of applications gets too large. This is a solution which does nothing to improve the quality of firefighters hired and is as likely to make the adverse impact worse as better. The study suggest using random sampling by specific groups (stratified), but that does not change the fact that people with lower tests scores are going to be chosen over those with higher ones. Not exactly a recipe for staying out of court.

I do not understand why the city sets its hiring schedule in such a boom or bust fashion. Test results are good for a year, so why not accept applications at several times during the year? RAND also makes other suggestions for managing the number of applications by putting more of the background screening at the front end (and online). Yes, all of this costs money, but so does scrapping a system, creating a new one, and hiring RAND to make recommendations. The cities focus should be on investing in a firefighter selection system that delivers the best available firefighters to the city while minimizing adverse impact and not making short-term decisions based on cost.

For more information about validated pre-employment test practices and services, please contact Warren Bobrow at 310 670-4175.

Creating Smarter Teams

The science behind selecting people who are good at their jobs is well established. We know, based on large research studies, that there are certain tests and assessments that predict different abilities. Those who possess abilities that are required for a specific job do that job better than those who do not have them.

However, there is a lot of work that takes place in groups where individual performance is difficult to measure and may not be relevant. Once we look at groups, we are not as interested in how well a person does as we are how well the team is performing. It makes sense to ask what are the personal attributes of the members that make for these high performing teams?

This article delves into the question of what makes a smart team. They demonstrate that teams have “intelligence” (as defined by consistent scores on a variety of group problem solving tasks). The authors cite research which shows the following to be predictive of intelligent teams:

  • The less variance in the contributions (conversational turn taking) by team members, the more productive the group. Put another way, if people are participating in roughly the same proportion, the team is more likely to be smarter.
  • The better the individuals in the group are, on average, in attending to the social needs of others, the more intelligent the group. This is particularly interesting because this factor was demonstrated in groups that meet face-to-face and those that meet virtually.
  • The more women in the group, the smarter it was. However, the authors caution that a big part of this is that women are better in attending to the social needs of others. Based on the data, they could have just as easily included Openness to Experience instead of gender to get the same results.

I have some research methods concerns about the studies besides the one above, but I think the main points are interesting. Of most interest to me is that we can quantify what makes for a smart team and the behaviors of those individuals who comprise them. This should allows us to design and use valid pre-employment tests to select people into the teams to make them more intelligent.

Where the research comes up short is in testing whether the smarter teams are better performing teams in real life situations. This would be difficult, but not impossible, to demonstrate for a variety of reasons. However, doing so could unlock a lot of productivity in companies.

For more information on using valid pre-employment testing to create smart teams, contact Warren Bobrow.

Does There Have to be One Boss?

It always sounds great to say, “Y’know, we are all on the same team. We don’t need a hierarchy. We can make decisions together.” But, for how long does that really work?

This article tells an interesting tale about a start-up and how the two founders tried to run things together, but eventually learned that they needed, “one strategy, one vision, and one decision maker.” Other cases are presented, but they always circle back to the idea of an agreement on vision and establishing accountability. And, we know that having co-CEOs doesn’t always work for big companies, either.

Having shared responsibilities is a good thing. Small businesses have always had partners, which can essentially be co-CEOs, but many have operating or managing partners as well, which indicates where the buck stops. Additionally, dividing up executive tasks can also lead to a better work-life balance for those in charge as one person does not have to do EVERYTHING.

There’s been some research on using co-CEOs. Interestingly, it shows that a clear, but not too large, power difference between the two leads to better organizational performance. This implies that there needs to be a clear number 1 (or 1a), but not so much so that one CEO feels that s/he is being pushed aside.

There are important implications of these ideas for teams that are not in the C-Suite as well. One is that having a leader, or, if you’d prefer, someone to set structure and hold people accountable, is critical for a group’s success. The other is that people on a team either want to know who the final decision maker is or will ask whichever co-leader they think they’ll get the “right” answer from when they have a question. For the purposes of “one strategy, one vision, and one decision maker,” the former is preferable.

For more information on organizational structure, contact me at [email protected].

Make Fish Want to Come to Your Pond

One of the tricky things about being in the pre-employment testing business is dealing with adverse impact. It is true that some demographic groups tend to score better on some tests than others, which can lead to problems. Some companies decide that the best solution is either lowering the passing score on the test or getting rid of the assessment all together. In both instances, they are shooting themselves in the foot as they are allowing a solvable issue to reduce the economic effectiveness of their selection process. A better approach is to draw better talent to your organization.

This article talks about how one high tech company approached the diversity issue. Rather than using quotas or dumbing down their hiring process, it made itself more attractive to female applicants so it had a more talented pool of candidates from which to draw.

One thing that helped their approach was that they had a very specific focus (females, particularly software engineers). There wasn’t a reference to a “right” number of hires, which I think is good. This means that they are concentrating on getting diverse talent and not jumping through a numeric hoop.

To recruit a diverse and talented pool of candidates, you also need to know where your process is excluding people disproportionately. That will tell you whether is a recruiting issues (we’re just not getting enough talented people in the pipeline) or potentially a bias issue (why do our interview outcomes look this way?).

For more information on pre-employment testing and recruiting practices, contact me at [email protected].

Make Fish Want to Come to Your Pond

One of the tricky things about being in the pre-employment testing business is dealing with adverse impact. It is true that some demographic groups tend to score better on some tests than others, which can lead to problems. Some companies decide that the best solution is either lowering the passing score on the test or getting rid of the assessment all together. In both instances, they are shooting themselves in the foot as they are allowing a solvable issue to reduce the economic effectiveness of their selection process. A better approach is to draw better talent to your organization.

This article talks about how one high tech company approached the diversity issue. Rather than using quotas or dumbing down their hiring process, it made itself more attractive to female applicants so it had a more talented pool of candidates from which to draw.

One thing that helped their approach was that they had a very specific focus (females, particularly software engineers). There wasn’t a reference to a “right” number of hires, which I think is good. This means that they are concentrating on getting diverse talent and not jumping through a numeric hoop.

To recruit a diverse and talented pool of candidates, you also need to know where your process is excluding people disproportionately. That will tell you whether is a recruiting issues (we’re just not getting enough talented people in the pipeline) or potentially a bias issue (why do our interview outcomes look this way?).

For more information on pre-employment testing and recruiting practices, contact me at [email protected].

Thanks for coming by!

Please provide this information so we can stay in touch.

CLOSE