I’ve blogged several times (here, here and here) about the City of LA’s firefighter selection process. More specifically, how factors besides the validity of the test, interviews, etc. are being used to cull applicants. Since my last post on the subject, RAND has completed a study of the city’s firefighter selection process. Full disclosure: my wife works at RAND, but she was not involved in this study.
The paper is a good read and provides a solid overview on conducting validation studies. As the title suggests, their task was to suggest to the city how to improve recruiting and hiring of firefighters. The study outlines how the city currently attracts applicants and screens them. The authors then provide recommendations on making these processes better in terms of streamlining and making them more valid.
What is clear throughout the study is that the city’s biggest issue in managing this process is the sheer number of applications they get. Several selection decisions are made based on reducing the number of people in the process. This was the driver behind the city stating that applications would be evaluated on a first come, first served basis, which lead to the application cutoff period being one minute after submission.
The city is between a rock and hard place when it comes to narrowing the applicant pool early in the process. The most pressing is that they do not have the budget to process as many applications as they receive. One would think that a solution to this would be to raise the passing score on the tests. However, based on the data presented, raising the passing scores on the written test will lead to adverse impact against African-Americans and Hispanics and doing so on the physical abilities test would negatively affect women. Interestingly, the city’s “first come, first served” policy led to even more adverse impact against racial minorities and women. Some will say this is because the policy was not well publicized outside of the fire department so this gave an advantage to friends and family members of existing firefighters (note that data shows that a very high percentage of new hires in the department are family members of current firefighters who tend to be white males). To its credit, the interview process, which can often lead to adverse impact, has been shown to be fair to racial minorities and provides an advantage to women over men.
I was pleased to see that RAND’s suggestions to reduce adverse impact were not to make the test(s) easier to pass, but to target recruiting efforts on minorities and women that would increase their passing rates. Specifically, the report suggested reaching out to female athletes (more likely to pass the physical abilities test) and minority valedictorians (more likely to pass the written test). The former is a solid idea. However, I’m thinking that school valedictorians (and their parents) are normally looking for a career path that includes a 4-year college and a job in a knowledge industry, but you never know.
Most interesting in the report is the city’s focus on managing the numbers rather than the quality of the process. The city insisted that RAND analyze the impact of randomly choosing people to continue in the process when the number of applications gets too large. This is a solution which does nothing to improve the quality of firefighters hired and is as likely to make the adverse impact worse as better. The study suggest using random sampling by specific groups (stratified), but that does not change the fact that people with lower tests scores are going to be chosen over those with higher ones. Not exactly a recipe for staying out of court.
I do not understand why the city sets its hiring schedule in such a boom or bust fashion. Test results are good for a year, so why not accept applications at several times during the year? RAND also makes other suggestions for managing the number of applications by putting more of the background screening at the front end (and online). Yes, all of this costs money, but so does scrapping a system, creating a new one, and hiring RAND to make recommendations. The cities focus should be on investing in a firefighter selection system that delivers the best available firefighters to the city while minimizing adverse impact and not making short-term decisions based on cost.
For more information about validated pre-employment test practices and services, please contact Warren Bobrow at 310 670-4175.