Criminal Investigative Analysis
Measuring Success (Part Three of Four)
By J. Amber Scherer, M.A., and John P. Jarvis, Ph.D.
The previous edition of the FBI Law Enforcement Bulletin featured the second of a four-part series on criminal investigative analysis (CIA). Part two, “Criminal Investigative Analysis: Skills, Expertise, and Training,” described the history of differing approaches to CIA processes and identified the necessary skills and training essential to conducting this important police work.
Part three focuses on another debate that sometimes emerges with not only CIA but with any crime analysis tool or technique—that is, the difficulties associated with measuring accuracy and defining success in the outcome of such efforts. From a profiling perspective, accuracy is not measured simply by how many characteristics are correct on an offender profile; perhaps, it should be gauged by how valuable this kind of analysis is to a law enforcement agency in obtaining useful investigative assistance. As such, at least one dimension of measuring success is, What does “useful” mean to an agency requesting this type of case analysis?
ACCURACY OF PROFILING
Some research has examined the accuracy and value of offender profiling.1 However, because accuracy relates to measuring the effectiveness of CIA, this task has proven elusive to demonstrate. In a particularly notable early study (the first), during the rise in popularity of profiling, researchers compared professional profilers’ expertise with that of experienced detectives, clinical psychologists, and college students in generating written profiles while using a mock-profiling task and found marginal support for the theory that profilers were more accurate than experienced detectives.2 While these results were not overwhelming, as with any analytical process the true determinant of success may not be with any single task, but with the overall success of the service in meeting consumer needs.
Some practitioners insist that despite clear limitations, services provided within CIA continue to yield effective results.3 They have concluded that “Success is the reason our services continue to be requested by law enforcement agencies.”4
Yet, empirical evidence demonstrating the consistent success of these services has been scant. This likely is due to little consensus as to what constitutes success. Some previous researchers have equated success with the accuracy of a profile.5 Other experts suggest measures of usefulness may be more appropriate for assessing success.6 For example, some common assertions made concerning CIA are that “It helps narrow down and prioritize a suspect list,” “There are aspects that led me to this offender,” and “It gives definitive lines of inquiry.”7 These qualitative claims often describe how useful the service was to the investigation. However, demonstrable empirical support for these contentions remains unclear.
One notable expert examined solved cases and concluded that profiling assistance was helpful in some way 83 percent of the time.8 More recently, yet still more than a decade ago, other researchers found that the outcome most often emanating from this kind of assistance was the interrogation of a suspect (62.2 percent), followed by investigative suggestions (58 percent).9 Yet, these researchers also reported that in 24.5 percent of the cases, profiles actually hindered the identification of a suspect.10 However, some experts have stated that successful consultative requests for these services often are determined by the usefulness reported by given agencies.11 Similarly, still another researcher described a thorough picture of behavioral advice provided to law enforcement agencies in the United Kingdom.12 Overall, these results showed that of the 186 detectives surveyed, 14.1 percent reported that the advice assisted in solving the case, while 82.6 percent reported that the advice they received was useful.13
Dr. Jarvis serves in the FBI’s Behavioral Research and Instruction Unit and chairs the Futures Working Group.
To get a more accurate picture of how beneficial profiling services are to law enforcement agencies, researchers in the Netherlands believed the best methodology would be to interview the police officers who actually use the services.14 In this study, six officers were interviewed regarding their opinions surrounding an offender profile they had received on a case. The researchers reported the broad conclusion that the majority of officers interviewed were satisfied with the services. However, it became apparent that it was not simply profiling advice that satisfied them. For example, without being prompted, officers expressed how much they learned by discussing the case with the profiler, describing how many new ideas they received and how useful they would be for future investigations.15 These results from nearly 20 years ago revealed two important points: 1) how CIA services have evolved beyond the mere profiling of unknown offenders and 2) how complex debates of these services’ success and their measurement have become.
Illustrating this latter point, subsequent researchers followed the same interview methodology to examine success of CIA processes and found mixed results.16 In this study, 11 police officers who had used profiling services in the United Kingdom were interviewed and questioned solely about their satisfaction with profiling services provided. The results showed overall dissatisfaction despite some positive observations about profiling and profilers. At best, they found the services helpful for overall decision making.17 Of course, these results are hampered because of the small sample of officers surveyed. Additionally, the results could be due to the fact that some of the officers focused solely on the profiling service of CIA, rather than any of the other investigative and prosecutorial services available. In hindsight, had the researchers asked the police officers if they had used any criminal investigative services other than profiling, the results may have differed markedly.
In keeping with many of these previous efforts, the present study sought to interview actual practitioners of CIA about their experience with providing profiling services. These subjects are thought to be particularly useful in gaining perspectives on what other analytical services have been provided in the context of CIA, as well as information on the success and limits of such services. As its overall goal, the study examined all the dimensions of CIA, but this current work reports only those findings relative to examining services delivered and measuring the success of outcomes.
The respondents in this study provided their perspectives regarding the accuracy of CIA as an investigative tool and gave their insights on what defines a successful consultation, what makes CIA effective, and what the strengths and weaknesses are of this specialized analysis during an investigation.18 The results are excerpted from a larger study that examined the background, training, skills, processes used, effectiveness, and use of CIA services in court.19 The subject pool included 40 practitioners of these services, and the demographic composition of the sample drawn is reported in earlier work.20 The overall objective was to gain knowledge and insight from those individuals professionally trained and experienced in delivering CIA. By doing so, it was hoped to reveal commonalities or differences in the respondents’ perspectives regarding the usefulness, strengths, weaknesses, and applications of successful services.
Utility vs. Success
Some researchers have contended that CIA has limited utility.21 When asked to respond to this assertion, while no exact percentage was recorded, an overwhelming majority of those interviewed refuted this contention by claiming that CIA does have its place in assisting law enforcement investigations. When asked to indicate what makes CIA effective, of all of the reasons given, 37.5 percent reported that the analyst’s level of experience was the best predictor of effectiveness. This was followed by providing investigative direction (32.5 percent), following an appropriate process (27.5 percent), being able to consult with others (17.5 percent), using research (17.5 percent), and having reliable data (12.5 percent) as the bases for the effectiveness of CIA services.
Respondents also identified specific benefits to using CIA. One participant’s answer serves as an example.
I think the important thing here is to understand that it is a tool...used to help the investigator more narrowly focus [an] investigation. I’ll tell you right now, the biggest critics of profiling are the guys who developed it. The biggest critics? We were. It was never intended to solve crime. It was intended to be an investigative tool for the investigator to use in [the] attempt to solve crime.
In a related question, the strengths of CIA were reported to include: offering a different perspective (70 percent), providing a methodology to follow (37.5 percent), opening new avenues for investigation (30 percent), and narrowing down the suspect pool (22.5 percent). Weaknesses of CIA also were noted. Sixty-five percent identified an over reliance on the analysis by clients or investigators, and 32.5 percent of the time unreliable data was reported as a significant potential weakness in CIA services. Other, less frequently noted weaknesses included tunnel vision on the part of the practitioner and the chance that the analysis could ultimately be wrong due to such weaknesses prevailing. Interestingly, respondents typically cited both strengths and weaknesses. While queried about both, few, if any, reported only a strength or only a weakness.
A question present since offender profiling began is, What constitutes a successful investigative consultation? In contrast to the effectiveness of the process in yielding results, respondents delineated what makes a consultation successful.
Variables Relating to Definitions of a Successful Consultation
|Successful Consultation Variable||Number of Respondents||Valid Percent|
|Move Case Forward||27||67.5|
|Give Investigators New Ideas||21||52.5|
|Provide New Investigative Avenues||20||50.0|
|Help Catch Offender||16||40.0|
|Offer New Leads||7||17.5|
|Prevent Wrongful Conviction||5||12.5|
Note: Percentages do not total 100 percent due to multiple responses.
It is important to note that these results (based on only 16 valid responses) do not indicate that success is solely about catching an offender. Clearly, CIA provides other avenues for success. Observations offered included moving the case forward in any way and helping investigators better understand the case; just having investigators listen with an open mind and educating them regarding certain aspects of the case was deemed a success to some practitioners. Others viewed success more as a savings of time and resources that the services can provide to law enforcement agencies.
Overall, success was not viewed as a simple measure nor commonly judged to be just catching an offender with an accurate profile. This is in stark contrast to many misunderstandings by some people in the general public, academia, and law enforcement agencies who may be swayed by media portrayals, rather than the actual practice of assisting law enforcement in these matters.
Remembering that CIA is an analytical tool to guide investigators and not an end in itself, one respondent’s quote illustrates how CIA is meant to assist law enforcement in advancing an investigation and also encapsulates notions of measuring success in these efforts.
There are different views regarding a successful consultation [emphasis added]. I think, for me, a consultation is a success if I feel that whoever I am assisting, consulting with, walks out of the process with a better understanding of the case, of the theories, of potential suspects, and why. If they have a better understanding of those things, I look at it, to be quite honest, I look at it as much as an educational sort of endeavor as anything else. I am a firm believer in training and imparting knowledge. It’s a train-the-trainers sort of thing. You want them to become good at what they do…. So, if they can walk out with a better understanding of that, that is a success to me. I don’t think you ever gauge success in this sort of field on arrests or convictions or stuff like that because that is not what you do, and most of the time the information you are giving them is not giving them their offender. It’s helping them in other ways, and I think it is more important.
Those who provide CIA services have reported that the process often can be misinterpreted, which creates problems for defining and measuring success. The results of this study reveal agreement with findings that suggest measures of usefulness are more appropriate for gauging success than the number of accurate characteristics on a particular offender profile.22 Comparing written profiles to show expertise or accuracy does not adequately measure the benefit or success of CIA.23 Thus, this research shows that previous academic research has used the wrong metrics to determine accuracy; therefore, prevailing academic notions about success or failure of CIA are likely misleading.24 Perhaps, metrics based upon consumer satisfaction models would be more appropriate measures for this purpose.
Debates regarding the utility of CIA are rooted in how it has been defined and measured. Further, determining the usefulness of this practice by monolithically measuring the accuracy of a constructed offender profile is invalid and only serves to exacerbate further debate. Attention should be given to what behavioral analysis can be appropriately expected to deliver and what is appropriate to accurately assess the overall utility of the services. Consumer satisfaction surveys frequently are used in marketing research to establish whether services meet expectations. With a form of a consumer satisfaction survey upon the completion of requested assistance, the practice of CIA then could likely be evaluated more accurately by both law enforcement agencies and academic scholars. While this may pose both practical and methodological challenges as well, the resulting metrics may be more meaningful than previous efforts to measure success.
Next month’s edition of the FBI Law Enforcement Bulletin will feature part four, the final article in this series. It will address the reported role and value of CIA services to prosecutors and the courts.
The authors wish to thank Dr. Gabrielle Salfati of John Jay College of Criminal Justice in New York, New York, for substantial assistance in formulating the research questions, constructing the protocols employed, and providing other general guidance in earlier stages of this research effort.
Investigators who believe their case can benefit from criminal investigative analysis services and would like to request such services should contact their local FBI field office. The field office representative will put investigators in contact with the National Center for the Analysis of Violent Crime (NCAVC) coordinator, who will assess the case and contact the NCAVC office in Quantico, VA.
“…the true determinant of success may not be with any single task, but with the overall success of the service in meeting consumer needs.”