Tightening the OODA Loop: Police Militarization, Race, and Algorithmic Surveillance

Jeffrey L. Vagle October, 2016

This Article examines how military automated surveillance and intelligence systems and techniques, when used by civilian police departments to enhance predictive policing programs, have reinforced racial bias in policing. I will focus on two facets of this problem. First, I investigate the role played by advanced military technologies and methods within civilian police departments. These approaches have enabled a new focus on deterrence and crime prevention by creating a system of structural surveillance where decision support relies increasingly upon algorithms and automated data analysis tools and automates de facto penalization and containment based on race. Second, I will explore these militarized systems, and their effects, from an outside-in perspective, paying particular attention to the racial, societal, economic, and geographic factors that play into the public perception of these new policing regimes. I will conclude by proposing potential solutions to this problem that incorporate tests for racial bias to create an alternative system that follows a true community policing model.

Introduction

As militaries transition from a war footing to a postwar posture, they inevitably shed excess equipment and technology that is outdated, no longer needed, or too expensive to maintain. In the drawdown from the wars in Iraq and Afghanistan, the U.S. military found willing recipients of this material in local and state police departments. In addition to these technology transfers, local police departments have increasingly adopted military tactics, techniques, and procedures (TTPs)—originally conceived and designed for military units in combat operations—for use in their day- to-day policing. This combination has led to the overall militarization of civilian police forces in the United States, putting wartime tools in the hands of peace officers.

This trend toward police militarization has found enthusiastic support from departments adhering to the “broken windows” theory of policing, especially with respect to technologies and TTPs meant for intelligence analysis or surveillance purposes. The locus of this relationship can be found in the information-centric approaches found in broken windows policing and other zero-tolerance, quality-of-life police programs. Since military intelligence has long prized information-centric methods as critical to the goal of actionable intelligence, militarized systems and approaches appear to be a perfect fit to modern policing. This phenomenon may be seen as the natural result of the industrial and post-industrial society’s desire to maximize control and efficiency—across all spheres of life— through careful observation and data analysis. While advances in policing techniques have garnered many societal benefits, they have also established a system of structural surveillance that has entered a renaissance with the help of military technologies. Automated surveillance analysis systems, developed in the wake of 9/11, have given police departments a powerful toolkit to advance algorithmic policing strategies.

But these algorithmic approaches too often target poor and minority communities, inserting a de facto racial component into the system, even when the automated intelligence systems are fed “objective” crime data. For example, in a recent investigation of software used by state and local criminal justice authorities to predict the risk of recidivism of those booked into city and county jails, researchers found that these algorithms were not only wildly inaccurate in their assessments, but were also highly likely to falsely flag Black defendants as future criminals.¹ Algorithmic “scoring” mechanisms like this are becoming increasingly common tools for criminal courts and law enforcement, which are seeking an information-based advantage in the control of criminal activity. These programs have become so popular, in fact, that Congress is considering mandating their use in federal prisons.²

This Article examines how military automated surveillance and intelligence systems, when used by civilian police departments to enhance predictive policing programs, have reinforced racial bias in policing. I will focus on two facets of this problem. First, my research studies the role played by advanced military technologies and methods within civilian police departments. These methods have enabled a new focus on deterrence and crime prevention by creating a system of structural surveillance where decision support relies increasingly upon algorithms and automated data analysis tools, and which automates de facto penalization and containment based on race. Second, I will explore these systems, and their effects, from an outside-in perspective, paying particular attention to racial, societal, economic, and geographic factors that play into the public perception of these policing regimes. I will conclude by proposing potential solutions to this problem that incorporate tests for racial bias to create an alternative system that follows a true community policing model.

I. The Militarizationof Police Intelligence Operations

The militarization of civilian law enforcement agencies (LEAs) has long been viewed as anathema to the founding principles of the United States and corrosive to civil liberties in a constitutional democracy generally.³ Paradoxically, contemporary American society has increasingly taken a distinctly militaristic approach to solving its (non-military) political, social, and economic issues, applying war metaphors to programs and policies to emphasize the seriousness of the problem and the approach to it, for example, the “war on drugs” or the “war on poverty.”⁴ These mixed signals have been the backdrop to a steady increase in the militarization of U.S. civilian police forces post-World War II, with racial tensions, “broken windows policing,” the “war on drugs,” and the “global war on terror” acting as the primary catalysts of this phenomenon.⁵ The most visible aspect of police militarization can be seen in the increased deployment within LEAs of weapons, equipment, and training designed for use in combat by militaries.⁶ While these manifestations might be the most outwardly obvious signs of this militarization trend, there is a more basic facet of it that LEAs have almost universally adopted: intelligence operations.⁷

A. Military Intelligence and the Development of the OODA Loop The importance of intelligence operations to the military is a long

accepted principle, since as often quite large and widely distributed organizations, militaries are expected to think and act as if they were a unitary being.⁸ The pace, environment, and sheer horror of combat combine to create a state of near chaos — “the realm of uncertainty” —through which militaries are forced to navigate.⁹ To mitigate at least some of the disorienting effects of warfare, modern militaries must organize themselves around rational, bureaucratic principles, with robust networks of communication and information management at their core.¹⁰ Military command structures simply cannot function without the timely communication of information on a wide range of broad and narrow topics including terrain, troop strength and movements, civilian considerations, transportation net-works, availability of supplies, enemy disposition and morale, weather and light conditions, and more—all in support of the theory that the better prepared and informed army has the advantage.¹¹ This general concept is often referred to as military intelligence, or more succinctly, “intelligence.”¹²

The term intelligence is not well defined, however, as it draws from a wide array of broader issues, such as strategy, command and control, and communications.¹³ It is clearly more than an exhaustive cataloging of all available information. Even if this were possible, such a tool would quickly prove useless to militaries as their organizations became mired in irrelevant information, and would be forced to spend valuable time and resources ferreting out the useful bits.¹⁴ Intelligence, therefore, must produce information in a form and quantity that can be used by the organization to make timely decisions regarding plans and operations. This characteristic is often summarized as “actionable” intelligence.¹⁵ The goal of perfectly actionable intelligence is often unattainable, however, and is best thought of in aspirational terms.¹⁶

The industrialization of the 18th, 19th, and 20th centuries yielded paradigmatic advances in the technology and, subsequently, the conduct of warfare, which in turn brought with it the critical need for more rapidly-made decisions based on fresher, more accurate, and more detailed intelligence.¹⁷ The United States military experience in Vietnam illustrated just how crucial intelligence and communications had become in modern warfare, where larger, better-equipped forces were often outmatched by much smaller, yet better informed, groups of guerrillas and regular army soldiers.¹⁸ The post-Vietnam collapse of the U.S. military sent shock waves through the Pentagon, whose leadership began the arduous process of rebuilding a communications and intelligence centric army capable of fighting a fourth generation war.¹⁹

During this period of U.S. military restructuring, a U.S. Air Force combat flight instructor named John Boyd, long known by military strategists for his highly analytical approach to solving military problems, began development of a general theory of military organizational analysis and action. He sought to address the challenge of intelligence and communication in a fast-moving conflict.²⁰ As a veteran of air combat flying the F-86 Sabre in “MiG Alley” during the Korean War, Boyd knew well the challenge for fighter pilots of gathering, processing, and acting on information in a very short amount of time, all while flying an aircraft filled with jet fuel and munitions at hundreds of miles per hour, often while being shot at.²¹ Boyd, a student not only of the great modern military theorists such as Clausewitz and J.F.C. Fuller, but also of philosophers, mathematicians, and physicists, including Kurt Gödel and Werner Heisenberg, actively sought out symmetries and commonalities in his analyses in an attempt to get at the true root of the problem at hand.²² His multidisciplinary approach allowed him to extrapolate common principles from his experiences, and in 1976, he authored the first of five essays on the cognitive, psychological, and temporal processes core to all military intelligence analysis and decision-making processes, from the highest command levels to the lowest.²³

Boyd’s described his groundbreaking theory as a cognitive cycle con-taining four tasks: Observation, Orientation, Decision, and Action (OODA) (see Figure 1).²⁴ These actions worked together to create a feedback loop that gave those who were more adept at accelerating this loop a tactical advantage over their less-nimble adversaries. As applied to Boyd’s initial use of the term in aviation combat, a pilot first considers the relevant information available about the situation (“observation”), such as an adversary’s weaponry, their level of training, and the environmental conditions. The pilot uses these observations to narrow the possible universe of decisions (“orientation”) in the tactical situation. Based on these first two steps, the pilot selects the best solution to their tactical problem (“decision”) and makes the necessary adjustments to realize their decision (“action”).

Figure 1 : The OODA Loop

Through informed use of the Boyd Cycle—more often referred to as the OODA loop—one could not only gain a strategic or tactical advantage over one’s enemy, but in turn, disrupt the enemy’s own OODA loop by denying them the ability to run through its stages due to the speed at which you are able to run through your own.²⁵ The goal for any military organization, according to Boyd, is to increase the speed at which it can navigate this cycle, or to “tighten” the OODA loop.²⁶ Therefore, the army that can affect the tightest OODA loop is able to “get inside” their opponent’s cycle, and thus disrupt their ability to gather, process, and act on intelligence.²⁷

The OODA loop, as Boyd observed, characterized the process all living things go through as part of their everyday survival.²⁸ Learning and adapting are what successful individuals and groups accomplish faster than their less lucky competitors. Boyd’s great contribution was distilling this naturally occurring process into a form, which could be analyzed and implemented by military strategists and tacticians. Boyd continued to develop this concept through the mid-1980s and continued to present his theories to large military and civilian audiences well into his retirement.²⁹ Boyd’s theories enjoyed moderate success among contemporary military leadership at the time, mainly among theorists and scholars. But in the early 2000s, his ideas were rediscovered and found a heightened relevance among a new generation of warriors joining a high-tech military in a post- 9/11 world.³⁰ This renewed popularity soared even further when nonmilitary and quasi-military organizations seeking a competitive advantage over crime through information-centric efficiency—universally accepted as the path to success since the earliest days of industrialization—increasingly turned to military organizational theories and doctrines for inspiration.³¹

B. The Audit Society and the Allure of Information Management

While Boyd’s OODA loop theories were revolutionary in their military context, their foundations began to form nearly two centuries earlier with the emergence of industrialization and the associated growth of information-centric organizational theory.³² The rise of bureaucracy as an efficient means of organizing at scales unnecessary in agrarian society brought with it a strong inclination toward surveillance, information processing, and data-based planning as a means of constant management and improvement, a school of thought both inspired and followed by military organizational theory.³³ Weber’s theory of legitimate order and authority within a bureaucratic structure, with its “dut[ies] without regard to personal con-siderations,” and an “obligation to obedience,” provided the sort of military-based organizational socialization needed—albeit with some allowance of modification for less martial pursuits—for the management of such modern concepts as large-scale factories, prisons, hospitals, and law enforcement.³⁴

Militaries of the 18th and 19th centuries were quick to adopt these organizational innovations, which fit well with their existing hierarchical frameworks and provided the tools to realize a modern form of military intelligence and personnel management, which contemporary organizational models would not support.³⁵ All of these advances were necessary to support and maintain large armies, but military intelligence was perhaps the biggest beneficiary of these innovations. The increasing amounts of information necessary to make military—or, in fact, any state—decisions in a rapidly modernizing world highlighted the importance of bureaucracy as an essential tool in dealing with the modern crisis of control.³⁶ But even with the benefit of a modern bureaucracy, some of which already existed within military organizations, an army’s ability to process intelligence information had been quite limited by existing technologies.³⁷ This limitation of a military organization’s ability to make informed decisions based on “coded” intelligence information was directly dependent upon its ability to communicate, store, and process that information, described in Weber’s concept of rationalization.³⁸ The promise of increasingly detailed intelligence pictures—improved “situational awareness”—generated by a combination of the improved organizational methods of Weberian bureaucracy and the advances in communication, transportation, and information processing technologies of industrialization, began to move military thinking toward an information-centric style of warfare.³⁹

This quest for more information, better data processing tools, and improved communication methods has become paradigmatic of modern (and post-modern) society.⁴⁰ This “control revolution,” as Beniger puts it, has grown to permeate every area of society where efficiency is sought, bringing with it a need to develop metrics for program effectiveness.⁴¹ The concept of civilian police agencies is driven primarily by as the realization of a society’s social control function, and as such, makes them prime candidates for the information-centric style of organization and leadership.⁴²

The quasi-military structure most police departments now adhere to can be found in the organization of Sir Robert Peel’s London Metropolitan Police in 1829, credited as the first modern civilian law enforcement organization.⁴³ The rapid industrialization of Europe and the United States in the first half of the 19th century brought with it a sudden increase in the population, much of it concentrated in urban areas.⁴⁴ This caused no small amount of concern among the upper classes, who feared disease, petty crime, property damage, and political insurrections, just to name a few of the phobias the ruling elite held regarding the “dangerous classes.”⁴⁵ Peel, as Chief Secretary for Ireland, introduced legislation creating paramilitary forces to suppress Catholic and nationalist “disturbances.” Peel applied the methods he had refined in Ireland to London and continued the use of the military organization as a model for civilian law enforcement.⁴⁶ In addition, Peel recognized that former members of the military made excellent candidates for the role of civilian police officer, since they would arrive on the job pre-acclimated to a quasi-military environment, and would have an instinctive preference for hierarchy, discipline, and order.⁴⁷

These somewhat practical decisions cannot, however, be separated entirely from the growth of information-centric organization theory, political economics, and surveillance technologies which were interlaced with much of the social, political, and economic activity of industrialization and modernity. In the period from the mid-18th through the mid- 19th centuries, as industrialization in western nations began to create increasingly complex systems of interdependencies between manufacturing, capital, energy production, labor, and markets, new means of communication and control were required to take full advantage of new economies of scale and realize productivity levels unheard of under earlier forms of management and organization.⁴⁸ As Giddens points out, advances in the information management enabled organizations to form more effective bureaucratic structures and gave these budding bureaucracies more control over the “timing and spacing” of human activities.⁴⁹ These advances, joined with the modern bureaucracy’s growing appetite for information ultimately led to the surveillant assemblage as integrated into our contemporary concept of governance.⁵⁰

In the mid-1990s, a movement among military theorists began to develop around the concept of exploiting a technological and communications advantage to create a new kind of army, where every soldier and piece of material was equipped with sensors that would allow direct communication of information at the lowest organizational levels, but would also pass this information up the chain of command to give military leaders a “God’s eye view” of the battle space, allowing for even tighter OODA loops.⁵¹ This concept, generally known as network-centric warfare, envisioned the sort of information-driven military hitherto impossible, now enabled by smaller, faster, and cheaper technology that was starting to drive American businesses, and especially advances in networking and communications.⁵² Military strategists picked up on the new style of “bottom-up” management enabled through these technological advances, where information, gathered from the very edges of an organization and passed to leadership in near real time, could allow leaders of even the largest businesses to view, analyze, and make decisions about detailed data — actionable intelligence — that Weber, Bentham, Mill, and their contemporaries could only dream of.⁵³ Following this model, an information- and network-centric military could be more agile and aware than its adversaries, thus allowing fewer troops to cover much wider geographic areas, with less equipment, and with dynamic, ad-hoc supply chains that could place material in the right place at the right time, a philosophy at the center of Secretary of Defense Donald Rumsfeld’s wholesale “Force Transformation” program of the early- and mid-2000s.⁵⁴

The Bush administration’s focus on transforming the military through a network-centric shift from platforms to networks, while viewed with skepticism by rank-and-file troops, was enthusiastically accepted by the U.S. military’s civilian and political leadership, and opened up a wide array of new business opportunities for contractors willing to help implement this vision.⁵⁵ Many of these new programs concentrated on the Command, Control, Communications, Computers, Intelligence, Surveillance, and Reconnaissance (C⁵⁶ISR) space, where the true power of distributed information superiority could be fully realized.⁵⁷ Key programs quickly emerged to build survivable, robust communications and sensor networks, information operations (IO) platforms, geospatial analysis tools, computer-assisted targeting platforms, and unmanned intelligence gathering platforms (colloquially known as drones, and later equipped with weapons systems of their own), all of which were designed to support an agile force far superior to the “muscle-bound and clumsy,” “industrial-age dinosaurs” that were the legacy of outmoded Cold War thinking.⁵⁸

As the U.S. military began to withdraw from their engagements in Afghanistan and Iraq, funding streams for contractors who had tooled up in support for network-centric warfare and force transformation also began to evaporate, and businesses looking for alternative markets for their C4ISR platforms found a ready partner in civilian LEAs.⁵⁹ Like the military of the mid-1990s, police departments had also discovered the promise of network-centricity, a concept that fit well within the quasi-military organizational structures found in most police organizations.⁶⁰ Beginning in the mid-1970s, police departments, especially those in larger U.S. cities, began to feel political pressure to address what was widely seen as a crime epidemic and the “downward spiral of urban decay” that accompanied a post-industrial economic slowdown.⁶¹ The standard tactics built around the police patrol car did not seem to be having any real success in reducing the crime rate, and LEAs were looking for alternatives.⁶²

In 1982, The Atlantic Monthly published an article written by two social scientists who, after studying the tactics of police departments, concluded that disorder and crime are inextricably linked, and, therefore by addressing the petty crimes associated with community disorder — such as loitering, vandalism, and public intoxication, “humble” crimes that were generally considered unworthy of police attention — LEAs will, in turn, prevent the more serious crimes from flourishing in those areas.⁶³ The approach became known as “broken windows policing,” named for the tendency for buildings with a broken window to encourage further window breaking and other forms of vandalism, a phenomenon described by Stanford psychologist Philip Zimbardo in his well-known abandoned car experiment.⁶⁴

Police departments in large American cities began to take an active interest in the broken windows theory, and by the late 1980s and early 1990s, police departments in New York City, Chicago, and Los Angeles had all implemented some version of this model.⁶⁵ In New York, then-mayor Rudy Giuliani introduced a version of broken windows known as “zero-tolerance” policing, which placed a greater emphasis on the “quality of life” issues that Wilson and Kelling pointed out in their original work.⁶⁶ As an early adopter of the broken windows model, the New York Police Department (NYPD) quickly discovered that any effective implementation of such a program would require curbing disorder not only on the streets, but also within the police department itself, which had been in a decades-long decline of poor leadership, corruption, and an overall breakdown of discipline.⁶⁷ Addressing these (not unrelated) problems in a city the size of New York, with a sworn police force numbering in the tens of thousands, would require an approach that could go beyond classical organizational techniques.⁶⁸ The data- and network-centric approaches made possible by the rapid technological advances beginning in the late 1980s and early 1990s, and implemented as a successful proof-of-concept by Wal-Mart during this time, began to instill in the NYPD other police departments a growing faith in algorithms and automated decision tools.⁶⁹

C. Drugs, Terrorism, and the Blurring of Military and Civilian Spheres

The confluence of military and police use of data- and network-centric approaches in their hitherto separate spheres can be traced in its earliest forms to the war on drugs.⁷⁰ As drug trafficking began to be seen not only as a law enforcement issue, but also a threat to national security, military and police agencies began to engage as partners in this effort, sometimes through the exchange of ideas, sometimes quite literally, through interagency actions.⁷¹ These activities were complicated by the fact that the 1878 Posse Comitatus Act generally prohibited the use of federal military forces for domestic law enforcement purposes.⁷² Congress amended the Posse Comitatus Act in 1981 to address this legal obstacle, which allowed for military support of civilian LEAs, with the clear legislative intent that this military-law enforcement cooperation be employed in counterdrug operations.⁷³

Both the concept and the realization of the war on drugs have a rather spotty record.⁷⁴ The political rhetoric that tends to accompany calls for such a war creates images of a “dangerous other,” who threatens social stability due to lax morals and inherent inabilities to exercise self-control.⁷⁵ Efforts within the war on drugs are therefore designed to “target” such “deviant” groups within society, and by doing so, promise to reestablish the societal values held by the upstanding citizens the laws are there to protect.⁷⁶ These words describing the dangerous other are carefully selected to mask their veiled intent: to play on white fears of racial minorities.⁷⁷ The disproportionate effects of this war on African-Americans and Hispanics have been well-documented, resulting in broken social structures, overcrowded prisons, and a retrenchment of Jim Crow policies (albeit under different names).⁷⁸

A vivid example of the literal law enforcement-military partnership in the drug war can be found in the 1988 formation of Joint Task Force-6, now known as Joint Task Force-South (JTF-South), which combined combat and reconnaissance forces from the Department of Defense (DoD), intelligence services, and civilian LEAs to patrol sections of the U.S.-Mexican border on drug interdiction missions.⁷⁹ The regular armed border patrols by military personnel abruptly stopped in 1997, after a young U.S. Marine shot and killed an unarmed civilian.⁸⁰ The American public’s appetite for the use of military troops within its borders evaporated after this incident, only to be revived on September 11, 2001.⁸¹

The terrorist attacks of 9/11 opened the floodgates on law enforcement-military cooperation, with many of the political and legal objections to such partnerships disappearing almost overnight. This sudden paradigm shift, brought on by a level of terroristic violence previously unthinkable in the United States, created an environment within which the traditionally separate spheres of military and civilian law enforcement began to significantly blur.⁸² The military contractors, large and small, that tooled up to support the war effort — both the literal combat operations in Afghanistan and Iraq, as well as the larger, more metaphorical sense — began making much of this material available to civilian LEAs, including machine guns, semi-automatic shotguns, night vision equipment, sniper rifles, combat uniforms, grenades, and high-tech surveillance gear.⁸³ Many civilian police departments were especially appreciative of the expansion of two DoD programs designed to equip LEAs with military gear through the transfer or direct purchase of material.⁸⁴ Because of the highly visible nature of military equipment such as armored personnel carriers, flash-bang grenades, and sniper rifles, much of the subsequent attention from those examining the increased militarization of civilian police forces has been focused on these items.⁸⁵ But the use of this sort of military gear by civilian law enforcement is largely limited to special police units and is not typically found on the average patrol officer.⁸⁶ It is the widespread and increased adoption by civilian police agencies of military intelligence technologies, many of which are integrated invisibly into existing police information and decision support tools, which likely has a more dramatic impact across entire police departments. This invisibility, coupled with an unproven — or misplaced — faith in these technologies, has led to a growing system of structural surveillance that has had a disparate racial impact in many cases.

II. The Expansionof Algorithmic Policing Strategies

A. Early Data-Centric Efforts: Compstat and Its Kin When then-mayor Rudy Giuliani first began implementing New

York City’s version of broken windows policing in the early 1990s, he recognized that the disorder to be addressed could be found not only on the city’s streets while also within the ranks of the NYPD.⁸⁷ Significantly changing the direction of an organization the size of the NYPD — as a shift to the broken windows policing model surely required — would be a difficult task in even the most functional of police departments, something New York City had not had for decades.⁸⁸ The organizational management tools necessary for such an endeavor simply did not exist until advances in information technology opened up the possibility of automated, data- and network-centric decision support systems that could take vast amounts of raw data as input, analyze those data, and provide critical insights to its human users, all within the relative blink of an eye.⁸⁹ It was exactly this sort of system that a few forward thinkers within the NYPD proposed as a solution to their burgeoning organizational management problem.

In 1994, NYPD Commissioner William Bratton revealed the Compstat program as a centralized solution to the department’s organizational dilemma.⁹⁰ The goal of the program was to obtain accurate, up-to-date crime statistics at every level within the department, something that had proven impossible up to that point.⁹¹ By requiring patrol officers to keep records of their daily activities, including stops, arrests, and the details of each incident, the NYPD could collect these data on a computer database, which allowed them to generate weekly books of city-wide statistics they could slice and dice as they saw fit: if they wanted statistics on gun crimes specifically, or wished to compare precinct activity, they could do so with relative ease.⁹² With these data and analysis tools, the NYPD could now begin to efficiently address the city’s broken windows trouble spots, and do so by directing the minimum amount of manpower to the right place at the right time—exactly the outcome Giuliani and Bratton sought.⁹³

The NYPD began to see a significant amount of success with the Compstat system, both with their ability to address the “quality of life crimes” highlighted by broken windows policing, as well as in their ability to effectively manage a large and growing police force.⁹⁴ This successful adoption gave Compstat a fair amount of national publicity, and it was touted, alongside broken windows policing, as the new paradigm of crime prevention in the United States.⁹⁵ Soon, other cities began to emulate and adapt Compstat systems within their own police departments.⁹⁶

With the early success of Compstat also came a redoubled faith in the possibilities of automated law enforcement intelligence systems, allowing police departments to do more with less.⁹⁷ As police departments became increasingly convinced that the broken windows model of policing, with its dynamic, problem-oriented approach, would replace the old, static bureaucratic model that police agencies had relied on for generations, they began to accept and explore more deeply the managerial tools and techniques offered by algorithmic, data-driven systems.⁹⁸ This new thinking sparked a flurry of data- and network-centric experiments in LEAs around the world, and revived a global interest in an intelligence-based model of policing that had been deployed by British police departments since the 1980s.⁹⁹

B. The Rise of Intelligence-Led Policing The British model of “intelligence-led policing” was developed in

part as a response to the privatization initiatives in the UK in the 1980s and 1990s, where portions of government services were either taken over by private companies or adopted a private business model within their organizations, effectively becoming quasi-private agencies.¹⁰⁰ The British National Criminal Intelligence Service (NCIS), originally organized to address drug trafficking, and later expanded to include organized crime generally, adopted a business data processing model to develop the National Intelligence Model (NIM), a nationwide system for use by all police agencies across the UK, replacing their existing bureaucratic management processes with an intelligence-led policing model.¹⁰¹ This model had been developed by British authorities to mimic the traditional military intelligence model, where data are collected and analyzed in order to identify patterns and generate actionable intelligence to best prioritize the deployment of patrols based on a set of problem-oriented goals.¹⁰²

It is not difficult to see the allure of such a system. The combination of tightening budgets, increased public concern over crime and disorder, and a rising perception that the world had become a more dangerous place (due in large part to drug trafficking, terrorism, and a general breakdown of civil order) created an environment that made automated solutions to these problems all the more credible. Rapid advances in technology and research into data mining and automated, intelligent decision support systems began to instill in police departments a newfound enthusiasm for technology not generally seen since the days of Sputnik and the space race. Researchers began looking for existing mathematical and physical models, which could provide even faster and more accurate intelligence.¹⁰³ By the late 1990s, police departments began to consider the possibility that intelligence-led policing, coupled with advanced technologies and analytical tools, could move LEAs beyond mere crime fighting and into the realm of crime prevention.¹⁰⁴

C. The Tantalizing Prospect of Predictive Policing

Organizations seeking to tighten their respective OODA loops have followed Boyd’s logic to an inevitable conclusion: Instead of merely seeking further methods to tighten the OODA loop, they find ways to tighten the loop to a point where it has “inverted into itself”—that is, the decision cycle becomes predictive , rather than reactive .¹⁰⁵ The possibility of prevent-ing crime before it actually happens has been the Holy Grail of police departments, especially when local, state, and federal governments were actively looking for ways to cut back on police budgets.¹⁰⁶ The apparent successes of broken windows and zero tolerance policing backed by sophisticated decision support systems like Compstat and intelligence-led policing, gave many in law enforcement the firm belief that these systems and methodologies would eventually yield tools that would collect the massive amounts of data now available from inside and outside of police depart-ments, swiftly store and analyze those data, and produce results “to anticipate, prevent and respond more effectively to future crime.”¹⁰⁷

A common trope among police department is that an experienced and talented officer can apply their knowledge and analytical skills to attain an imperfect version of predictive policing, but that the model does not scale well.¹⁰⁸ The benefit these new analytical tools and methods could bring to these officers could mean the difference between investigating a crime that just occurred versus preventing the crime from happening in the first place.¹⁰⁹ The savings to society in administrative costs, property damage, and human lives alone make this a worthy goal for a data- and network-centric police force.

But for such a police intelligence system to work as advertised, one needs to provide it with as much good data as possible from as broad a sampling as possible so that the pattern recognition models can achieve the nearest thing to a God’s eye view, thus allowing the analyst to find the proverbial needle in the haystack.¹¹⁰ This means police agencies need to turn to nontraditional sources of information, such as social media, as well as developing and refining internal data sources.¹¹¹ The risks inherent in such an approach—especially if implementation or strategy is rushed or otherwise undertaken without a full understanding of the implications— can include not only the more obvious issues of privacy and fairness, but also technical liabilities attributable to cyber security and the long-term effects on due process, all of which present serious ethical questions and responsibilities.¹¹²

III. The Enhancementof Structural Surveillance

and De Facto Race Bias A. Criticisms of the Broken Windows Policing Model

The broken windows policing model, along with the collection of technologies and techniques supporting this approach, have met with a growing body of criticisms, even as police departments continue to adopt and promote these methods.¹¹³ Central to many of these criticisms is the core role that “disorder” plays in the broken windows model, specifically, the subjective definition and measurement of the term, as well as the limited beneficiaries of this approach.¹¹⁴ These criticisms have seriously questioned the premise of broken windows, citing statistics that indicate that the broken windows model has a measurable, direct negative effect on the very neighborhoods and communities whose “hot spots” were supposedly the beneficiaries of increased police focus.¹¹⁵

Among the sharpest critiques of the broken windows policing model is that, whatever the original intent was of such programs, their implementations have been less about policing disorder than about the control of poor neighborhoods and poor people, most of whom are racial minorities.¹¹⁶ The longstanding practice within the American legal system of using race as a signal of increased risk of criminal behavior combined with programs encouraging police officers to gather “intelligence” data through arbitrary stop and frisk programs (sometimes referred to as “enhanced Terry stops”) to feed into automated intelligence systems, has created an environment where racial minorities end up bearing the costs of broken windows, while wealthy, White communities tend to see the majority of its bene-fits.¹¹⁷ The social disorder that broken windows policing targets has been shown to be a very fluid concept, where the perceptions of a minority neighborhood’s residents are often far different than those of the police officers patrolling those neighborhoods, who frequently come from other, wealthier neighborhoods.¹¹⁸

The result of the intelligence data collected based on race or class bias, or parochial perceptions of social disorder, creates an inherent bias in automated decision support systems that tends to be reinforced with every trip around the OODA loop.¹¹⁹ This result can be directly traced to the sort of feedback loops data-centric decision support systems like Compstat are prone to encounter.¹²⁰ That is, if the data used to initiate an automated decision support tool is biased or otherwise flawed, the “actionable intelligence” that emerges will likely also be biased or flawed. If this bad intelligence is then acted upon, the resulting stops or arrests will likely generate even more bad data, which is then fed back into the decision support system, and so on.¹²¹

B. Algorithms, Data, Neutrality, and Bias What makes automated, predictive policing systems an attractive as a

solution for LEAs seeking to increase efficiency is the same aspect that makes them potentially dangerous: We tend to implicitly trust algorithms and data since we assume that computers have no bias, and numbers do not lie.¹²² This common misconception is based on two fundamental misunderstandings of automated decision support or expert systems. First, while it is true that computers, as finite state machines that at their core (no pun intended) are strictly limited to the instructions we give them through their programming, it does not follow that the algorithms we run on these computers are necessarily unbiased. Computer programs—algorithms coded by humans into a form a computer’s chipset can interpret—are written with the objectives, design choices, and general experiences of the programmer as background. The series of instructions, data structures, and design choices that end up in a finished computer program can often translate subtle biases, often in unexpected ways.¹²³

For example, from an algorithmic modeling perspective, incorrect or imbalanced input data has long been shown to lead to biased results.¹²⁴ Perhaps the most commonly used statistical method in predictive modeling systems is linear regression,¹²⁵ particularly a maximum likelihood linear regression, which is a widely used technique in military and police predictive intelligence systems.¹²⁶ In systems such as these, the data used often contain a large number of events belonging to one class, while the other class contains only a few data points, a data disparity known as a class imbalance.¹²⁷ This phenomenon often occurs in data where the event of interest (the crime) is sampled far more frequently than non-events.¹²⁸ This problem also manifests itself in the differences in class data representations between the sample set and the actual population, known as sampling bias.¹²⁹ There are, of course, statistical sampling methods to mitigate these effects, but there is no clear consensus as to which method of class distribution sampling will work best in all—or even most—situations.¹³⁰ The best solutions tend to be those that are specially selected based on such factors as the statistical methods employed, the population size, the sample size, and specifics regarding the event in question.¹³¹ In plain language, this means that a one-size-fits-all solution is likely to produce questionable results, at best, and at worst, dangerously biased results. This danger becomes increasingly amplified when one examines the trend of police analysis being extended from the realm of geospatial analysis—predicting which neighborhoods are most likely to be crime hot spots—to the individual, where police keep close tabs on people who, according to predictive algorithms, are more likely to be involved in future crimes.¹³²

C. The Introduction of Automation Bias The rise of the information and audit society and the associated increase in the use of automated information systems in organizational decision making often leads to an overreliance on—and overconfidence in— the results of these systems. This automation bias leads to misuse of auto-mated intelligence systems combined with automation-induced user complacency.¹³³ The negative effects of this automation bias have been seen in healthcare, transportation, power distribution, defense, and space exploration, often with serious, life-threatening consequences.¹³⁴ There are multiple reasons for this behavior, including our natural tendencies to seek out paths of least cognitive effort, to expend less energy when part of a team (including teams with automated members), and to treat computers as decision-making authorities.¹³⁵ These errors have been further categorized into two classes that manifest in automated environments: omission errors, where operators fail to respond to system anomalies because the automated system fails to detect or warn of them; and commission errors, where users blindly follow incorrect guidance from automated systems in spite of contraindications from other information sources.¹³⁶ Studies have repeatedly shown that automation bias of both types leads to users making incorrect decisions at a rate as high as 75%, even when the information they needed to make the correct decision was readily available.¹³⁷

Automation bias becomes especially dangerous when life or liberty is at stake. Multiple studies in domains such as health care, air transportation, and military command and control have repeatedly shown how bias and complacency lead users of automated systems to make very costly mistakes.¹³⁸ In military environments especially, overconfidence in the authority of automated decision support systems can be particularly catastrophic, where the importance of situational awareness is paramount.¹³⁹ The average person is well equipped to engage in naturalistic decision-making processes, where one is expected to solve real-world problems under a certain amount of stress.¹⁴⁰ We are, however, prone to overreliance on sources of information that we regard as authoritative.¹⁴¹ Military intelligent decision support systems operate within organizational hierarchies wherein users are predisposed, through their training, to defer to authorities within their supervisory structure, a trait that remains in effect when users of these systems seek guidance from algorithms and data structures.¹⁴² The natural result is an amplification of automation bias in these overtly hierarchical environments, where users exhibit tendencies to rely exclusively on automated systems, even when conflicting information is presented by other available systems.¹⁴³ The transfer of military intelligent decision support systems to civilian law enforcement organizations, where the paramilitary organizational structure closely resembles that of the military, makes police susceptible to the same dangerous automation bias exhibited in military environments. So how do law enforcement organizations take advantage of continued advances in automated intelligence and decision support techniques without either further alienating the communities they serve or succumbing to various data- and algorithm-based biases?

IV. A Community Policing Solutionto

Algorithmic Race Bias

Of course, to completely ignore the opportunities presented by advances in automation makes no sense. There are many tasks that computers simply do better than humans, such as repetitive tasks, rapid response to control tasks, rule-based deductive reasoning, and simultaneous task handling.¹⁴⁴ As our systems—both human and computer—grow increasingly complex, we need automation to give us the enhanced capabilities to handle time-critical and complex control environments. The trick, then, is to recognize the critical role automated information systems play in these domains, but at the same time, maintain an informed awareness of the pitfalls an overreliance on automated decision support can bring. This is of great importance to LEAs, who have a special duty to their communities, not only to enforce the laws but also to protect and maintain the health and safety of everyone in those communities. Allocating the appropriate amount of functionality between police officers and automated systems is critical to this role. In this Section, I recommend a two-element approach to this problem that takes into account both the important social role police play within their communities as well as the phenomenon of automation and data bias that can artificially reinforce racial disparities in police treatment.

A. A Return to the Original Intent of Community Policing

A significant part of the original broken windows policing concept articulated by Wilson and Kelling was the role of the police officer in reassuring community members of their safety, and maintaining a high degree of sensitivity to signaling by community residents, with respect not only to criminal activity, but also with their comfort with the police agency itself.¹⁴⁵ Community policing requires a more holistic approach to the problem of public safety that goes beyond mere crime fighting to encompass overall community health, safety, and quality of life.¹⁴⁶ Under this model, crime fighting was not an end in itself, but the means toward healthier communities, and was seen as a more modern, inclusive method of policing.¹⁴⁷

Too many implementations of broken windows and intelligence-led policing models, however, failed to follow through on this part of the theory. Rather than measure their performance—and direct their activities— using data that reflected a community’s overall health and quality of life, systems like Compstat relied heavily, sometimes exclusively, on traditional crime statistics, such as the number of stops, arrests, and clearance rates, despite the fact that these metrics have repeatedly been shown to have little to no bearing on overall community safety.¹⁴⁸ Sadly, this flawed approach to the original community-policing concept originates in the fact that these crime statistics are easy to collect and measure, and police depart-ments have developed a high degree of comfort with these metrics over the years.¹⁴⁹ Thus, the most direct approach to solving the problem of bad data leading to biased results from data-centric decision support tools is to require that police departments retool their data collection and analysis efforts toward more meaningful metrics.

Another misinterpreted requirement of the original Wilson and Kelling model is the concept of proactive policing. Most police departments implementing broken windows models have designed their systems as incident-oriented frameworks, which do a poor job at addressing levels of criminality in a community and serve mainly to feed a cycle of incarceration.¹⁵⁰ The Wilson and Kelling method of proactive policing focuses instead on the root causes of criminality, such as poverty, economic and ecological injustice, and racism, where police departments form part of a larger community team to go beyond the punishing of window breakers, and actually fix the broken windows.¹⁵¹

Finally, community policing requires police departments to hone their sensitivities to cultural norms and the needs of the community, since perceptions of social disorder can be highly dependent upon time, place, and circumstance, and most police officers are not residents of the communities they patrol.¹⁵² Public perceptions can also vary by context, which means social disorder often falls into the “I can’t define it, but I know it when I see it” category.¹⁵³ When police officers develop a feel for their communities they patrol, they will also benefit their departments by providing better, more meaningful intelligence data, which can then be used to improve their decision support outcomes.¹⁵⁴

Police department policies and procedures that are put in place with automated decision support tools like Compstat, which supports the broken windows policing regimes, have had a distinctly negative effect on minority communities.¹⁵⁵ As discussed in Section III above, this problem is not a new one, but is instead one that has been exacerbated by the use of military tactics and technologies.¹⁵⁶ LEAs, already obligated to collect evidence, especially feel the pressure in information societies to observe, collect, record, and evaluate all available data.¹⁵⁷ This predilection leads police to treat their communities as intelligence landscapes and people as “data elements,”¹⁵⁸ resulting in dissociative or adversarial relationships between cop and citizen.¹⁵⁹

This result is counter to the original intent behind community policing.¹⁶⁰ In hindsight, there is a certain level of disconnect between the algorithmic policing model and the community policing model, since algorithmic policing seeks to build a centralized, automated police force that operates from the inside out, while the community policing model requires a process that begins and ends with the citizen.¹⁶¹ But a key driver behind broken windows and algorithmic policing is a heightened fear of crime and disorder, a fear that has not diminished significantly over the past few decades, despite the nearly universal drop in crime rates nationally.¹⁶² A necessary part of this transformation, therefore, is to cultivate a base level of trust within police organizations as well as stakeholder communities. One method of building this trust is to continue to use the auto-mated policing systems while redesigning them to focus on local community goals rather than those of police departments or outside interests. Transparency is critical to this step’s success—police management must create a system through which community members can not only seek police assistance, but can examine the systems and goals the police themselves use to guide their day-to-day patrol activities. Another critical part of this transformation is an immediate increase in accountability within police departments. Automation bias, data bias, and corrupted procedures and goals often give police departments a certain amount of artificial cover when things go badly. However, technological opacity is no substitute for human accountability up and down the LEA management chain. Finally, law enforcement resources must be shifted towards a citizen-focused organization, giving rank-and-file police officers the power to help drive automation policy, while also giving them the discretion to problem solve independent of the automated process.

B. Incorporate Outcome and Process Feedback into Existing Systems We live in a time in which our lives are increasingly influenced and

affected—whether we know it or not—by data, algorithms, and machine learning.¹⁶³ It would be a mistake to believe that police departments would somehow be immune to this trend. Therefore, the solution to modern (or post-modern) problems of public safety is not to go the way of Ned Ludd, but to develop police decision support systems with an eye toward civil rights, and avoiding race discrimination and economic injustice.

The key principle in developing any of these systems is transparency. In Floyd et al. v. City of New York ,¹⁶⁴ the court held that the NYPD’s stop and frisk program violated the Fourth Amendment by systematically con-ducting warrantless searches of pedestrians, the majority of which were African-American or Hispanic. Under the NYPD’s stop-and-frisk program, police made over 4.4 million stops between 2004 and 2012.¹⁶⁵ Over half of these stops were of Black citizens, thirty percent were of Hispanic citizens, and only ten percent were of White citizens.¹⁶⁶ The guidelines that backed such a system were based on opaque police policy decisions implementing a broken windows policing model, and thereby creating the sort of insular environment that often fails to punish bad actors and creates perverse incentives.¹⁶⁷ Further, since most police departments do not employ software developers or statisticians, they are forced to purchase their decision support systems from third party contractors.¹⁶⁸ The same principles of transparency should apply to contractors so as to avoid these problems.

Next, algorithmic and predictive policing systems need to be based on accurate and meaningful data. For example, police stop and frisk pro-grams, despite criticisms and claims of racial bias inherent in these pro-grams, have been widely supported by LEAs based on their assertions that, while more minorities are subject to these stops, it is only because minorities commit disproportionately more crimes than Whites, and not due to any particular bias of police officers and departments.¹⁶⁹ To support these claims, police often cite automated police intelligence systems, such as Compstat, to justify these stops.¹⁷⁰ Statistical studies conducted on stop and frisk data, however, have shown that minorities are far more likely subjects of these programs, with statistical patterns pointing toward a structural racial bias, reinforced through automated decision support tools.¹⁷¹

Finally, we must be cognizant of the limitations of purely technical solutions to human problems. Mathematical and computer models of real-life systems can, of course, provide critical insights into complex systems, but they are imperfect. Algorithms for deriving patterns from large amounts of seemingly random data are getting better as research and technology progress, but their most effective use within the broken windows policing model is as a supplemental tool informing human decision making, not as a digital crutch upon which bad practices and biased policing may rest.

Further, known biases in automation systems can be mitigated through a number of established means. For example, automation bias and automation complacency can be avoided by increasing accountability by the users of a system.¹⁷² By requiring system operators to provide complete justifications for their decisions—beyond “the machine told me so”—users will be driven toward deeper cognitive engagement and awareness of alternative information sources.¹⁷³ The level of automation used in a situation should also be carefully assessed on a domain basis. That is, the level of automation available can exceed the level of automation necessary for a given situation.¹⁷⁴ Repetitive, rigid tasks that expect no user decision-making flexibility are often good candidates for a high degree of automation, while those tasks that rely on human intuition, pattern perception, and contextual reasoning, are best served with lower levels of automation.¹⁷⁵ Much of the work of law enforcement falls into this latter category.

For the time being, algorithmic and predictive approaches are only as useful as their human creators. Selecting the proper policing model for every instance is likely impossible, so one of the most important decisions in the field of machine learning is the selection of the model that will provide superior results for a particular problem, a task which still requires an experienced and informed human in the loop.¹⁷⁶ But this limit can be leveraged as a benefit, rather than a liability, by using predictive systems to decentralize police command structures, and allow more creativity and initiative among rank-and-file patrol officers, characteristics that are critical to a true community-policing model.¹⁷⁷

Conclusion

The problem of bias in algorithmic policing has deep roots, as evidenced above. Solutions to this problem cannot ignore technological advances that help us make better, more efficient decisions, but they also cannot allow these technologies to subvert the proper role of public safety in our communities. We live in an information society that is, once again, experiencing a crisis of control that we are naturally inclined to solve through data and analytic methods. But our approach must be based on lessons learned from our successes and failures in this arena. Many of these failures have led to a trust deficit between authorities and the communities they govern, especially where racial prejudices have been part of these fail-ures. Addressing these disparities in algorithmic policing cannot solve all of these problems, but it is a good start.