The article well describes three flaws in experimental designs typically used to examine biases. However, there is another fundamental flaw that has been overlooked: consequences. Poor performance has implications in real-world tasks where both slow and inaccurate responses lead to disastrous outcomes. The three flaws address each problem as it exists in a laboratory setting, yet there is no discussion regarding how the consequences of a decision might influence behavior. Specifically, your behavior can change when you know the situation is real and not just a simulation.

Nowhere is this omission more salient than in the first-person shooter paradigm. The author discusses dispatch priming (Taylor, Reference Taylor2020), contextual information about the environment (Correll, Wittenbrink, Park, Judd, & Goyle, Reference Correll, Wittenbrink, Park, Judd and Goyle2011), realism through enhanced simulator scenarios (James, Klinger, & Vila, Reference James, Klinger and Vila2014; James, Vila, & Daratha, Reference James, Vila and Daratha2013), and even a thorough discussion about training (Cesario & Carrillo, Reference Cesario, Carrillo, Carlston, Johnson and Hugenbergin press; Cox & Devine, Reference Cox and Devine2016; Sim, Correll, & Sadler, Reference Sim, Correll and Sadler2013). Consequences remain conspicuously absent. For example, laboratory-based paradigms do not regularly impose consequences after making a poor decision. Shooting tasks rarely impose any penalty for firing upon an unarmed person, and when they do, the consequence is more likely to be a point-based deduction (Biggs, Cain, & Mitroff, Reference Biggs, Cain and Mitroff2015). Experiments normally just proceed to the next trial, whereas these real-world errors are followed by detailed officer-involved shooting investigations and sometimes criminal punishment.

There are also no consequences to the shooter for moving too slowly. For all the deliberation about experimental paradigms, there is no discussion about a hazard present in many lethal force encounters – hostiles can shoot back. Realistic lethal force engagements carry life-or-death significance for the shooter too as moving too slowly could mean being shot by a hostile adversary. This threat imposes consequences for failing to act in addition to the consequences for making the wrong decision. There is an entire literature on this topic absent from the discussion that thoroughly addresses the stress and anxiety present in lethal force scenarios because of pressure and consequence (Nieuwenhuys & Oudejans, Reference Nieuwenhuys and Oudejans2010, Reference Nieuwenhuys and Oudejans2011; Oudejans, Reference Oudejans2008; Patton & Gamble, Reference Patton, Gamble, Schmorrow and Fidopiastis2016). Among the various influences that might alter performance in a shooting task, hostile action should be represented with the same prominence and concern as using a realistic weapon. Moreover, consequences do not readily fit into any of the missing categories, which is why they should be described as a fourth flaw.

Contrast this prospective influence due to a course of action with the stated three flaws. For the missing information flaw, the concern is creating unrealistic conditions in the laboratory for the sake of experimental control. This flaw emphasizes making artificial factors more authentic, albeit the unintended consequence might be embracing superficially related components as more genuine when they are not as interrelated. One such instance is how marksmanship and decision-making are more disconnected than they seem, making marksmanship an orthogonal factor to the lethal force decision-making process (Blacker, Pettijohn, Roush, & Biggs, Reference Blacker, Pettijohn, Roush and Biggs2021). The irony is that authentic grip and firearm functions suffice for realism without being concerned about what the bullet does when it hits the target.

For the missing forces flaw, the focus is on context and frequency. Inter-trial features and background context become tools to prime decision-making similar to how go/no-go trial ratios influence the strength of prepotent motor activity during inhibitory control tasks (Wessel, Reference Wessel2018). The focus is again upon influencing decision-making factors without concern for how one decision affects subsequent decisions.

For the missing contingencies flaw, trained personnel will know the difference between live fire and simulation better than anyone else. The role of consequences may be more illuminating for them given that they have a true understanding of the difference between firing a real weapon versus mimicking an action. One phenomenal missing contingencies argument involves the reliance upon misidentifying harmless objects as a crux of first-person shooter tasks. There are other ways to explore errors in lethal force decisions by intentionally introducing ambiguity into the task (Biggs, Pettijohn, & Gardony, Reference Biggs, Pettijohn and Gardony2021), which shooting paradigms could exploit.

Still, the common missing element across all three flaws is consequence – shooters can fire too slowly without getting hurt or shoot unarmed targets without punishment. Training instructors cannot avoid this topic in the same way as experiments that design around the problem. Handing someone a live weapon versus a plastic toy will inevitably impose some level of stress and anxiety. Rather than avoid the challenge, trainers sometimes address anxiety and realism with non-lethal training ammunition (Taverniers & De Boeck, Reference Taverniers and De Boeck2014; Taverniers, Smeets, Van Ruysseveldt, Syroit, & von Grumbkow, Reference Taverniers, Smeets, Van Ruysseveldt, Syroit and von Grumbkow2011). The simple solution is to impose a consequence. Shooters will feel the pain sensation of being shot, and they know their own behavior might inflict pain on someone else (Biggs & Doubrava, Reference Biggs and Doubrava2019). Because the simulation is now conducted against a dynamic and thinking opponent, with the consequence of being shot, the result is a more realistic training environment. It just cannot be easily replicated in a laboratory setting. The challenge is transitioning the experiment to the field conditions rather than trying to make the laboratory more like the field. Find the operational need first, figure out how it is trained, and make the experiment match that scenario. Do not design in reverse and try to find an operational need that fits your experiment without acknowledging the applied limitations of this approach.

By focusing on the operational needs first, and then building a laboratory paradigm to replicate that need, the experimental flaws are far less likely to be overlooked. Methodological issues such as measuring reaction time with training weapons should be overcome with innovation rather than built into studies as experimental flaws. Moreover, the resulting study is more likely to have a real-world consequence as there could be a method to measure results, compare them to existing procedures, and finally integrate changes into training. Begin with a transition plan focused on the end user – and if the experimental flaws are not avoided, they will become clear.