1. Introduction
Collider experiments in High Energy Physics (HEP) frequently deploy analysis strategies called signature-based model-independent (SBMI) searches. The idea of these analyses is to look for new phenomena in a data set by applying event selections that are not optimized for a particular beyond-the-standard-model (BSM) theory. Examples of such analyses can be found in various experiments (see CDF Collaboration Reference Collaboration2010; ATLAS Collaboration Reference Collaboration2017; CMS Collaboration Reference Collaboration2018). These can be contrasted with BSM-oriented searches in which an optimization procedure is carried out by explicitly scanning over the BSM model parameter space and defining, for each parameter-space point, which set of selections maximizes a figure of merit such as the significance of an excess over the Standard Model (SM) expectations. The goal of the SBMI search analyses is to avoid biasing the discovery potential of the overall experiment toward any particular BSM theory assumption, therefore increasing the chances that the experiment makes an unexpected discovery.
In what follows, it will be argued that, despite the intentions, SBMI searches for evidence of a signal beyond the SM constitute explorative strategies that do depend on multiple theory assumptions including preconceptions about the target phenomena. The core of the argument is that these experimental strategies are articulated together with theory-based searches, in a complementary way, in order to optimize the scientific products of the experiment, along with a safety principle to be defined in section 3. The dependence of SBMI searches on BSM theories is therefore indirect but nevertheless central to the potential success of the experiment with respect to its objectives.
The next section will put SBMI searches in the context of experimental explorative strategies. Section 3 will propose a safety principle relevant to understand how the objectives of SBMI searches affect their experimental strategies. Finally, in section 4, it will be argued that SBMI searches are theory dependent. Philosophical consequences will be outlined in the conclusion.
2. Signature-Based Model-Independent Searches and Explorative Experiments
The idea of characterizing a class of experimental procedures as being specifically explorative has been pioneered by Steinle (Reference Steinle1997, Reference Steinle2002) and further discussed by Burian (Reference Burian1997), Elliott (Reference Elliott2007), and Waters (Reference Waters2007), among others. Their account of experimental exploration (EE) can be summarized as a set of epistemic strategies consisting of a systematic variation of the parameters and conditions of an experiment with the aim of developing new ideas, related to empirical regularities, deployed when there is no well-formulated theoretical framework guiding the experiment. Scientists would use such experimental strategies to empirically discover something new and unexpected to generate new theory developments. For Steinle and Burian in particular, the role played by theory in EE is minimal. This seems to adequately describe the intentions of SBMI searches.
Karaca (Reference Karaca2017) however argued that the above conception of EE is too restrictive to properly account for the class of experiments that are in a discovery mode, that is, aiming at empirically producing and studying new phenomena that will in turn allow the formulation of new theories. In large experiments like the Large Hadron Collider (LHC), theories appear indispensable to identify interesting and relevant phenomena against a vast background. Karaca proposed to extend EE to strategy-driven experiments (SDE) to characterize the guidance provided by theory when designing search strategies. His main argument is that the data collection procedures used in LHC experiments, through their trigger menu, have to be designed in a way that allows for the discovery of new physics.Footnote 1 LHC physicists guarantee this by building the trigger menu using the simulation of BSM models.
It is however not a priori obvious that Karaca’s account of SDE applies to SBMI searches. From their intentions, it seems that these are simply EE in Steinle’s sense. First Karaca gives too much importance to the trigger phase of HEP experiments. Trigger menus are designed to be much more general than offline analyses: a given trigger element covers multiple event signatures. In the LHC jargon, a signature is an event that refers to a combination of instrumental signal components. Different final state particles will interact differently with the various parts of a LHC detector such as ATLAS. It is the set of all the detected (and reconstructed) final state particles that form an event. The same set of detectable particles can represent both SM and BSM underlying processes; they constitute a signature of multiple target processes. The trigger menu focuses on the detectable particle signals and must accommodate multiple objectives of the experiment, not just searches for new physics. It aims at keeping as many events as possible, and there is a large overlap between the data selected by each of these objectives. In order to determine whether an event is interesting, offline-level sensitivity studies are performed using Monte Carlo simulations. The impact of BSM models on the design of a trigger menu is therefore limited to some very specific offline-level sensitivity studies. So, even if the trigger chronologically comes before offline data analyses, its dependence on specific theories points back to the offline component of the experimental investigation, a nuance that Karaca failed to identify.
Second, SBMI searches consider signatures that could be associated to a process for which the theory has not yet been developed. They would simply use existing triggers without the need to be justified by specific BSM theories. It is also possible to add the signature of interest to a trigger menu without adding a large extra cost in CPU and memory usage because of the bandwidth overlap it would have with existing trigger elements. It therefore seems that SBMI search strategies do not rely on BSM theories at trigger and at offline levels. They could be considered as EE when the focus on the signature is systematically varied until a discovery is made.
It is however important to realize that the key point of Karaca’s argument is not just about triggers but about the importance of theory guidance, which is lacking in Steinle’s conception of EE, to guarantee that the experimental strategy put in place can meet its objectives. Karaca argued that at the LHC, BSM theories about the target must be used to guide the development of the explorative experimental procedures to make sure a discovery is possible. To determine whether SBMI searches are EE or SDE, the question is, therefore, Are BSM theories used to guide SBMI searches in order to allow them to meet their objective of finding new physics? To answer this question, a better characterization of the objectives of searches at the LHC is first needed, and this will be done through the introduction of a safeness principle.
3. Safeness Criteria
3.1. Type I Safeness
Statistical analyses help define two types of safeness criteria on the basis of the statistical errors that can be made when making test hypothesis inferences from data. In his description of the various steps that led to the discovery of the Higgs boson, Franklin (Reference Franklin2017) alluded to the effort of experimenters to reach “safe claims” in the interpretation of their results. He however did not attempt to define what safeness is, nor did he try to apply it to the experimental procedures deployed upstream of the interpretation phase of the experiment. The safeness criteria that will be defined here are intended for evidence obtained and used to discover new phenomena.Footnote 2
The first criterion defines safeness on the basis of analogy to type I error, which consists of making a false discovery claim (see Behnke et al. Reference Behnke, Kroninger, Schott and Schorner-Sadenius2013, 155–56). A type I safeness criterion can be formulated as follows:
An experimental result is type I safe if the evidence it provides for a discovery claim allows one to place an upper bound of α on the probability that such a discovery claim is false.
The value of α quantifying such a criterion is not arbitrary but is a consequence of the validation process performed within the experimental procedure. It is determined by the statistical and systematic uncertainties estimated as part of the measurement result and by the level of discrepancy between the null hypothesis and the observation. The choice of α is not absolute either: the evaluation of the systematic uncertainties often reflects the level of aversion of the experimenters to type I errors (Nachman and Rudelius Reference Nachman and Rudelius2012). In addition, as argued by Beauchemin (Reference Beauchemin2017), the quantification of α follows from an a priori assumption about how the measurement results would distribute in the uncertainty interval quantified by the measurement process if the measurement were repeated in different conditions. A type I safeness criterion therefore implies a certain arbitrariness that is nevertheless rooted in the practice of producing scientific statements that have a strong epistemic value in a field of science from empirical evidence.
The question is, then, How much does the type I safeness affect the strategy of an HEP experiment that aims to discover new physics? The answer is that the experimental procedure leading to a discovery claim will expend a large effort at making sure that no source of uncertainties has been forgotten and that the size of these uncertainties reflects how the results depend on the assumptions used to obtain them. Failure to pay sufficient attention to type I safeness would lead to a high probability of false discoveries that would shake the confidence of the larger HEP community in the quality of the experiment.
However, as argued by Humphreys (Reference Humphreys2004), the safety of empirical evidence is frequently revised from a series of adjustment processes within the experiment. That will affect the discovery claims made and the type I error estimated. When false discovery claims are made, new pieces of evidence or theoretical development should eventually lead to the withdrawal of the discovery claim. One recent example of this in HEP is the third and fourth versions of the OPERA Collaboration publication on their measurement of the neutrino velocity in which they corrected the experimental error affecting the results they reported in the first two versions of the article, withdrawing their evidence for an anomalous superluminous signal (OPERA Collaboration Reference Collaboration2012). The scientific program of the experiment kept going according to the original plans despite this glitch in the process. There is therefore an impact of type I safeness on the design of an experimental procedure, but it is not irreversible.
3.2. Type II Safeness
A second safeness criterion can be defined on the basis of an analogy to type II statistical error (Behnke et al. Reference Behnke, Kroninger, Schott and Schorner-Sadenius2013, 155–56), which consists in not being able to make a discovery claim when it should. Such a criterion can be formulated as follows:
An experimental result is type II safe if the probability that the experiment fails to provide evidence for a discovery that it could make is limited to at most β.
To quantify β, the uncertainty on the measurement result must be known, which means that the choice of α made by an experimenter also affects the value of β. Such quantification also requires a sensitivity analysis, that is, the comparison of the measurement results with the theoretical predictions about the new phenomena to be discovered, knowing the uncertainties on both. However, because such a sensitivity study is dependent on the comparison to some theoretical predictions, it cannot account for discoveries of something unexpected, as is intended by SBMI searches. The value of β cannot therefore be unambiguously quantified for the entire explorative component of an experiment. In the case of SBMI searches, it relies more in the appreciation of the experimenters for the designed experimental procedures, and it relies on the knowledge of the β values in BSM-oriented searches. This is similar to the fact that we cannot be sure that all the sources of systematic uncertainties have been accounted for when assessing α. This is why these safeness criteria follow from analogies to statistical errors, rather than being determined by them. They nevertheless capture well the concerns of experimenters when they design and perform experiments with the expectation of producing secure empirical evidence about some target phenomena.
In contrast to type I error, a claim for no discovery following an error of type II cannot usually be reverted by reviewing the experimental procedure: either the discovery will be made by another experiment, or it will never be made. The assessment of type II safeness by experimenters will therefore involve a strong pragmatic component, explicit or implicit, used to evaluate the consequences of such an error on the scientific program of the experiment and to prioritize the objectives of the experiment. Ensuring type II safety is therefore a crucial element for the explorative component of an experiment. The question to be addressed with this article can thus be formulated as:
Do the SBMI search explorative strategies deployed by LHC experiments require a significant amount of theory input about BSM models in order to be type II safe?
Before answering, we first need to assess what are the conditions for LHC experiments to be type II safe.
3.3. Type II Safeness at the LHC
Since the discovery of the Higgs boson at the LHC (ATLAS Collaboration Reference Collaboration2012; CMS Collaboration Reference Collaboration2012), the SM of particle physics has been fully validated. It is certainly possible, and even important, to improve on the description of some aspects of the SM, but the core of the theory has been fully accepted by the HEP community. Since 2012, the primary purpose of the LHC experiments is therefore to discover phenomena beyond the SM. There is some dissatisfaction with SM, and even some inconsistencies, that requires extensions of the SM. A plethora of models have been formulated by theorists to address some of the issues with the SM (Lykken Reference Lykken, Grojean and Spiropulu2010). According to many of these models, the LHC has the right energy to provide evidence for their empirical adequacy. There are therefore strong scientific reasons to expect to discover new phenomena at the LHC. These reasons urge LHC experimenters to keep low β values, that is, to design as strongly type II safe explorative strategies as possible. If it is expected that some discoveries can be made, then the experimental strategies put in place must be such that the discovery will be made.
However, these justifications for keeping β low are exacerbated by further pragmatic considerations. The cost of an experiment like the LHC is about 10 billion US dollars. All the funds almost fully come from public sources, coordinated among over many countries. In addition, the time invested in the experiment, from conception to conclusion, covers more than 3 decades. The LHC might well be the last of these very high-energy collider experiments. Scientifically, it means that the puzzles raised by theorists about the SM might never be answered. It also has a profound impact on the careers of the young physicists involved in the experiments. It is therefore of the utmost importance that LHC experiments succeed in making all the discovery claims that are within their reach. The cost of failing to do so might be as large as ending HEP as a field of research. Both pragmatic and scientific considerations therefore push HEP physicists to develop explorative strategies that would make their experimental results as type II safe as possible, even favoring type II safeness over type I.
Up to now, there is no empirical evidence that the SM cannot explain all the HEP phenomena below the TeV scale. There is no experimental support for any known BSM theory. In addition, many of these theories cannot be fully trusted as they correspond to a minimal viable model. They will require adjustments in the sense of Humphreys (Reference Humphreys2004) when new phenomena are discovered. Finally, the history of science provides a long list of “accidental discoveries,” that is, discoveries that were not predicted by any untested theory. These considerations indicate that it is highly possible that if there is new physics within reach of the LHC, it would take a different form than any of the currently postulated BSM models. This forces HEP physicists to include in their experimental explorative program strategies that are not biased toward any existing BSM theories. This is exactly what SBMI searches are. In order to maximize the chance of being type II safe, the LHC physics program must therefore complement BSM-guided searches by SBMI searches. The question remains: Are BSM theories needed to design these SBMI searches in order to maximize their type II safeness?
4. Theory Dependence of Signature-Based Model-Independent Searches
4.1. Acceptability of Some Theory Dependence in Explorative Experiments
Complex experiments like those at the LHC have a very broad research program that makes room for many different experimental tasks such as testing, measuring, calibrating, discovering, and so on. These tasks develop into different instrumental designs (including software) and experimental procedures, including multiple correction inferences and uncertainty treatments. As was argued by Beauchemin (Reference Beauchemin2017), these processes, which lead to the production of experimental evidence, inevitably rely on theory input to be possible. Humphreys (Reference Humphreys2004) similarly argued that the security of evidence is guaranteed by such theory dependence of the observations. While Beauchemin and Humphreys concentrated their analysis on measurements and theory tests, Franklin (Reference Franklin2017) argued that the sensitivity of theory-based searches for hitherto unobserved particles, such as the quest for the Higgs boson before 2012, also depends on models, ancillary assumptions, uncertainty estimates, modeling of detector effects, and decision criteria.
SBMI searches proceed from event reconstructions, estimates of the SM contribution to a data set of interest, evaluation of the various uncertainties associated with this estimate, and statistical tests used to quantify the level of agreement or discrepancy between the data and the SM expectations, similarly to measurement and theory-based searches. They hence also require a large contribution of theory input in order to lead to empirical evidence for new physics. If these theoretical assumptions were changed, the signature could be changed as well. Franklin (Reference Franklin2005) argued that EE, however, does not preclude such theory dependence: the lack of theory guidance that is relevant to Steinle’s account of EE should concern theories about the target phenomena (i.e., about BSM signals), not about any background knowledge used to yield the experimental results. From that point of view, SBMI searches can still be considered as EE if no BSM-oriented theory is used to guide their experimental strategy; otherwise they would be SDE in Karaca’s sense. To find out whether BSM-oriented theories are used as input to the experimental strategy defining SBMI searches, a more detailed analysis of such searches is needed.
4.2. The Indirect Dependence of SBMI Searches on BSM Theories
In order for SBMI searches to optimize the type II safeness of the scientific endeavor of LHC experiments (i.e., in order to maximize their chance of discovery), BSM models have to be invoked to define the signatures in which SBMI searches will be performed. First of all, there is a very large, if not infinite, number of signatures that could be used to search for new physics. Many of these signatures differ only in the value of some of the kinematic cuts used to define them. The same new physics signal could therefore populate many of these signatures. Since there is a penalty to pay for collectively looking for new phenomena in many different signatures (the so-called trial factor), not all of the signatures will be used to define an SBMI search; otherwise it would increase the value of the β of the type II safeness criterion. A choice of signatures must therefore be judiciously made on the basis of the discovery expectations for such signatures. For example, events with a large amount of missing transverse momentum (ET miss) will systematically be used to design a class of SBMI searches because of the expectations HEP physicists have for dark matter at the LHC. SBMI searches therefore depend on BSM theories to justify that a choice of a signature is promising for a new physics discovery. BSM theories thus affect the strategy adopted by SBMI searches and the level of increase in the type II safeness such a search adds to the explorative component of the experiments.
Moreover, the results of a SBMI search are always interpreted in terms of some BSM benchmark models. The observational claims published by HEP experimental collaborations report some constraints on BSM theories, even if these constraints have been obtained from SBMI searches. It allows for the comparison of results obtained by different experiments on similar analyses and so is important for the evaluation of the conclusions, obtained by an experimental collaboration, by the rest of the HEP community. This is not a mere interpretation of the observations: it includes, in the uncertainty defining the measurement results, sources from the BSM theory. In addition, details of the benchmark models will be fed back into an adjustment of the SBMI searches, if the newly acquired empirical information is a null result, or into the development of a new set of measurements aiming at characterizing better the process underlying the discovery, in the case of a positive result. For example, after failing to find a new physics in a given jets + ET miss signature, a benchmark BSM model might be used to refine the selections of this search in a way that will improve the discovery potential to dark matter with this signature. Such an adjustment process depending on BSM theory is a fundamental factor of the type II safeness of LHC experiments. From these considerations, it becomes clear that SBMI searches cannot happen without contribution from some BSM models, beyond the simple innocuous guidance of what a group of scientists chooses to investigate.
4.3. Are SBMI Different from BSM-Oriented Searches?
Do the above arguments in favor of a theory dependence of SBMI searches imply that such search strategies are no different from BSM-oriented strategies and that the attention paid by HEP physicists to SBMI is in vain? The answer is no. There are some differences that justify designing the LHC physics program with the inclusion of these two search strategies. The difference lies in the data selection strategy of each type of search. While both BSM-oriented and SBMI search discovery strategies globally can be sensitive to the same pool of potential new physics phenomena, the quantitative sensitivity to new physics will differ. The main purpose of SBMI searches is therefore to optimize the overall sensitivity to new physics of the entire experiment by covering sensitivity “holes” in the net of BSM searches. While the phase space covered by the selections optimized on the set of all BSM-oriented searches has a strong sensitivity to any conceivable new physics scenario consistent with these BSM models, sensitivity to new physics can significantly drop when moving away from these phase space points. SBMI search is therefore a strategy to perform further search for new physics in these regions of the phase space where the sensitivity to known BSM models might drop. SBMI searches are therefore epistemically similar to BSM-oriented searches, but they cover regions of potential discoveries that are suboptimal for these BSM-oriented searches. Both approaches are therefore complementary; they are designed in tandem. This makes the entire explorative component of the LHC physics program as type II safe as it can be, thanks to the theory guidance that is used to develop these search strategies in parallel.
5. Philosophical Conclusions
There is a strong explorative component to HEP experiments: the search for new physics. It operates under a principle of type II safeness that depends on the objectives of the experiment, plus some pragmatic considerations, and determines important aspects of the search. The objective is simple: make sure that if there is a discovery within reach of the experiment, its explorative program will have a strong chance of finding the new phenomena. In order to optimize its type II safeness, the physics program of LHC experiments deploys two complementary discovery strategies: BSM-oriented and SBMI searches for new physics. While BSM-oriented searches optimize the discovery power of the experiment on the basis of known extensions of the SM, SBMI are designed to cover holes in the new physics space of the BSM searches. The discovery objectives of an experiment can however only be met (i.e., the experiment can only be as type II safe as possible), if both search strategies are guided by theory about the target phenomena. SBMI searches therefore do not constitute a purely empirical core of an experiment, exempt of assumptions and conceptions about what the target phenomena could be: theories about what can be discovered are critical to the experimental process, even if the discovery to be made is very different from what has been conceived. This is the main conclusion.
This has profound philosophical consequences. One of them is to provide further support to the idea of SDE developed by Karaca from the example of explicit BSM searches. SBMI searches correspond to an explorative aspect of the LHC experimental program because they aim at formulating new theories. They do not function in the same way as theory-testing experiments. In order for these searches to meet their goal, they rely on theories about the target phenomena. Such theory dependence directly affects the experimental strategy that defines the observation, the justification of the experimental process, and the claims that are made from data. Because of this theory dependence, SBMI searches evade Steinle’s account of explorative experiments but fit Karaca’s account of SDE.
Another consequence is that the analysis presented above adds extra layers of theory dependence to observations over what Humphreys (Reference Humphreys2004), Beauchemin (Reference Beauchemin2017), and Karaca (Reference Karaca2017), among others, already argued: there is not a single component of complex experiments like those at the LHC that do not fundamentally rely on theory input about the targeted phenomena to reach their full epistemic value. Such a theory dependence does not jeopardize the security of the evidence used to yield knowledge in particle physics. Quite to the contrary, it allows obtaining secure evidence by making it as type I and type II safe as possible. It also does not imply that empirical evidence does not play a crucial role in the acceptance of claims that leads to scientific knowledge. It rather means that theory and experiments are two scientific activities fundamentally intertwined in the production of scientific knowledge.
A final consequence of the main conclusion is that the standards of evidence for having a phenomenon cannot be reached without the help of theory about what these phenomena are. This offers a new ground for rethinking the relation between data and phenomena outlined by Bogen and Woodward (Reference Bogen and Woodward1988, Reference Bogen and Woodward1989): the fundamental interdependence between theory and experiment in the production of observational claims makes it hard to clearly separate data from phenomena. The distinction would therefore only reflect a pragmatic division of scientific research efforts and focus of study. This point however goes beyond the scope of this article and would worth being developed more thoroughly.
