Research question and hypotheses

The ESM and parallel data are typically combined to (1) enrich subjective self-assessments of ESM with an ‘objective’ proxy or complementary data source, or (2) provide interpretable contextual or ‘ground truth’ labels next to volatile high-frequency time series. During the definition of the research question and its hypotheses, the limitations of both data types have to be carefully considered. A graphical representation of the anticipated results of the two variables of interest can be useful to grasp the fluctuation patterns within our variables of interest (Fig. 1). During this hypothetical exploration of the combined dataset, the question ‘What information does ESM data add to parallel data (or vice versa)?’ has to be leading. An understanding of the data collection, operationalization, and analytical techniques is therefore required. Answering this question helps to consider whether the combined use of ESM and parallel data is justified, meaning whether each data type has its own unique contribution while also maintaining synergy.

Since the variables of interest will result from two different data types and cover different timelines, it is essential to understand the expected temporal relationship between those variables and to specify the assumptions about the direction of the association of interest. For instance, are we interested in the parallel data features such as for example GPS location, preceding, following, or simultaneous to the ESM assessment of for example momentary anxiety (Fig. 2)? Or are we interested in relationships over time, the effect of moment n on moment n + 1, and if so, how long should the time lag be? In addition, the duration of parallel data corresponding with the ESM measure has to be determined. All these considerations should lead to a schematic draft of the temporal relationship between the variables of interest as seen in Fig. 2. Preregistration of the hypotheses and the study design will help solidify the expected associations and increase the robustness of the results (Nosek & Lakens, Reference Nosek and Lakens2014).

Fig. 2.

Visualization of three different pre-processing approaches to combine ESM and parallel data. (a and b) Two examples where parallel data are down-sampled for analysis (i.e. a summary statistic will be computed over a specific period), which leads to eight data-points per day for analysis. The parallel data used can be taken from the window preceding (i.e. a) or proceeding (i.e. b) each ESM observation (blue dot), dependent on the research question and hypothesis. (c) An example where ESM data are up-sampled (i.e. in-between observations are generated), which leads to two continuous time series for analysis. Panel (c) uses a least squares polynomial fit (Numpy.Polyfit – NumPy v1.20 Manual, 2022) for demonstration purposes. (a) Parallel data down-sampled: ESM v. preceding parallel data; (b) parallel data down-sampled: ESM v. subsequent parallel data; (c) ESM data up-sampled: both data as continuous processes.

It is important to note that these directional patterns can help ‘unpack’ mutually occurring temporal relationships but they cannot prove causal relations due to the observational nature of data collected in the flow of daily-life (Holleman, Hooge, Kemner, & Hessels, Reference Holleman, Hooge, Kemner and Hessels2020). Causality depends on several co-occurring factors and therefore cannot be claimed in such study designs (Rohrer, Reference Rohrer2018; Rubin, Reference Rubin2007). Researchers who are interested in a causal relationship in an ecologically valid context should consider an experimental design in which a daily-life variable, for example stress, is actively manipulated in order to test its effect on another variable (Smets, De Raedt, & Van Hoof, Reference Smets, De Raedt and Van Hoof2019). Although, no absolute claims about causality can be made, ecological data collection allows more control over a certain variable in a more natural environment. This benefit has to be carefully weighed against the threat that it poses to the ecological validity (Holleman et al., Reference Holleman, Hooge, Kemner and Hessels2020). Findings should thus be interpreted with this in mind.

Variable fluctuation and volatility

Once the variables of interest and the nature of their relationship have been identified, we need to understand their variability (i.e. how much do they fluctuate?) and volatility (i.e. how fast do they fluctuate?). These factors will determine data collection technicalities such as sensitivity (what is the smallest detectable symptom difference?), frequency (e.g. continuous high frequent sampling, multiple times per day or week), timing (e.g. morning, evening, event-triggered), and duration (days, weeks, months). Since the aim is to combine two separate data time series, it is essential at this time to define the temporal resolution of the anticipated ‘outcome variables’ or ‘fluctuation scores’ of both data types. If one data type results in outcomes of a higher frequency, a valid and meaningful aggregation method has to be designed to enable matching information from both data types.

ESM variables may have different fluctuations and volatilities, for instance depressive feelings may fluctuate slowly compared to anxiety, which may be more volatile. While it is tempting to select a higher ESM sampling frequency, this may cause increased participant burden, lower compliance, and lower data quality (Eisele et al., Reference Eisele, Vachon, Lafit, Kuppens, Houben, Myin-Germeys and Viechtbauer2020; Fuller-Tyszkiewicz et al., Reference Fuller-Tyszkiewicz, Skouteris, Richardson, Blore, Holmes and Mills2013; Trull & Ebner-Priemer, Reference Trull and Ebner-Priemer2020). Akin to parallel data variables, sampled data-points should not miss relevant fluctuations, but large amounts of redundant data should also be avoided.

Specific temporal associations between variables can drive the timing of the data collection. An association that includes a time-lag will determine the interval-period between consecutive assessments. On the other hand, when the focus is on a specific event-dependent time window, a data collection strategy based on a specific occurrence will be required; for example, triggering an ESM measurement following changes in physiology (van Halem et al., Reference van Halem, van Roekel, Kroencke, Kuper and Denissen2020).

It is of note, that variable fluctuations can vary depending on the population's health, social, economic, or cultural characteristics (Okun, Reference Okun2019). It is therefore advised to use explicitly validated variables or conduct a pilot study testing whether a protocol captures the expected fluctuations in the population of interest. Furthermore, studies including multiple parallel data types should address these questions for each data type separately.

The above-mentioned questions should be carefully addressed prior to data collection and ideally be pre-registered in one of the open science platforms most relevant to the specific field. There are many available platforms and guidelines to help explore different options (Kathawalla, Silverstein, & Syed, Reference Kathawalla, Silverstein and Syed2021). Due to the multiple types of data and the different steps necessary to track and report research, a platform that allows for greater flexibility, such as the Open Science Framework (OSF), is advised. The OSF provides a common place to enact all open science practices, such as pre-registration, data storage and sharing, code sharing, pre-prints, to name a few (Foster & Deardorff, Reference Foster and Deardorff2017).

Data analysis

Thus far, defining detailed hypotheses and their variables of interest including their intended operationalization and expected fluctuation patterns have been discussed. It is time to determine the pre-processing and statistical analysis that best answers our research question. Although a detailed statistical discussion is beyond the scope of this paper, it is necessary to highlight the importance of choosing the right analysis prior to data collection. The chosen methods of data pre-processing and analysis will likely influence the required study design, but also potentially limit the ability to answer the intended research question.

Literature exists on data analysis for ESM specifically, which helps researchers to consider and perform for instance power size calculations (Lafit et al., Reference Lafit, Adolf, Dejonckheere, Myin-Germeys, Viechtbauer and Ceulemans2020; Scherbaum & Pesner, Reference Scherbaum, Pesner, Humphrey and LeBreton2019) and multilevel statistics which account for the data hierarchy as well as disentangle between- and within-person differences (Bolger & Laurenceau, Reference Bolger and Laurenceau2013; Mehl & Conner, Reference Mehl and Conner2013; Singer, Willett, Willett, & Willett, Reference Singer, Willett, Willett and Willett2003). On the other hand, parallel data sources can require various analytic approaches depending on the type and format of the data. Some of these approaches are already showcased specifically in relation to ESM (Baumeister & Montag, Reference Baumeister and Montag2019; Rehg et al., Reference Rehg, Murphy and Kumar2017). However, in the case of parallel data sources that are not yet referenced, it is advised to consult existing analyses in the specific field of interest and ideally with similar variables.

Broadly, ESM and parallel data studies are typically limited to either describing the association between variables, or developing and/or validating a model that predicts a variable prospectively based on the other variable(s) (Baumeister & Montag, Reference Baumeister and Montag2019; Pencina, Goldstein, & D'Agostino, Reference Pencina, Goldstein and D'Agostino2020; Yarkoni & Westfall, Reference Yarkoni and Westfall2017). To avoid spurious findings in predictive modeling, cross-validation is advised by splitting the collected data in a separate ‘training data set’ and a ‘test data set’, both containing all types of collected data (Kubben, Dumontier, & Dekker, Reference Kubben, Dumontier and Dekker2018).

Likewise, we recommend pre-registering the statistical analysis plan, including the code, prior to data collection. Once the data are collected, we advocate that it should be shared in an appropriate database so that other researchers may replicate the work (Turkyilmaz-van der Velden, Dintzner, & & Teperek, Reference Turkyilmaz-van der Velden, Dintzner and & Teperek2020).

Device selection

Device selection is a vital component of studies measuring parallel data, and an important decision to ensure good compliance and data quality. There are many commercial wearable devices which collect data with minimal intervention, but all of these perform differently and have specific limitations that may change by (patient) populations (Fuller et al., Reference Fuller, Colwell, Low, Orychock, Tobin, Simango and Taylor2020; Lai et al., Reference Lai, Sasaki, Jeng, Cederberg, Bamman and Motl2020; Nelson & Allen, Reference Nelson and Allen2019). It is important to ensure the device collects accurate data that are reliable and valid. Moreover, the device must capture the required data with the right frequency and be validated in the appropriate population. Although they are not available for every application, systemic reviews or comprehensive guidelines exist to help researchers select wearables devices for specific scientific applications (Kunkels, van Roon, Wichers, & Riese, Reference Kunkels, van Roon, Wichers and Riese2021; Nelson et al., Reference Nelson, Low, Jacobson, Areán, Torous and Allen2020).

In addition to data quality, there are other topics deserving attention such as patient comfort and burden, privacy regulations, data security, storage and ownership, and battery life (Rehg et al., Reference Rehg, Murphy and Kumar2017). If real-time evaluation is desired, for example in case of event-dependent ESM assessment, connectivity and data sharing issues should be considered (Cornet & Holden, Reference Cornet and Holden2018; Kohrt et al., Reference Kohrt, Rai, Vilakazi, Thapa, Bhardwaj and van Heerden2019; Trifan, Oliveira, & Oliveira, Reference Trifan, Oliveira and Oliveira2019).

Furthermore, it is essential that devices reliably log the timestamps in universal comparable time. Timestamps simply note when an assessment took place. Some devices need to be synchronized at the beginning of a recording session, and some are subject to drift, which means the timestamp accuracy decreases over time. Overall, inaccuracies within second ranges are negligible since ESM answers do not represent events at a (micro-)second level.

An important distinction in the available devices is whether they provide raw data or proxy data. Data provenance should be well-known prior to data collection, that is the various pre-processing steps taken to transform raw data into meaningful information (Rehg et al., Reference Rehg, Murphy and Kumar2017). Proxy data, often provided by commercial devices, are already processed or summarized into assessment scores, such as activity rates per day or per hour. When proxy data are preferred, or it is not possible to obtain raw data, it is vital that the algorithm used to compute the proxy data is known, or at least well-understood, and most importantly validated (Feehan et al., Reference Feehan, Geldman, Sayre, Park, Ezzat, Yoo and Li2018; Horton, Stergiou, Fung, & Katz, Reference Horton, Stergiou, Fung and Katz2017). Not understanding the essence of the proxy data may heavily affect the validity and interpretation of the obtained findings.

At this stage it is also relevant to consider a data management plan (Wilkinson et al., Reference Wilkinson, Dumontier, Aalbersberg, Appleton, Axton, Baak and Mons2016). That is where data will be stored (short and long term), how will it be preserved, and who will have access to it. A proper data management plan is essential for all studies but especially in this case where there are many data sources with different formats and sizes. Part of this plan should include comprehensive information on what the datasets contain, if the data are raw or have been through any preprocessing steps.

Sampling frequency

Sampling frequency should be defined carefully for the reasons stated above. For variables with a stable volatility, the Nyquist theorem can be used. The Nyquist theorem is commonly used in signal processing, and dictates the sampling frequency to be larger than twice the frequency of the smallest fluctuation in the variable of interest (Bogdan, Reference Bogdan2009). Violating the Nyquist theorem by under-sampling can lead to aliasing; the incorrect extraction of peaks and frequencies from a raw signal. Aliasing is more applicable for high-frequency sampling of parallel data than data collected with ESM.

On the contrary, variables with unstable volatilities are more complicated, such as stress-reactivity, or geolocation. Methodologies from studies or reviews assessing the same variable can often provide evidence regarding specific sampling frequencies. Some examples are heart rate variability (Shaffer & Ginsberg, Reference Shaffer and Ginsberg2017), GPS-based out-of-home activity (Kondo et al., Reference Kondo, Triguero-Mas, Donaire-Gonzalez, Seto, Valentín, Hurst and Nieuwenhuijsen2020; Liao, Song, Robertson, Cox-Martin, & Basen-Engquist, Reference Liao, Song, Robertson, Cox-Martin and Basen-Engquist2020; Zeng, Fraccaro, & Peek, Reference Zeng, Fraccaro, Peek, Riaño, Wilk and ten Teije2019), and accelerometry-based activity monitoring (Kolar et al., Reference Kolar, Neumayr, Roth, Voderholzer, Perthes and Schlegl2020; Niazi et al., Reference Niazi, Yazdansepas, Gay, Maier, Ramaswamy, Rasheed and Buman2017). It is important to stress that under-sampling issues cannot be resolved by simply collecting more data over longer periods of time. Larger datasets which still do not capture the fluctuation of the variable(s) of interest will not lead to meaningful interpretations.

Feature extraction

For a meaningful interpretation of parallel data, information must be extracted from the raw parallel data in a way that it represents the variable of interest. In signal processing terms, the values containing this information are called features. The period of data used to calculate one feature is called the feature window. In some specific cases, the raw data contain the desired information, and no feature extraction is required (e.g. body temperature at specific moments). However, in general, raw parallel data will need to be pre-processed via feature extraction. The type and timescale of features is dependent on the type of data, and the exact variable of interest. For example, proximities to outdoor natural environments can be extracted from GPS-data per 10 min (Kondo et al., Reference Kondo, Triguero-Mas, Donaire-Gonzalez, Seto, Valentín, Hurst and Nieuwenhuijsen2020), while physiological features like heart rate or movement need to be calculated over (milli)seconds. Choosing, or finding, the right feature window size is important since various window sizes may lead to different results (Heijmans, Habets, Kuijf, Kubben, & Herff, Reference Heijmans, Habets, Kuijf, Kubben and Herff2019b). For some parallel data or hypotheses, aggregation of high-frequency features over longer windows might be necessary.

For both data types, it is important to consider these (pre-)processing steps including how to store and annotate the raw data, the features, and preferably the code. For this it is highly recommended to pre-register the study prior to conducting data collection. Publishing detailed scripts of the performed pre-processing steps and analyses, including possible post hoc or additional analysis, will further improve the study's scientific quality and reproducibility. Several resources are available to help with these steps such as the ESM pre-registration template (Kirtley, Lafit, Achterhof, Hiekkaranta, & Myin-Germeys, Reference Kirtley, Lafit, Achterhof, Hiekkaranta and Myin-Germeys2020) and guidance for scientific data care (Goodman et al., Reference Goodman, Pepe, Blocker, Borgman, Cranmer, Crosas and Slavkovic2014).

Temporal feature aggregation

Assuming the parallel data features and ESM answers differ in sampling frequency, we can regard these two time series of data-points both as snapshots of an ongoing, continuous fluctuating process (Fig. 2). To describe an association between them, we need to either up-sample, or down-sample one of them, or both. Up-sampling, via value imputation or extrapolation, tends to generate uncertainty, especially when it needs to be done repeatedly. Down-sampling however, can lead to important information loss. It is common practice to down-sample the parallel data to ESM data sampling frequency.

The most straightforward comparison between ESM and parallel data is to regard each completed ESM questionnaire as a single event, and to compare it with the parallel data collected in the corresponding time window (see Fig. 2). In this case, the higher-frequency parallel data are down-sampled via feature extraction. For this we need to define the duration (how many seconds or minutes), and timing (prior or after the ESM completion) of the window of parallel data corresponding with the ESM variable. The correct duration and timing will depend on the subjective experience that is assessed with the ESM item, the ESM instruction given to the participant, and the formulated hypothesis. The selected parallel data window will then by translated into the variable(s) of interest via the described feature extraction process (Habets et al., Reference Habets, Heijmans, Leentjens, Simons, Temel, Kuijf and Herff2021).

The comparison of ESM and parallel data in which both data types are regarded as ongoing processes over time is based on a different theoretical principle and requires different statistical approaches. This could be especially relevant for hypotheses focusing on continuous processes. Instead of down-sampling the parallel data, up-sampling of the ESM data is required via for example extrapolation (Fig. 2). It is important to carefully consider each statistical method's limitations and potential bias.

Missing data

Similar to the statistical analyses, a detailed description of missing data management is beyond the scope of this work. However, ignoring it entirely would be a significant omission. In general, missing data can be handled in various ways (Little & Rubin, Reference Little, Rubin, Little and Rubin2002b). Broadly, it is important to assess whether the missing data-points are missing at random or may represent completion bias. Especially in the case of ESM, missing data can be caused by (disease) specific reasons and can contain significant information (Cursio, Mermelstein, & Hedeker, Reference Cursio, Mermelstein and Hedeker2019). In addition to asking participants why they missed entries, analyzing both the corresponding parallel data as well as the non-missing ESM data can be considered. Many sources already exist to help researchers handle missing data in a multilevel structured dataset (van Ginkel, Linting, Rippe, & van der Voort, Reference van Ginkel, Linting, Rippe and van der Voort2020), as well as in time-series data (De Waal, Pannekoek, & Scholtus, Reference De Waal, Pannekoek and Scholtus2011). There are also resources to consider regarding for example data imputation techniques (Beard et al., Reference Beard, Marsden, Brown, Tombor, Stapleton, Michie and West2019; van Breda et al., Reference van Breda, Pastor, Hoogendoorn, Ruwaard, Asselbergs and Riper2016) and the possible bias it may introduce (Little & Rubin, Reference Little, Rubin, Little and Rubin2002a).

Practical checklist

To provide researchers with an easy overview of the key elements to consider when conducting a study that combines ESM and parallel data, we provide a detailed checklist to guide in the different stages of the study design (see Table 1). This is intended to be a useful advisory research tool, rather than a rigid guideline.

Table 1.

Considerations for study design checklist

Open-source example

A comprehensive practical example has been drafted to demonstrate how to apply all the different considerations mentioned in this paper (see Table 2). Here, we present how the checklist can be used in practice. For easy access and replicability, we use a publicly available dataset containing ESM and parallel data (i.e. movement data) collected from 20 patients with Parkinson's disease over the course of 14 consecutive days, without restrictions, in daily-life (Habets et al., Reference Habets, Heijmans, Leentjens, Simons, Temel, Kuijf and Herff2021; Habets & Kubben, Reference Habets and Kubben2020). The ESM assessment contains psychological items assessing affect and mood, as well as questions on motor symptoms, physical ability, and contextual questions (Habets et al., Reference Habets, Heijmans, Herff, Simons, Leentjens, Temel and Kubben2020). The movement data consist of raw acceleration (accelerometer) and rotation (gyroscope) time-series data derived from a wrist-worn movement sensor. From these data Parkinson motor symptom variables, such as tremor, were calculated. The database and accompanying Python Notebooks (Perez & Granger, Reference Perez and Granger2007) with example code on data pre-processing and data merging have been published (Habets, 2020; Habets et al., Reference Habets, Heijmans, Leentjens, Simons, Temel, Kuijf and Herff2021) and are available at: https://zenodo.org/record/4734199#.YJAOZRQza3J. For computational details and background on the movement data derived tremor scores we refer to a previous publication (Heijmans et al., Reference Heijmans, Habets, Kuijf, Kubben and Herff2019b). A detailed description of this example can be found in the Supplementary Material.

Table 2.

Practical example of how to apply the considerations of study design checklist

For this example, we are interested in the effect that Parkinsonian tremor severity has on negative affect. Interesting within-subject relationships between mood and tremor severity are showed in a longitudinal n = 1 study (van der Velden, Mulders, Drukker, Kuijf, & Leentjens, Reference van der Velden, Mulders, Drukker, Kuijf and Leentjens2018). This study only used ESM data and did not include any parallel data. With complementary movement data, we want to reproduce and further explore this relationship. Since this data repository is already available, we assess whether it is suitable to answer our question.