MINDSET SHIFT USING SIDE-BY-SIDE EXAMPLES AND A FOUR-STEP MODELING APPROACH

Students who have some statistical inference background but are learning Bayesian statistics for the first time often struggle to understand the difference between estimating an unknown parameter in a non-Bayesian manner and updating the posterior distribution of the “parameter” in a Bayesian framework.

The confusion is not difficult to clarify, but for that reason it can be overlooked by instructors. It is common to read phrases like “the estimator of the parameter ( $ \boldsymbol{\unicode{x03B8}} $ )” and “the true value of $ \boldsymbol{\unicode{x03B8}} $ .” Many researchers use a half-Bayesian approach in which they adopt the Bayesian computation apparatus but assume that a single fixed value of the parameters generated the data. If students are not familiar with the procedures, this half-Bayesian approach can contribute to their confusion. Using this approach, it is correct to state that the posterior distribution approaches that fixed value of $ \boldsymbol{\unicode{x03B8}} $ when the size of the data increases, but it is conceptually incorrect using the Bayesian framework to state the inferential question as learning the “true” value of $ \boldsymbol{\unicode{x03B8}} $ because $ \boldsymbol{\unicode{x03B8}} $ is a random variable—even when we assume that there is an underlying true fixed value of the parameter behind the data-generating process. More precisely, what we learn in Bayesian approaches is not the “true” value of $ \boldsymbol{\unicode{x03B8}} $ but rather its distribution—or the distribution of our beliefs about possible values of $ \boldsymbol{\unicode{x03B8}} $ —after we take into account the data. Arguably, there is nothing wrong with a half-Bayesian approach; however, it can be confusing for a student learning the subject for the first time.

To facilitate a mindset shift from a non-Bayesian to a Bayesian paradigm, instructors can combine a four-step modeling approach when developing examples and periodically provide side-by-side solutions using the frequentist—or perhaps the “likelihoodist”—approach and the Bayesian paradigm.

The four-step approach follows four modeling steps that the instructor can illustrate explicitly—and with students’ participation—in every example of application of Bayesian models. The four steps are as follows:

1. 1. Select the data model.

2. 2. Select the prior model.

3. 3. Derive the posterior.

4. 4. Compute the posterior quantities of interest.

There are many benefits to explicitly following these four steps in every example and emphasizing each one as the examples progress. One benefit is that students do not see the analysis as a “bag of models” that we merely mechanically plug into the data but rather as a logical process in which the analyst makes a series of modeling decisions in each step. Instructors can use this to emphasize that the quality and adequacy of those modeling decisions must be assessed and to demonstrate that such an assessment is an essential part of the modeling process. Students will better understand the importance of grasping the logic and steps behind Bayesian modeling and not that they simply must learn an assortment of seemingly unrelated models. Another benefit is that the four steps empower students by providing a structure for modeling substantive problems and they can follow these steps themselves.

I use this four-step procedure as a guideline and motivation for all other topics I cover in introductory courses, and I explicitly restate and follow the steps in all examples. In terms of motivation to introduce other topics, step 2 leads naturally to issues of prior selection (e.g., Jeffrey’s prior, conjugate priors, objective Bayes, and sensitivity) and how to handle them. Some challenges emerge for students when selecting the prior in step 2, which is discussed in the next section. Steps 3 and 4 lead naturally to issues of computing integrals when there is no analytical solution, which helps to motivate Monte Carlo methods, MCMC, and the issues that emerge with those procedures, including MCMC implementation, convergence diagnostics, and stopping rules.

I explicitly construct every example following these four steps and I often present Bayesian and non-Bayesian solutions side by side to help students recognize differences between the paradigms. Here is an example of a side-by-side comparison that instructors can adapt and use in their own course. Suppose we installed an alarm system in our house and we go away on a trip. We set up the alarm to ring on our cell phone, and we can remotely activate an additional anti-burglary tech system if we believe a burglary is in progress. The anti-burglary system is expensive to recharge after it has been activated, so we should use it only if we are highly confident that there is a burglar. These are our indicators:

$$ {\displaystyle \begin{array}{l}X\hskip1.5pt =\hskip1.5pt \left\{\begin{array}{ll}1,& \hskip1.5pt \mathrm{alarm}\ \mathrm{is}\ \mathrm{a}\mathrm{ctive}\\ {}0,& \mathrm{a}\mathrm{larm}\ \mathrm{is}\ \mathrm{no}\mathrm{t}\ \mathrm{a}\mathrm{ctive}\end{array}\right.\\ {}\boldsymbol{\unicode{x03B8}} \hskip1.5pt =\hskip1.5pt \left\{\begin{array}{ll}1,& \mathrm{there}\ \mathrm{is}\ \mathrm{a}\hskip0.24em \mathrm{burglary}\ \mathrm{happening}\\ {}0,& \mathrm{there}\ \mathrm{is}\ \mathrm{no}\hskip0.24em \mathrm{burglary}\ \mathrm{happening}\end{array}\right.\end{array}} $$

The alarm manual contains information summarized in table 1.

Table 1

Anti-burglary Example

If we hear the alarm ( $ X\hskip1.5pt =\hskip1.5pt 1 $ ), is there a burglary happening ( $ \boldsymbol{\unicode{x03B8}} \hskip1.5pt =\hskip1.5pt 1 $ )? A non-Bayesian way to answer this question is to use the likelihood function to estimate $ \boldsymbol{\unicode{x03B8}} $ . We choose the value of $ \boldsymbol{\unicode{x03B8}} $ that maximizes $ p\left(X\hskip1.5pt =\hskip1.5pt 1\mid \boldsymbol{\unicode{x03B8}} \right) $ . Comparing $ p\left(X\hskip1.5pt =\hskip1.5pt 1\mid \boldsymbol{\unicode{x03B8}} \hskip1.5pt =\hskip1.5pt 0\right)\hskip1.5pt =\hskip1.5pt 0.05 $ to $ p\left(x\hskip1.5pt =\hskip1.5pt 1\;|\boldsymbol{\unicode{x03B8}} \hskip1.5pt =\hskip1.5pt 1\right)\hskip1.5pt =\hskip1.5pt 0.99 $ , we choose $ \boldsymbol{\unicode{x03B8}} \hskip1.5pt =\hskip1.5pt 1 $ . That is, we choose the $ \boldsymbol{\unicode{x03B8}} $ that makes the data more likely. This is the estimation step. Regardless of anything else that may be occurring, if we hear the alarm, we should adopt $ \boldsymbol{\unicode{x03B8}} \hskip1.5pt =\hskip1.5pt 1 $ . There is no way to solve this problem using a frequentist approach, but we can decide to reject (or not) that estimated value using a likelihood method to estimate $ \boldsymbol{\unicode{x03B8}} $ . I use these points to briefly discuss inference and compare the paradigms.

If we select $ \boldsymbol{\unicode{x03B8}} \hskip1.5pt =\hskip1.5pt 1 $ , we are still not completely certain if there is a burglary happening because there is a small chance that it is a false alarm. What is the chance that there is a burglary happening if we hear the alarm? Formally restated, what is $ p\left(\boldsymbol{\unicode{x03B8}} \mid x\hskip1.5pt =\hskip1.5pt 1\right) $ ? Only the Bayesian paradigm can answer that question, which is as follows:

$$ p\left(\boldsymbol{\unicode{x03B8}} \mid X\hskip1.5pt =\hskip1.5pt 1\right)\hskip1.5pt =\hskip1.5pt \frac{p\left(X\hskip1.5pt =\hskip1.5pt 1\mid \boldsymbol{\unicode{x03B8}} \right)p\left(\boldsymbol{\unicode{x03B8}} \right)}{\sum_{\boldsymbol{\unicode{x03B8}} \hskip1.5pt =\hskip1.5pt }^1p\left(X\hskip1.5pt =\hskip1.5pt 1\mid \boldsymbol{\unicode{x03B8}} \right)p\left(\boldsymbol{\unicode{x03B8}} \right)} $$

The only unknown is $ p\left(\boldsymbol{\unicode{x03B8}} \right) $ , the prior. I use two possible choices for that quantity. In one case, we have no clue about the overall probability of burglary ( $ p\left(\boldsymbol{\unicode{x03B8}} \hskip1.5pt =\hskip1.5pt 1\right)\hskip1.5pt =\hskip1.5pt p\left(\boldsymbol{\unicode{x03B8}} \hskip1.5pt =\hskip1.5pt 0\right)\hskip1.5pt =\hskip1.5pt 0.5 $ ). In the other scenario, we know that the neighborhood is extremely safe ( $ p\left(\boldsymbol{\unicode{x03B8}} \hskip1.5pt =\hskip1.5pt 0\right)\hskip1.5pt =\hskip1.5pt 0.9 $ ). In this case, we reason about the probability of $ \boldsymbol{\unicode{x03B8}} $ —that is, the fixed observed data $ X\hskip1.5pt =\hskip1.5pt 1 $ changed our belief about $ \boldsymbol{\unicode{x03B8}} $ .

We can work with other examples to emphasize the differences between confidence intervals and posterior probabilities. Here is one possible example: Assume that there is a probability $ \boldsymbol{\unicode{x03B8}} $ that a random person will vote for a party $ A $ . We take a random sample and 40 of 100 people state that they will vote for that party.

I discuss with students the selection of the data model that results in the choice of a binomial distribution (i.e., step 1 of the four-step procedure) and a uniform distribution for the prior (i.e., step 2) and show (i.e., step 3) that:

$$ p\left(\boldsymbol{\unicode{x03B8}} \mid x\right)\hskip1.5pt \propto \hskip1.5pt p\left(x\mid \boldsymbol{\unicode{x03B8}} \right)p\left(\boldsymbol{\unicode{x03B8}} \right)\hskip1.5pt =\hskip1.5pt {\boldsymbol{\unicode{x03B8}}}^x{\left(1-\boldsymbol{\unicode{x03B8}} \right)}^{n-x}\hskip1.5pt =\hskip1.5pt {\boldsymbol{\unicode{x03B8}}}^{40}{\left(1-\boldsymbol{\unicode{x03B8}} \right)}^{60} $$

We can use a simplified version of non-Bayesian inference side by side in this example. We can compute the point estimate and the confidence interval for the problem. Thus, we can show that in frequentist approaches, we compute the point estimate; however, we must ask what would happen with that estimator if we had observed a different sample. It helps to show that we must rely on the sampling distribution of the estimator—usually based on asymptotic results—think about repeated sampling, and construct confidence intervals based on the variance of the estimator. For this example, it is as follows:

$$ \begin{array}{c}\hat{\boldsymbol{\unicode{x03B8}}}\hskip1.5pt =\hskip1.5pt \frac{40}{100}\hskip1em ;\hskip1em CI\left(\hat{\boldsymbol{\unicode{x03B8}}}\right)\hskip1.5pt =\hskip1.5pt \left[0.4-{t}_{\alpha}\sqrt[]{\unicode{x1D54D}\hskip1.5pt \mathrm{ar}\left[\hat{\boldsymbol{\unicode{x03B8}}}\right]},0.4+{t}_{\alpha}\sqrt[]{\unicode{x1D54D}\hskip1.5pt \mathrm{ar}\left[\hat{\boldsymbol{\unicode{x03B8}}}\right]}\right]\\ {}\hskip-23.50pt =\hskip1.5pt \left[0.303,0.496\right]\end{array} $$

I use simplified versions of non-Bayesian procedures because teaching advanced, non-Bayesian procedures is not the goal of the course. Readers can find good discussions and additional examples to compare different inferential paradigms in Christensen (Reference Christensen2005). The number of details to include depends on the students’ background. The idea is to leverage the understanding of Bayesian inferential reasoning using students’ previous backgrounds and to avoid creating confusion between the paradigms.

During the course, I incrementally construct table 2 until it is fully filled, which naturally occurs as we approach the final lectures. The table is stylized to emphasize key differences, and it works well to help with the mindset shift between Bayesian and non-Bayesian paradigms. After constructing intentionally simple examples of Bayesian inference—such as those presented previously—to compare against non-Bayesian cases, I present the table again and derive the non-Bayesian and Bayesian inference, side by side. Although I use the four-step procedure to develop each example, I compare Bayesian and non-Bayesian approaches in only some of them.

Table 2

Comparing Bayesian and Non-Bayesian Approaches

In summary, in addition to helping with the mindset shift and to internalizing and understanding the Bayesian framework, the side-by-side examples and the four-step approach serve three other purposes: (1) to ground the comparison between Bayesian and other approaches and make their differences more apparent; (2) to describe a simple step-by-step procedure to derive Bayesian models; and (3) to motivate the discussion of other topics, such as MCMC and sensitivity analysis.

A BAG OF PRIOR DISTRIBUTIONS

Another challenge that can emerge when teaching Bayesian statistics is related to the process of selecting prior distributions (i.e., step 2 in the four-step procedure). Students often lack familiarity with both the process of selecting a distribution to model the data and the properties of various distributions often used in Bayesian analysis to model the prior. These properties include the support, the functional form of the density, the kernel, the meaning of the parameters, and the shape of the distribution as a function of the parameters. It is likely that many students have never learned about Beta, Gamma, and other common distributions. This issue is easy to address and, for that reason, it also often is overlooked by instructors—which can lead to confusion later when introducing more complex models.

Arguably, the support of a distribution and how its shape depends on the parameters are the more important aspects for practical applications. Therefore, it is worth covering those topics using examples, which can be constructed using the four-step procedure described in the previous section.

It also is useful to provide a table such as table 3 that students can reference quickly when trying to find a prior model. Following the previous example with the proportions, we can present a table with prior options and compare their properties. This develops the students’ intuition of the process behind prior selection, introduces some distributions, emphasizes the distributions’ support, and teaches how to think about their parameters and how they affect the shape of the distribution. Which prior should be selected from the previous example about the likelihood of a random person voting for party A? Some of the options are as follows:

Table 3

Example of a Table for Selection of Prior Distributions

This exercise of prior selection is repeated in each example I present and allows students to choose among the options of priors I provide, which usually is a superset with options the support of which is outside of the values that the parameter $ \boldsymbol{\unicode{x03B8}} $ can take. The exercise can be supplemented with animations to show the effect of the parameter values on the shape of the prior; subsequently, similar animations can be used to show their effect on the posterior (i.e., sensitivity).