Robins and Wasserman Respond to a Nobel Prize Winner Continued: A Counterexample to Bayesian Inference?

This is a response to Chris Sims’ comments on our previous blog post. Because of the length of our response, we are making this a new post rather than putting it in the comments of the last post.

Recall that we observe \({n}\) iid observations \({O=\left( X,R,RY\right)}\), where \({Y}\) and \({R}\) are Bernoulli and independent given \({X}\).
Define \({\theta \left( X\right) \equiv E \left[ Y|X\right]}\) and \({\pi \left( X\right) \equiv E\left[ R|X\right]}\). We assume that \({\pi \left( \cdot \right) }\) is a known function. Also the marginal
density \({p\left( x\right)}\) of \({X=\left( X_{1},...,X_{d}\right)}\) (with \({d=100,000}\)) is known and uniform on the unit cube in \({R^{d}}\). Our goal is estimation of

$$\psi \equiv E\left[ Y\right] =E\left\{ E\left[ Y|X\right] \right\} = E\left\{E\left[ Y|X,R=1\right] \right\} =\int_{[0,1]^d} \theta \left( x\right) dx.$$

The likelihood

$$\prod_{i=1}^{n}p(X_{i})p(R_{i}|X_{i})p(Y_{i}|X_{i})^{R_{i}}=\left\{ \prod_{i}\pi (X_{i})^{R_{i}}(1-\pi (X_{i}))^{1-R_{i}}\right\} \left\{ \prod_{i}\,\theta (X_{i})^{Y_{i}R_{i}}(1-\theta (X_{i}))^{(1-Y_{i})R_{i}}\right\} \$$

factors into two parts – the first depending on \({\pi \left( \cdot \right)}\) and the second on \({\theta \left( \cdot \right)}\).

1. Selection Bias

This is a point of agreement. There is selection bias if and only if

$$Cov\left\{ \theta \left(X\right) ,\pi \left( X\right) \right\}=0.$$

Note that

$$E\left[ Y\right] =E\left[ Y|R=1\right] - \frac{Cov\left\{ \theta \left(X\right) ,\pi \left( X\right) \right\} }{E[R]}.$$

Hence, if \({Cov\left\{ \theta \left(X\right) ,\pi \left( X\right) \right\}=0}\) then
the sample average of \({Y}\) in the subset whose \({Y}\) is observed (R=1) is unbiased for

$$\psi \equiv E\left[ Y\right] =\int_{[0,1]^d}\ \theta \left( x\right) dx.$$

In this case, inference is easy for the Bayesian and the frequentist and there is no issue. So we all agree that the interesting case is where there is selection bias,
that is, where \({Cov\left\{ \theta \left(X\right) ,\pi \left( X\right) \right\} \neq 0.}\)

2. Posterior Dependence on \({\pi}\)

If the prior \({W}\) on the functions \({\pi(\cdot)}\) and \({\theta(\cdot)}\) is such that \({W(\pi,\theta)= W(\pi)W(\theta)}\) then the posterior does not depend on \({\pi}\) and the posterior for \({\psi}\) will not concentrate around the true value of \({\psi}\). Again, we believe we all agree on this point.

We note that no one, Bayesian or frequentist, has ever proposed using an estimator that does not depend on \({\pi \left( \cdot \right) }\) in the selection bias case, i.e when \({Cov\left\{ \theta \left( X\right) ,\pi\left( X\right) \right\} }\) is non-zero. (See addendum for more on this point.)

3. Prior Independence Versus Selection Bias

Reading Chris’ comments, the reader might get the impression that prior independence rules out selection bias, i.e. that is,

$$W(\pi,\theta) =W(\pi)W(\theta)\ \ \ \ {\rm implies \ that\ }\ \ \ Cov\left\{ \theta \left(X\right) ,\pi \left( X\right) \right\}=0.$$

Therefore, one might conclude that if we want to discuss the interesting case where there is selection bias, then we cannot have \({W(\pi,\theta) =W(\pi)W(\theta)}\).

But this is incorrect. \({W(\pi,\theta) =W(\pi)W(\theta)}\) does not imply that \({Cov\left\{ \theta \left(X\right) ,\pi \left( X\right) \right\}=0.}\) To see this, consider the following example.

Suppose that \({X}\) is one dimensional and a Bayesian’s prior \({W}\) for \({\left( \theta \left( \cdot \right) ,\pi \left(\cdot \right) \right)}\) depends only on the two parameters \({\left( \alpha_{\theta },\alpha _{\pi }\right) }\) as follows:

$$\theta \left( x\right) =\alpha _{\theta }x,\ \ \ \pi \left( x\right) =\alpha _{\pi}x+1/10 \ \ \ \text{with}\ \alpha _{\theta }\text{ and }\alpha _{\pi }\text{ a\ priori\ independent, }$$

where \({\alpha _{\theta }}\) is uniform on \({\left( 0,1\right)}\) and \({\alpha_{\pi }}\) is uniform on \((0,9/10)\).
Then, clearly \({\theta\left( \cdot \right)}\) and \({\pi \left( \cdot \right)}\) are independent under \({W}\). However, recalling \({X}\) is uniform so \({p\left( x\right) \equiv 1,}\) we have that for for any fixed \({\left( \alpha _{\theta },\alpha _{\pi }\right)}\),

$$Cov\left\{ \theta \left( X\right) ,\pi \left( X\right) \right\} =\int_{0}^{1}\theta \left( x\right) \pi \left( x\right) dx-\int_{0}^{1}\ \pi \left( x\right) dx\int_{0}^{1}\theta \left( x\right) dx \\ =\alpha _{\theta }\alpha _{\pi }\left( \int_{0}^{1}x^{2}dx-\left\{ \int_{0}^{1}xdx\right\} ^{2}\right) =\alpha _{\theta }\alpha _{\pi }/12 .$$

Hence

$$W({\rm there\ exists\ selection\ bias})=W\Biggl( Cov\left\{ \theta \left( X\right) ,\pi \left( X\right) \right\} >0 \Biggr) =1$$

since \({\alpha _{\theta }}\) and \({\alpha_{\pi }}\) are both positive with \({W-}\)probability 1.

4. Other Justifications For Prior Dependence?

Since prior independence of \({\pi}\) and \({\theta}\) does not imply “no selection bias,” one might instead argue that it is practically unrealistic to have \({W(\theta,\pi)=W(\theta)W(\pi)}\). But we now show that it is realistic.

Suppose a new HMO needs to estimate the fraction \({\psi }\) of its patient population that will have a MI \({(Y)}\) in the next year, so as to determine the number of cardiac unit beds needed. Each HMO member has had 300 potential risk factors \({X=(X_{1},...,X_{300})}\) measured: age, weight height, blood pressure, multiple tests of liver, renal, pulmonary, and cardiac function, good and bad cholesterol, packs per day smoked, years smoked, etc. (We will get to 100,000 once routine genomic testing becomes feasible). A general epidemiologist had earlier studied risk factors for MI
by following 5000 of the 50,000 HMO members for a year. Because MI is a rare event, he oversampled subjects whose \({X}\), in his opinion, indicated a
smaller probability \({\theta \left( X\right) }\) of an MI (\({Y=1)}\). Hence the
sampling fraction \({\pi \left( X\right) =P\left( R=1|X\right) }\) was a known, but complex function chosen so as to try to make \({\theta \left( X\right) }\) and \({\pi \left( X\right) }\) negatively correlated.

The world’s leading heart expert, our Bayesian, was hired to estimate \({\psi =\int \theta \left( x\right) p\left( x\right) dx}\) based on distribution \({ p\left( x\right) }\) of \({X}\) in HMO members and the data \({\left(X_{i},R_{i},R_{i}Y_{i}\right) ,i=\left( 1,...,5000\right) }\) from the study.
As world’s expert, his beliefs about the risk function \({\theta \left( \cdot \right) }\) would not change upon learning \({\pi \left( \cdot \right) ,}\) as \({ \pi \left( \cdot \right) }\) only reflects a nonexpert’s beliefs. Hence \({ \theta \left( \cdot \right) }\) and \({\pi \left( \cdot \right) }\) are a priori independent. Nonetheless, knowing that the epidemiologist had carefully read the expert literature on risk factors for MI, he also believes with high probability that epidemiologist succeeded in having the random variables \({ \theta \left( X\right) }\) and \({\pi \left( X\right) }\) be negatively correlated.

What’s more, Robins and Ritov (1997) showed that, if before seeing the data, any Bayesian, cardiac expert or not, thoroughly queries the epidemiologist
(who selected \({\pi \left( \cdot \right) }\)) about the epidemiologist’s reasoned opinions concerning \({\theta (\cdot )}\) (but not about \({\pi (\cdot )}\) ), the Bayesian will then have independent priors. The idea is that once you are satisfied that you have learned from the epidemiologist all he knows about \({\theta (\cdot )}\) that you did not, you will have an updated prior for
\({\theta \left( \cdot \right) }\). Your prior for \({\theta \left( \cdot \right) \ (}\)now updated) cannot then change if you subsequently are told \({\pi \left( \cdot \right) .}\) Hence, we could take as many Bayesians as you please and arrange it so all had \({\theta \left( \cdot \right) }\) and \({\pi \left( \cdot \right) }\) apriori independent. This last argument is quite general, applying to many settings.

5. Alternative Interpretation

An alternative reading of Chris’s third response and his subsequent post is that, rather than placing a joint prior \({W}\) over the functions
\({\theta\left( \cdot \right) =\left\{ \theta \left( x\right) ;x\in \left[ 0,1\right]^{d}\right\} }\) and \({\pi \left( \cdot \right) =\left\{ \pi \left( x\right);x\in \left[ 0,1\right] ^{d}\right\}}\) as above,
his prior is placed over the joint distribution of the random variables \({\theta \left( X\right) }\) and \({\pi \left( X\right)}\).
If so, he is then correct that making \({\theta \left(X\right) }\) and \({\pi \left( X\right) }\) independent with prior probability one
also implies \({Cov\left\{ \theta \left( X\ \right) ,\pi \left( X\ \right)\right\} =0}\) and thus no selection bias.

However, it appears that from this, he concludes that selection bias, in itself, licenses the dependence of his posterior on \({\pi \left( \cdot \right)}\).
This is incorrect. As noted above, it is prior dependence of \({\theta \left( \cdot \right) }\) and \({\pi\left( \cdot \right) }\) that licenses posterior dependence on \({\pi \left(\cdot \right)}\) – not prior dependence of \({\theta \left( X\right) }\) and
\({\pi\left( X\right)}\). Were he correct, our Bayesian cardiac expert’s prior on \({\theta \left( \cdot \right)}\) could have changed upon learning the epidemiologists \({\pi \left( \cdot \right)}\).

6. What If We Do Use a Prior That Depends on \({\pi}\)?

In the above scenario, \({W(\theta)}\) should ot depend on \({\pi}\). But suppose, for whatever reason, one insists on letting \({W(\theta)}\) depend on \({\pi}\).

That still does not mean the posterior will concentrate. Having an estimator that depends on \({\pi}\) is necessary, but not sufficient, to get consistency and fast rates. It is not enough to use a prior \({W(\theta)}\) that is a function of \({\pi}\). The prior still has to be carefully engineered to ensure that the posterior for \({\psi}\) will concentrate around the truth.

Chris hints that he can construct such a prior but does not provide an explicit algorithm nor an argument as to why the estimator would be expected to be locally semiparametric efficient. However, it is simple to construct a \({n^{1/2}}\)-consistent
locally semiparametric efficient Bayes estimator \({\hat{\psi}_{Bayes}}\) as follows.

We tentatively model \({\theta(x) =P(Y=1|X=x)}\) as a finite dimensional parametric function \({b\left( x;\eta_{1},\ldots ,\eta_{k},\omega \right) }\)
with either a smooth or noninformative prior on the parameters \({\left( \eta_{1},\ldots ,\eta_{k},\omega \right)}\), where we take

$$b\left( x;\eta_{1},\ldots ,\eta _{k},\omega \right) = \mathrm{expit}\left( \sum_{m=1}^{k}\eta_{m}\phi _{m}(x)+\frac{\omega }{\pi (x)}\right) ,$$

\({\mathrm{expit}(a)=e^{a}/(1+e^{a})}\), and the \({\phi _{m}\left( x\right)}\) are basis functions. Then the posterior mean
\({\hat{\psi}_{\rm Bayes}}\) of \({\psi =\int \theta \left( x\right) dx}\) will have the same asymptotic distribution as the locally semiparametric efficient regression estimator of Scharfstein et al. (1999) described in our original post. Note that the estimator is \({n^{1/2}}\) consistent, even if the model \({\theta(x) =P(Y=1|X=x)=b\left( x;\eta_{1},\ldots ,\eta_{k},\omega \right)}\) is wrong.

Of course, this estimator is a clear case of frequentist pursuit Bayes.

7. Conclusion

Here are the main points:

  1. If \({W(\theta,\pi) = W(\theta)W(\pi)}\) then the posterior will not concentrate.
    Thus, if a Bayesian wants the posterior for \({\psi}\) to concentrate around the true value,
    he must justify having a prior \({W(\theta)}\) that is a function of \({\pi}\).
  2. \({W(\theta,\pi) = W(\theta)W(\pi)}\) does not imply an absence of selection bias.
    Therefore, an argument of the form: “we want selection bias so we cannot have prior independence” fails.
  3. One can try to argue that prior independence is unrealistic. But as we have shown, this is not the case.
  4. But, if after all this, we do insist on letting \({W(\theta)}\) depend on \({\pi}\),
    it is still not enough. Dependence on \({\pi}\) is necessary but not sufficient.

We conclude Bayes fails in our example unless one uses a special prior designed just to mimic the frequentist estimator.

8. Addendum: What happens If The Estimator Does Note Depend on \({\pi}\)?

The theorem of Robins and Ritov, quoted in our initial post, says that no uniformly consistent estimator that does not depend on \({\pi \left( \cdot \right) }\) can exist in the model \({\mathcal{P}}\) which contains all measurable \({\pi \left( x\right) }\) and \({\theta \left( x\right) }\) subject
to \({\pi \left( X\right) >\delta >0}\) with probability 1. Take \({\delta =1/8}\) for
concreteness. In fact, even when we assume \({\theta \left( \cdot \right) }\)
and \({\pi \left( \cdot \right) }\) are quite smooth, there will be little
improvement in performance.

Given \({X}\) has 100,000 dimensions, we can ask how many derivatives \({\beta _{\theta }}\) and \({\beta _{\pi }}\) must \({\theta \left( \cdot \right) }\)
and \({\pi \left( \cdot \right) }\) have so that it is possible to construct an estimator of \({\psi }\), not depending on \({\pi \left( \cdot \right) ,}\) that
converges at rate \({n^{-\frac{1}{2}}}\)uniformly to \({\psi }\) over a submodel \({ \mathcal{P}_{smooth}}\). Robins et al. (2008) show that it is necessary and sufficient that \({\beta _{\theta }}\)+\({\beta _{\pi }=50,000}\) and provide an
explicit estimator. More generally, if \({\beta _{\theta }}\) +\({\beta _{\pi }=s}\) derivatives with \({0<s<50,000}\) , the optimal rate is \({n^{- \frac{s/50,000}{\left( 1+s/50,000\right) }}}\) which is approximately \({ n^{-s/50,000}}\) when \({s}\) is small compared to \({50,000.}\) An explicit estimator
is constructed in Robins et al (2008) ; Robins et al (2009) prove that the rate cannot be improved on. Given these asymptotic mathematical results, we
doubt any reader can exhibit an estimator, not depending on \({\pi \left(\cdot \right) ,}\) that will have reasonable finite sample performance under
model \({\mathcal{P}}\) or even \({\mathcal{P}_{smooth}}\) with, say, \({s=25,000}\)
and a sample size of 5,000. By reasonable finite sample performance, we mean an interval estimator that will cover the true \({\psi }\) at least 95% of the time and that has average length less than or equal to intervals estimators
centered on the \({n^{1/2}-consistent}\) improved HT estimators. Nonetheless, we
await any candidate estimators, accompanied by at least some simulation
evidence backing up your claim.

9. References

  1. Robins JM, Tchetgen E, Li L, van der Vaart A. (2009).
    Semiparametric Minimax Rates. Electron. J. Statist. Volume 3 (2009),
    1305-1321.
  2. Robins JM, Li L, Tchetgen E, van der Vaart A. (2008). Higher order influence
    functions and minimax estimation of nonlinear functionals. Probability and
    Statistics: Essays in Honor of David A. Freedman 2:335-421
  3. Robins JM, Ritov Y. (1997). Toward a curse of dimensionality appropriate
    (CODA) asymptotic theory for semi-parametric models. Statistics in Medicine,
    16:285-319.

This entry was written by , posted on September 2, 2012 at 5:28 pm, filed under Uncategorized. Bookmark the permalink. Follow any comments here with the RSS feed for this post. Both comments and trackbacks are currently closed.