Two Sample Hypothesis Tests

Introduction

The two-sample hypothesis testing strategies are based on Algorithm 1 by Gandy & Scott (2021)[gandyandscott2021].

TwoSampleTest

The first basic strategy generates the treatment group

\[\begin{aligned} \theta,\; y_{\text{trtm}} &\sim p\left(\theta, y\right) \\ \theta_{\text{trtm}} &\sim K\left(\theta, \cdot\right), \end{aligned}\]

and the control group as

\[\begin{aligned} \theta_{\text{ctrl}},\; y_{\text{ctrl}} &\sim p\left(\theta, y\right), \\ \end{aligned}\]

where the test compares

\[(\theta_{\text{trtm}}, \, y_{\text{trtm}}) \quad\text{versus}\quad (\theta_{\text{ctrl}}, \, y_{\text{ctrl}}).\]

MCMCTesting.TwoSampleTest — Type
TwoSampleTest(n_samples, n_mcmc_steps; n_control, n_treatment, n_mcmc_thin)

Two-sample hypothesis testing strategy. Algorithm 1 in Gandy & Scott (2021).

Arguments

  • n_samples::Int: Number of samples from the joint p(θ, y) used for the computing the p-values.
  • n_mcmc_steps::Int: Number of times the MCMC kernel is applied to initial sample from the joint. (Increasing this value improves the power of the test.)

Keyword Arguments

  • n_control::Int: Number of pure samples from the joint (control group). (Default: n_samples)
  • n_treatment::Int: Number of samples from the MCMC kernel (treatment group). (Default: n_samples)
  • n_mcmc_thin::Int: Number of thinning applied to the MCMC chain. The effect of this argument is the same as n_mcmc_steps.

Returns

  • pvalues: P-value computed for each dimension of the statistic returned from statistics.

Requirements

This test requires the following functions for model and kernel to be implemented:

  • markovchain_transition
  • sample_joint

Keyword Arguments for Tests

When calling mcmctest or seqmcmctest, this tests has an additional keyword argument:

  • two_sample_test_pvalue: The p-value calculation strategy.

The default strategy is an approximate two-sample Kolmogorov-Smirnov test. Any function returning a single p-value from a two-sample hypothesis test will work. The format is as follows:

two_sample_test_pvalue(x::AbstractVector, y::AbstractVector)::Real
source

TwoSampleGibbsTest

The second strategy performs applies an additional Gibbs sampling step when generating the treatment group as

\[\begin{aligned} \theta,\; y &\sim p\left(\theta, y\right) \\ \theta_{\text{trtm}} &\sim K\left(\theta, \cdot\right), \\ y_{\text{trtm}} &\sim p\left(y \mid \theta_{\text{trtm}}\right) \end{aligned}\]

resulting in the treatment group

\[(\theta_{\text{trtm}}, \, y_{\text{trtm}}).\]

The control group is generated the same as TwoSampleTest.

To use this test, the user needs to implement the following interface for simulating from the conditional likelihood.

MCMCTesting.sample_predictive — Function
sample_predictive(rng, model, θ)

Sample from the predictive distribution of model conditionally on θ

Arguments

  • rng::Random.AbstractRNG: Random number generator.
  • model: Model subject to test.
  • θ: Model parameters to condition on.

Returns

  • y: Data generated from conditionally on θ from p(y|θ)
source
MCMCTesting.TwoSampleGibbsTest — Type
TwoSampleTest(n_samples, n_mcmc_steps; n_control, n_treatment, n_mcmc_thin)

Two-sample hypothesis testing strategy with an additional Gibbs sampling step. Modified version of Algorithm 1 in Gandy & Scott 2021 for increased power.

Arguments

  • n_samples::Int: Number of samples from the joint p(θ, y) used for the computing the p-values.
  • n_mcmc_steps::Int: Number of times the MCMC kernel is applied to initial sample from the joint. (Increasing this value improves the power of the test.)

Keyword Arguments

  • n_control::Int: Number of pure samples from the joint (control group). (Default: n_samples)
  • n_treatment::Int: Number of samples from the MCMC kernel (treatment group). (Default: n_samples)
  • n_mcmc_thin::Int: Number of thinning applied to the MCMC chain. The effect of this argument is the same as n_mcmc_steps.

Returns

  • pvalues: P-value computed for each dimension of the statistic returned from statistics.

Requirements

This test requires the following functions for model and kernel to be implemented:

  • markovchain_transition
  • sample_joint
  • sample_predictive

Keyword Arguments for Tests

When calling mcmctest or seqmcmctest, this tests has an additional keyword argument:

  • two_sample_test_pvalue: The p-value calculation strategy.

The default strategy is an approximate two-sample Kolmogorov-Smirnov test. Any function returning a p-value to two sample groups will work. The format is as follows:

two_sample_test_pvalue(x::AbstractVector, y::AbstractVector)::Real

References

source

References

  • gandyandscott2021Gandy, A., & Scott, J. (2020). Unit testing for MCMC and other Monte Carlo methods. arXiv preprint arXiv:2001.06465.