uncertainty.testing¶

Probability statements about predictions, robust to the number of draws.

The question "how probable is it that an output satisfies a hypothesis H, given everything that is uncertain?" is answered from the draws of a propagation: the estimate is the fraction of the draws which satisfy H, an estimate of the posterior predictive probability P(H) (Gelman et al. 2013). P(H) is a property of the model and of what is uncertain, not of the number of draws: more draws estimate it more precisely and change nothing else.

Anytime-valid inference. More draws can always be simulated, so a statement must stay valid whenever one looks at it and adds draws (optional stopping). A confidence interval of a fixed number of draws, e.g. the Wilson interval, does not: looked at after every batch, it misses the true probability far more often than its level says. The statements use the confidence sequence of the beta-binomial mixture instead (Robbins 1970; Lai 1976; Howard et al. 2021): with the mixture of the likelihood ratios of the first m draws, M_m(p) = B(a + k_m, b + m - k_m) / B(a, b) / (p^k_m (1 - p)^(m - k_m)) with the Jeffreys mixture a = b = 1/2, a nonnegative martingale of mean 1 under the true probability, the set {p: M_m(p) < 1 / alpha} contains the true probability at every m at once with probability at least 1 - alpha (Ville's inequality, alpha = 1 - level). The sequence is intersected over the numbers of draws, so it is nested: it only shrinks with more draws (should the sets of two numbers of draws exclude each other, which happens with probability at most alpha, it stays the last set which is not empty). Its anytime-valid p-value of a probability p0 is min(1, 1 / max_m M_m(p0)) over the numbers of draws whose estimate is on the side of the alternative, valid for the composite null of that side.

Decisions. A statement with a threshold (at_least=p0) is decided by the p-values at alpha after every number of draws: accepted when P(H) >= p0 is shown, rejected when P(H) < p0 is shown, equivalent when P(H) is shown to be within the margin of the threshold (two one-sided tests, Schuirmann 1987), undecided otherwise, with the number of draws which would decide it at the current estimate (n_needed). More draws only make a statement more precise: an undecided one becomes decided, an equivalence becomes the direction which is shown later, and a direction once shown is kept (the direction shown first, at n_decided draws): a decision is never reversed. Every wrong decision of a statement is an event of the martingale of its true probability, so a statement is wrong with probability at most alpha. The margin is the difference of a probability which does not matter, 0.05 by default; it makes every statement decidable with a bounded number of draws, also one whose probability is the threshold.

Several statements. A TestReport decides m statements at the level of Bonferroni, alpha / m each (and gives their simultaneous intervals at that level), so the probability that any of them is wrong is at most alpha (the mean alpha of statements of other levels, TestReport.family_error), whatever the dependence between them, which share their draws: every statement with a threshold has a true null (both directions at the threshold, one direction and the equivalence elsewhere), so a correct decision of one statement must not relax the level of the others. The decisions of a report are the decisions of single statements at a fixed level, so they are never reversed by more draws either; the conservative level costs draws only, about log(m) times the information per statement, which more draws provide.

The draws which are NaN (failed points) are excluded and counted, infinite values are values. The validity assumes independent draws, which design="random" of a propagation gives; the counts of quasi-random draws fluctuate less, for which the sequence is conservative in practice but not by a theorem. A probability statement is about the predictive distribution, i.e. about the uncertainty of the inputs, not a test of observed data: the draws are not subjects of a study.

Robbins H (1970). Statistical methods related to the law of the iterated logarithm. Ann Math Stat 41:1397-1409. Lai TL (1976). On confidence sequences. Ann Stat 4:265-280. Howard SR, Ramdas A, McAuliffe J, Sekhon J (2021). Time-uniform, nonparametric, nonasymptotic confidence sequences. Ann Stat 49:1055-1080. Schuirmann DJ (1987). A comparison of the two one-sided tests procedure and the power approach for assessing the equivalence of average bioavailability. J Pharmacokinet Biopharm 15:657-680. Gelman A, Carlin JB, Stern HS, Dunson DB, Vehtari A, Rubin DB (2013). Bayesian Data Analysis, 3^rd ed. CRC Press.

TestResult(
    hypothesis,
    estimate,
    low,
    high,
    n,
    n_failed,
    k,
    threshold,
    decision,
    n_needed=None,
    n_decided=None,
    level=0.95,
    margin=None,
    p_greater=nan,
    p_less=nan,
    p_equivalent=nan,
    satisfied=None,
    method="fraction of the draws, beta-binomial confidence sequence",
    references=REFERENCES,
)

p_value()

at_level(level)

Hypothesis(
    output,
    greater=None,
    less=None,
    between=None,
    at_least=None,
    scenario=None,
    contrast=None,
    kind="ratio",
    margin=MARGIN,
    level=0.95,
)

evaluate(values, level=None)

to_dict()

TestReport(tests=list(), adjust='bonferroni')

m

family_error

adjusted()

table()

confidence_sequence(satisfied, level=0.95)

probability(
    values,
    *,
    greater=None,
    less=None,
    between=None,
    at_least=None,
    margin=MARGIN,
    level=0.95,
)

family_level(level, m, adjust='bonferroni')