Simulation-Based Calibration (SBC)

Repeatedly draws a "true" parameter from the prior, simulates data, and ranks the true parameter within posterior samples conditioned on that data. If the posterior is well calibrated, the ranks are uniformly distributed.

Usage

sbc(
  fit,
  simulator,
  prior = fit$prior,
  n_sbc = 200L,
  n_posterior_samples = 1000L,
  sim_args = list(),
  seed = NULL,
  ...
)

Arguments

fit: An nsbi_npe fit from npe(), an nsbi_nle fit from nle(), or an nsbi_nre fit from nre(). With an NLE or NRE fit every trial is a separate MCMC run, so start with a small n_sbc and raise it once the cost is known.
simulator: The simulator used for inference; called once per trial (see nsbi_simulator).
prior: The prior to draw the true parameters from (defaults to fit$prior). SBC is a test of the posterior against the prior it was trained on, so the default is the only choice that answers "is this fit calibrated". Overriding it changes the question to how the fit behaves on parameters drawn from somewhere else, which is a reasonable local check but is no longer SBC. It must cover the same parameters as the fit.
n_sbc: Number of SBC trials (fresh (theta, x) pairs).
n_posterior_samples: Posterior draws per trial (rank resolution).
sim_args: Named list of extra arguments passed to every simulator call; see nsbi_simulator.
seed: Optional seed.
...: Passed to posterior(), which is how the MCMC controls (n_chains, warmup, thin, sampler) reach an NLE or NRE fit.

Value

An object of class nsbi_sbc with the rank matrix and a per-parameter uniformity test.

Details

A trial whose simulation returns non-finite output is dropped, which lowers the effective n_sbc. A trial whose posterior comes back with fewer than n_posterior_samples draws, which happens when a bounded prior and a leaky estimator defeat rejection sampling, is an error: ranks are binned against n_posterior_samples, so a short draw would be scored on a scale it was never drawn on and would read as miscalibration.

The n_sbc simulations run across future workers when a plan is set (see nsbi_parallel); the ranking loop that follows calls the trained network and always runs locally.