Interpreting Results

GeoSC produces a suite of statistical indicators to help determine the validity and magnitude of an experiment’s impact.

Main Statistical Outputs

  • status: Result contract status. ok means the core ATT, p-value, and uncertainty fields are available; partial means the estimate ran but one or more uncertainty fields are unavailable; failed means required estimator outputs are missing and the effect should not be interpreted.
  • att (Average Treatment Effect on the Treated): The average difference between the observed outcome and the synthetic counterfactual during the post-treatment period.
  • p_value: A donor-pool placebo p-value: the share of placebo effects at least as extreme as the observed ATT under the implemented SparseSC placebo comparison.
  • ci_lower / ci_upper: The SparseSC placebo confidence interval for the unscaled ATT when available.
  • se: Backward-compatible approximate SE derived from the reported placebo confidence interval width. This is not an estimator-provided sampling standard error.
  • se_approx_from_ci: The same approximation exposed under an explicit name. se_available remains false unless a future estimator provides a true sampling SE.
  • significant: Boolean flag indicating if the p-value falls below the configured alpha threshold (usually 0.05) when significance_available is true. Failed results always report significant: false.
  • assumption_status: Summary of gate-eligible assumption checks. skipped means the parallel-trends diagnostic was not completed, pass means it did not fail, and fail or error means causal interpretation needs review. It is not a certificate of causal validity.

Placebo p-values are finite-sample and donor-pool dependent. With few eligible donor units, p-values are coarse. Missing confidence intervals must not be interpreted as zero uncertainty. Check ci.unavailable_reason to distinguish disabled CI computation from missing or non-finite estimator outputs. Check p_value_placebo_mode before relying on reproducibility: sampled means SparseSC capped the placebo enumeration.

Operational Interpretation Gates

Use status, data_validation.json, and assumption_validation.json together. status: "ok" means the artifact contract completed; it does not certify donor comparability, absence of spillovers, or production safety. The advisory interference screen is recorded in assumption_validation.json as signal, no_signal, or indeterminate. It adds a caveat for a signal or indeterminate screen, but does not alter result status or significance. Required parallel-trends failures mark inference as failed.

Do not combine scaled diagnostics with unscaled ATT, or compare GeoSC ATT to revenue, platform-reported lift, or other measurement-source outputs, unless the estimands, units, and time windows have been harmonized upstream.

Business Metrics

GeoSC provides raw lift estimates. It does not output built-in ROI or business-impact metrics (e.g., incremental revenue, ROAS). Use the statistical outputs (att) multiplied by your unit volumes and margin data in downstream reporting tools to calculate ROI.