Comparison to Other Geo-Experiment Libraries

GeoSC is an end-to-end Python toolkit for planning, running, and interpreting geo-level marketing lift tests with SparseSC-backed synthetic control. It helps teams choose markets with data-driven donor evaluation and power analysis, estimate incremental impact, and produce decision-ready diagnostics, plots, and artifacts. It overlaps with several other geo-experiment libraries, but it is not a drop-in replacement for all of them.

This page is a methodology-level guide, not a live feature matrix. Upstream libraries change. Before choosing a tool for a production measurement program, check the current upstream documentation, licence terms, and supported runtime surface.

Quick Positioning

GeoSC vs Google GeoX / TBR / GBR

Google’s geo-experiment literature and tools include time-based regression (TBR), geo-based regression (GBR), matched-market design, and related GeoX workflows. These methods are useful when the analyst wants an explicitly regression-based design and can defend the model form.

GeoSC differs in the counterfactual construction:

• Google TBR/GBR workflows model treatment outcomes from aggregate or matched control signals through a regression-style model.
• GeoSC constructs a counterfactual as a SparseSC-weighted combination of individual donor geographies.
• TBR/GBR diagnostics are naturally regression diagnostics: residual behaviour, model fit, and stability of the treatment-control relationship.
• GeoSC diagnostics emphasize donor comparability, pre-period synthetic-control fit, selected donor concentration, placebo granularity, and artifact status.

Operationally, this means GeoSC is usually more inspectable at the donor level: the analyst can review which donor geos were recommended or influential, whether the donor pool is concentrated, and whether placebo p-values are coarse. That does not make GeoSC automatically safer. If the donor pool is small, non-comparable, or exposed to spillovers, a sparse synthetic control can still be misleading.

Use Google-style TBR/GBR or matched-market tools when:

• your organization already uses the Google geo-experiment workflow;
• regression assumptions and aggregate controls are acceptable for the decision;
• assignment constraints and matched-market design are the main problem;
• you need outputs that align with existing Google measurement or Meridian calibration processes.

Use GeoSC when:

• individual donor geography diagnostics matter;
• you want a Python package and CLI with YAML-backed runs;
• you need explicit JSON/CSV artifacts for validation and audit;
• you want SparseSC interpolation rather than an aggregate regression counterfactual.

GeoSC vs Meta GeoLift

Meta GeoLift and GeoSC are closer methodologically than TBR/GBR because both are synthetic-control-oriented geo-experiment tools. The key difference is the synthetic-control variant.

Meta GeoLift is built around augmented synthetic control methods. The augmentation uses a prognostic outcome model, such as ridge regression or generalized synthetic control options, to improve the counterfactual when raw synthetic-control balance is imperfect. It also provides an end-to-end R workflow for market selection, power analysis, analysis, and plotting.

GeoSC uses SparseSC as its estimator backend. SparseSC regularizes the matching problem and uses cross-validation to select penalty settings. GeoSC wraps this with Python-native operational controls: data validation artifacts, assumption validation artifacts, donor evaluation outputs, power simulation metadata, and release-preflight checks.

The practical distinction:

• Meta GeoLift is often a better fit for teams already using the Meta R package, its design functions, and its conformal/augmented synthetic-control workflow.
• GeoSC is a better fit for Python teams that want SparseSC-based inference, explicit artifact contracts, and a CLI-oriented production workflow.
• Neither package removes the need to validate donor comparability, treatment isolation, measurement-source consistency, and estimand alignment.

GeoSC vs Bayesian Causal-Impact Tools

GeoSC is not Bayesian and does not use PyMC. It does not produce posterior credible intervals, hierarchical meta-analysis estimates, or Bayesian structural time-series decompositions.

Bayesian causal-impact tools are useful when the analysis goal is to fit an explicit Bayesian time-series or Bayesian synthetic-control model and interpret posterior uncertainty under that model. GeoSC is useful when the analysis goal is SparseSC-based geo lift with donor-level diagnostics and production artifacts.

Do not pool or directly compare posterior effects from Bayesian tools with GeoSC’s ATT unless the estimand, scaling, time window, outcome definition, and measurement source are aligned.

What GeoSC Adds Operationally

GeoSC is designed to make production readiness demonstrable:

• geosc power writes power_analysis_results.csv with simulation seeds, failed-run counts, failure rates, effective placebo settings, backend fields, and valid flags.
• geosc donors writes donor_eval_results.csv and donor_pool_quality.json with quality bands, warning flags, and donor concentration diagnostics.
• geosc infer writes geolift_results.json, geolift_diagnostics.json, data_validation.json, and assumption_validation.json.
• result status is explicit: ok, partial, or failed.
• selected_weight in donor evaluation is a design-stage recommendation and concentration signal, not a fitted SparseSC donor weight.
• seeded CPU power runs are reproducible; GPU behaviour can vary by backend, hardware, and numerical library.

These are engineering and auditability features. They are not evidence that a specific experiment is causally valid.

Selection Checklist

Choose the tool based on the estimand and workflow, not the package name:

• Use a regression-based tool when a regression counterfactual is the estimand you can defend.
• Use Meta GeoLift when the augmented synthetic-control workflow and R ecosystem are the desired operational surface.
• Use GeoSC when SparseSC, Python, donor-level diagnostics, and artifact validation are the desired operational surface.
• Use Bayesian causal-impact tools when posterior model uncertainty is the target output.

For any tool, treat small samples carefully. Placebo p-values are coarse when few eligible donor geos are available, and no software package can repair a non-comparable donor pool or contaminated experiment.

References

• Google Research: “A Time-Based Regression Matched Markets Approach for Designing Geo Experiments”.
• Google GeoexperimentsResearch API documentation.
• Meta GeoLift documentation and facebookincubator/GeoLift.
• Ben-Michael, Feller, and Rothstein: “The Augmented Synthetic Control Method”.
• Local design note: assets/presentations/GeoX_vs_GeoLift_comparison/main.tex.

Frequently Asked Questions

Quick answers to the most common GeoSC questions.

Business Questions

Q: What problem does GeoSC solve?

A: GeoSC estimates the incremental impact of regional marketing campaigns under a synthetic-control design. It answers the practical question: “How much did observed outcomes differ from a SparseSC counterfactual for the treated geographies?” The estimate is only credible when the donor pool, treatment timing, measurement source, and spillover assumptions are defensible.

Q: Why is this important for my business?

A:

• Estimate incremental lift: Quantify observed outcomes against a documented counterfactual.
• Improve budget decisions: Compare campaigns using a consistent measurement design.
• Avoid unsupported claims: Separate evidence of lift from descriptive trends where the design is weak.
• Support governance: Preserve JSON/CSV artifacts for review, audit, and sensitivity analysis.

Q: Is this the only measurement tool I need?

A: No, GeoSC is one component of a comprehensive measurement strategy:

• Best for: Regional campaigns, store rollouts, geo-targeted advertising
• Complements: Media Mix Modelling (MMM), multi-touch attribution, A/B testing
• Integrates with: Your existing analytics stack (Google Analytics, Adobe, etc.)
• Use alongside: Brand studies, customer surveys, and other measurement approaches

Q: What’s the cost and expected ROI?

A: GeoSC does not estimate commercial ROI by itself. Cost and ROI depend on your license terms, campaign spend, margin assumptions, operational constraints, and whether the experiment produces a credible effect estimate. Use GeoSC’s statistical outputs as inputs to a separate business-case calculation.

Q: How does the process work?

A: GeoSC supports a three-stage workflow:

1. Evaluate donors: Identify control geographies that are plausible comparisons for the treated geographies.
2. Check test strength: Use simulation-based power analysis to assess detectable effects and failure rates under explicit DGP assumptions.
3. Measure lift: Run SparseSC inference and review result, data-validation, diagnostic, and assumption-validation artifacts.

Elapsed time depends on data readiness, review requirements, donor-pool quality, and campaign complexity.

Getting Started

Q1: What data do I need to run a GeoSC analysis?

A: You need:

• Time series data: At least the configured minimum number of pre-treatment observations (12 in the shipped inference example). More history is usually preferable when it is comparable to the campaign period.
• Geographic units: Markets, DMAs, states, or regions
• Outcome metric: Sales, conversions, or other KPIs
• Treatment assignment: Which markets received the campaign

Data format: CSV with columns for date, geographic unit, outcome metric, and treatment information. The exact required columns and date format come from your YAML configuration.

Q2: How long should I run my campaign to get reliable results?

A: Use the power-analysis command to assess duration and detectable effects:

geosc power --config data-config/power_analysis_config.yaml --use-gpu --jobs -1

This source-checkout example uses the shipped demo configuration. With a packaged installation, supply your own configuration path instead. Review power_analysis_results.csv for simulated power by duration and effect size, including valid, failure_rate, and Monte Carlo uncertainty fields.

There is no universal duration that guarantees power. Longer post periods can help, but power also depends on outcome noise, effect size, donor comparability, placebo support, and campaign timing.

Q3: How many control markets do I need?

A: There is no fixed safe number. You need enough comparable, uncontaminated control geographies to support a useful synthetic counterfactual and placebo reference set. Use the donor evaluator to inspect candidate quality:

geosc donors --config data-config/donor_eval_config.yaml --jobs -1

Outputs are saved in outputs/multicell_donor_eval/ (CSV with scores and donor maps).

Analysis Issues

Q4: My pre-period fit looks poor. What should I do?

A: Investigate the design before changing the estimator:

1. Check data quality: Confirm consistent measurement methodology, treatment timing, and geography identifiers.
2. Inspect unusual periods: Document holidays, promotions, outages, or other shocks. Do not remove observations simply to improve fit; pre-specify exclusions where possible and report sensitivity analyses.
3. Extend comparable history: Add more pre-treatment observations only if measurement definitions and market conditions remain comparable.
4. Revisit the design: Consider an alternative, defensible geographic aggregation or donor pool if the current one cannot support a credible counterfactual.

Q5: My results show no significant effect. What went wrong?

A: Common causes:

• Low statistical power: Campaign too short or effect too small
• Poor control selection: Controls don’t match treatment markets well
• External factors: Market disruptions during campaign period
• Data issues: Measurement problems or missing data

Check power analysis first to ensure the study design was capable of detecting the expected effect size.

Q6: The effect size seems too large to be believable. Is this normal?

A: Large effects can be real, but verify:

1. Check data definitions: Ensure consistent measurement
2. Review external events: Major market changes during campaign
3. Validate treatment assignment: Confirm which markets got treatment
4. Run sensitivity analysis: Test with different time periods or alternative donor pools.

Interpretation

Q7: How do I interpret the confidence intervals?

A: Confidence intervals show the range of effect sizes implied by the SparseSC placebo interval when the estimator returns finite bounds:

• Narrow intervals: More precise estimates
• Wide intervals: More uncertainty in the estimate
• Intervals excluding zero: Statistically significant effects

Example interpretation:

Relative Lift: +15.2% (95% CI: +7.8% to +22.6%)

Read this as: “SparseSC returned a finite 95% placebo interval from +7.8% to +22.6% for this configured analysis.” It is not proof that the campaign-level causal assumptions hold.

Q8: What’s the difference between absolute and relative lift?

A:

• Absolute lift: Raw units of incremental impact (e.g., +1,000 sales)
• Relative lift: Percentage increase over baseline (e.g., +15%)

Both are important:

• Use absolute lift for revenue calculations
• Use relative lift for comparing across campaigns

Q9: How do I calculate ROI from the results?

A: GeoSC reports statistical effect estimates (e.g. ATT, confidence intervals). ROI calculations depend on your business context (price, margin, cost). Combine effect estimates with your cost and unit economics to derive ROI externally.

Q10: When should I trust the results vs. be sceptical?

A: Treat results as more credible when:

• status is ok, data validation is clean, and required assumption checks have passed.
• Pre-period fit and donor comparability are good, with an adequate placebo reference set.
• The effect remains interpretable across pre-specified sensitivity analyses.
• Treatment isolation, outcome definitions, and the measurement window are defensible for the estimand.

Be sceptical when:

• Poor pre-period fit or large residuals
• status: partial or failed, missing uncertainty metadata, or failed required assumption checks
• Very large or very small effect sizes
• A coarse or unstable placebo reference set, including very few eligible donors
• Results change dramatically with small specification changes

A small placebo p-value is evidence within the configured reference set, not a standalone proof of a campaign-level causal effect.

Need More Help?

• Technical details: See API Reference or CLI Reference
• Step-by-step guidance: Check How-To Guides
• Mathematical background: Review Mathematical Formalism
• Quick start: Try Quick Start Guide

Interpreting Results

GeoSC produces a suite of statistical indicators to help determine the validity and magnitude of an experiment’s impact.

Main Statistical Outputs

• status: Result contract status. ok means the core ATT, p-value, and uncertainty fields are available; partial means the estimate ran but one or more uncertainty fields are unavailable; failed means required estimator outputs are missing and the effect should not be interpreted.
• att (Average Treatment Effect on the Treated): The average difference between the observed outcome and the synthetic counterfactual during the post-treatment period.
• p_value: A donor-pool placebo p-value: the share of placebo effects at least as extreme as the observed ATT under the implemented SparseSC placebo comparison.
• ci_lower / ci_upper: The SparseSC placebo confidence interval for the unscaled ATT when available.
• se: Backward-compatible approximate SE derived from the reported placebo confidence interval width. This is not an estimator-provided sampling standard error.
• se_approx_from_ci: The same approximation exposed under an explicit name. se_available remains false unless a future estimator provides a true sampling SE.
• significant: Boolean flag indicating if the p-value falls below the configured alpha threshold (usually 0.05) when significance_available is true. Failed results always report significant: false.
• assumption_status: Summary of gate-eligible assumption checks. skipped means the parallel-trends diagnostic was not completed, pass means it did not fail, and fail or error means causal interpretation needs review. It is not a certificate of causal validity.

Placebo p-values are finite-sample and donor-pool dependent. With few eligible donor units, p-values are coarse. Missing confidence intervals must not be interpreted as zero uncertainty. Check ci.unavailable_reason to distinguish disabled CI computation from missing or non-finite estimator outputs. Check p_value_placebo_mode before relying on reproducibility: sampled means SparseSC capped the placebo enumeration.

Operational Interpretation Gates

Use status, data_validation.json, and assumption_validation.json together. status: "ok" means the artifact contract completed; it does not certify donor comparability, absence of spillovers, or production safety. The advisory interference screen is recorded in assumption_validation.json as signal, no_signal, or indeterminate. It adds a caveat for a signal or indeterminate screen, but does not alter result status or significance. Required parallel-trends failures mark inference as failed.

Do not combine scaled diagnostics with unscaled ATT, or compare GeoSC ATT to revenue, platform-reported lift, or other measurement-source outputs, unless the estimands, units, and time windows have been harmonized upstream.

Business Metrics

GeoSC provides raw lift estimates. It does not output built-in ROI or business-impact metrics (e.g., incremental revenue, ROAS). Use the statistical outputs (att) multiplied by your unit volumes and margin data in downstream reporting tools to calculate ROI.

Methodology & Advanced Topics

This document provides a technical overview of the advanced features and mathematical concepts underlying GeoSC, focusing on how SparseSC is implemented and operationalised.

For a more rigorous econometric exposition intended for Data Scientists and Statisticians, see the Mathematical Formalism of Synthetic Controls.

SparseSC Mathematical Foundation

The standard Synthetic Control Method minimises pre-treatment prediction error by constructing a convex combination of donor units.

GeoSC uses the vendored SparseSC implementation, which extends traditional synthetic control by learning a regularized match space and donor weights from the pre-treatment data.

Two objects are useful for interpretation:

1. Feature weights / match space (V): Which pre-treatment features or time periods matter most for matching.
2. Unit weights (W): How control units combine to form the synthetic counterfactual.

SparseSC exposes penalties for unit-weight shrinkage (w_pen) and feature weighting (v_pen). GeoSC’s default inference configuration uses SparseSC’s fast path, which constructs the match space through RidgeCV-backed machinery in the vendored implementation. The non-fast path uses SparseSC’s coordinate descent and cross-validation over feature-weight penalties.

Regularization can reduce overfitting to pre-treatment noise and often produces more interpretable donor structure, but it does not make a weak donor pool valid. The result still depends on comparable controls, adequate pre-treatment support, and uncontaminated treatment timing.

Statistical Inference in GeoSC

Because regional marketing experiments typically involve a very small number of treated units (often just one), asymptotic statistical inference is usually not appropriate. GeoSC uses SparseSC placebo inference to compute empirical p-values and, when SparseSC returns finite bounds, placebo confidence intervals.

Placebo Inference (Permutation Tests)

The primary method for significance testing is in-space placebo permutations.

1. The algorithm iteratively reassigns the “treatment” status to every untreated unit in the donor pool.
2. It fits a completely new SparseSC model for each of these placebo units.
3. It calculates the pseudo-treatment effect (the prediction error in the post-treatment period) for each placebo.
4. The empirical p-value is calculated as the proportion of placebo effects that are at least as extreme as the actual estimated treatment effect.

This is a finite-sample comparison against the observed donor pool, not an unconditional production-safety guarantee. With few donors the p-value grid is coarse, and non-comparable geographies, spillovers, or poor pre-period fit can make the placebo reference set uninformative.

Advanced Configuration & Tuning

Regularisation Selection (Cross-Validation)

SparseSC selects penalty settings using cross-validation over pre-treatment information. The exact machinery depends on whether the configured run uses the fast path.

If the default automatic search does not yield a good fit, the configuration allows for direct intervention:

# Inference configuration options
sparse_sc_model_type: "retrospective"
sparse_sc_fast_estimation: true
sparse_sc_level: 0.95

• sparse_sc_fast_estimation: When set to true, GeoSC calls SparseSC’s fast estimator. This uses a RidgeCV-backed match-space path and is generally faster than the full coordinate-descent estimator. Treat any speed/accuracy tradeoff as data-dependent.
• sparse_sc_model_type: Setting this to retrospective ensures the model adheres strictly to the pre/post treatment boundary defined in the configuration.

Donor Pool Constraints

By default, GeoSC includes all non-treated units in the donor pool. However, if business logic dictates that certain regions should not be used (e.g., they experienced a supply chain shock), they should be excluded from the input CSV entirely prior to analysis.

The recipes/donor_evaluator.py tool helps systematically identify the highest quality donors prior to running inference.

Performance Considerations

GeoSC relies heavily on matrix operations during the cross-validation and optimisation phases. The following options are supported to improve execution speed:

• Power Analysis Parallelism: Set jobs: -1 (or pass --jobs -1 via CLI) to use all available CPU cores when simulating hundreds of treatment effects.
• GPU Acceleration: The maintained GPU path is optional CuPy-backed acceleration for the power calculator only. Pass --use-gpu to request it for performance-oriented power runs. This requires cupy to be installed and matched to your system’s CUDA version (e.g., pip install cupy-cuda12x). Seeded power runs disable GPU DGP/generation paths so CPU simulation draws are reproducible.
• Inference Computation: The main SparseSC inference step runs on CPU. The legacy geolift.gpu_accelerator module is experimental compatibility code and is not wired into the maintained inference pipeline. To avoid thread oversubscription when running on multi-core servers, set environment variables such as OMP_NUM_THREADS or MKL_NUM_THREADS to a sensible value relative to your total core count.

Need More Help?

• Implementation Details: See Mathematical Formalism
• Step-by-step guidance: Check How-To Guides
• Technical specifications: Review the Configuration Reference