Running experiments¶

How do you model an A/B test, train/test split, or cohort-evolution experiment? Four tools: treatment / control (per-segment A/B split), per-metric treatment (treatment affects only a named metric), arrival distributions (when each entity actually starts), and holdout split (temporal train/test split for ML).

Treatment / control¶

A segment's treatment block carves the segment into two arms — a treatment arm that receives a known log-odds lift starting at a named period, and a control arm that doesn't.

PythonYAML

from plotsim import create

config = create(
    about="Marketing email A/B test",
    unit="customer",
    seed=42,
    window=("2024-01", "2024-12", "monthly"),
    metrics=[
        {"name": "open_rate",  "type": "score", "polarity": "positive"},
        {"name": "click_rate", "type": "score", "polarity": "positive"},
    ],
    segments=[
        {
            "name": "test_cohort",
            "count": 200,
            "archetype": "flat",
            "treatment": {
                "fraction":        0.5,        # 50/50 split
                "lift_log_odds":   0.45,       # known effect size
                "start_period":    3,          # treatment from period 3 onward
                "treatment_label": "variant",
                "control_label":   "holdout",
            },
        },
    ],
)

about: Marketing email A/B test
unit: customer
seed: 42
window: { start: "2024-01", end: "2024-12", every: monthly }

metrics:
  - { name: open_rate,  type: score, polarity: positive }
  - { name: click_rate, type: score, polarity: positive }

segments:
  - name: test_cohort
    count: 200
    archetype: flat
    treatment:
      fraction:        0.5
      lift_log_odds:   0.45
      start_period:    3
      treatment_label: variant
      control_label:   holdout

Mechanics:

fraction is the share in the treatment arm. The remainder goes to control. Both arms get the control_label / treatment_label cohort tag for the manifest.
lift_log_odds is applied via sigmoid(logit(p) + lift) to the entity's effective trajectory position for periods ≥ start_period. A positive lift pushes treatment-arm entities toward higher positions; a negative lift pushes them lower.
Before start_period, treatment and control share identical trajectory positions — the AC for "pre-treatment baseline is identical."
Sensible lifts are in [-2.0, 2.0] log-odds units. Extreme values (> 1e6 magnitude) are rejected as non-finite.

The manifest's treatment_cohorts section records every per-entity assignment plus per-cohort summaries. Use it as the answer key for "who actually got the treatment?"

RNG isolation¶

Treatment assignment uses a salted RNG independent of the arrival RNG. The same (seed, treatment_config) produces identical assignments regardless of what arrival shape the segment carries.

Per-metric treatment¶

The default treatment lift is trajectory-wide — it shifts every metric the entity carries. When the treatment only affects one named metric (say, the email open rate, leaving click rate untouched), use target_metric:

PythonYAML

segments=[
    {
        "name": "email_test",
        "count": 200,
        "archetype": "flat",
        "treatment": {
            "fraction":       0.5,
            "lift_log_odds":  0.45,
            "start_period":   3,
            "target_metric":  "open_rate",       # only opens shift
        },
    },
]

segments:
  - name: email_test
    count: 200
    archetype: flat
    treatment:
      fraction:      0.5
      lift_log_odds: 0.45
      start_period:  3
      target_metric: open_rate

When target_metric is set, every other metric on the treatment-arm entity is byte-identical to its control-arm draw. The non-targeted metrics see no lift.

Correlated-metric leakage caveat: when the targeted metric is correlated with another, some of the lift propagates through the Cholesky copula. The residual is bounded but non-zero — for ρ ≈ 0.4, the non-targeted mean shift is materially smaller than the targeted shift, but not zero. Treat this as expected, not a bug.

The load-time validator rejects target_metric values not in the declared metrics list — a typo cannot silently produce a dataset where the lift is invisible.

Arrival distributions¶

A segment's arrival block controls when each entity actually starts. Default behavior: every entity starts at period 0. Set arrival to draw per-entity start_period values from a chosen distribution.

Four shapes:

Shape	Behavior
`uniform`	Even draws across `[start, end)`
`linear`	Triangular CDF — `increasing` back-loads (more late arrivals), `decreasing` front-loads
`step`	Discrete blocks: `[(period, fraction), ...]`, fractions sum to 1.0
`explicit`	Per-entity start periods supplied directly (length must match segment count)

PythonYAML

segments=[
    {
        "name": "organic_growth",
        "count": 200,
        "archetype": "growth",
        "arrival": {
            "kind":      "linear",
            "start":     0,
            "end":       18,                 # last arrival at period 17
            "direction": "increasing",       # back-loaded
        },
    },
    {
        "name": "cohort_cuts",
        "count": 100,
        "archetype": "flat",
        "arrival": {
            "kind":   "step",
            "blocks": [
                {"period": 0,  "fraction": 0.5},   # 50 % at launch
                {"period": 6,  "fraction": 0.3},   # 30 % at month 6
                {"period": 12, "fraction": 0.2},   # 20 % at month 12
            ],
        },
    },
]

segments:
  - name: organic_growth
    count: 200
    archetype: growth
    arrival:
      kind:      linear
      start:     0
      end:       18
      direction: increasing

  - name: cohort_cuts
    count: 100
    archetype: flat
    arrival:
      kind: step
      blocks:
        - { period: 0,  fraction: 0.5 }
        - { period: 6,  fraction: 0.3 }
        - { period: 12, fraction: 0.2 }

Per-entity cells before the entity's start_period have NaN trajectory positions. The metric generator emits None for those cells, and _drop_cold_start_rows strips them from per_entity_per_period facts after the build — your fact tables contain only the periods each entity was actually active.

start: 0 and end: None are the defaults for uniform and linear arrivals; end: None resolves to "leave each entity at least MIN_ACTIVE_PERIODS active periods."

Holdout split¶

A temporal train/test split for ML workflows. The engine slices every per_entity_per_period fact at cutoff = n_periods - holdout_periods and emits both halves alongside the unsplit fact:

PythonYAML

config = create(
    about="Credit risk training data",
    unit="customer",
    window=("2024-01", "2024-12", "monthly"),
    metrics=[
        {"name": "credit_score", "type": "amount", "polarity": "positive", "range": [300, 850]},
        {"name": "default_risk", "type": "score",  "polarity": "negative"},
    ],
    holdout={
        "target":  "default_risk",
        "periods": 2,                 # last two months held out
    },
    segments=[{"name": "applicants", "count": 500, "archetype": "growth"}],
)

about: Credit risk training data
unit: customer
window: { start: "2024-01", end: "2024-12", every: monthly }

metrics:
  - { name: credit_score, type: amount, polarity: positive, range: [300, 850] }
  - { name: default_risk, type: score,  polarity: negative }

holdout:
  target:  default_risk
  periods: 2

segments:
  - { name: applicants, count: 500, archetype: growth }

On disk you get <fact>.csv, <fact>_train.csv, and <fact>_holdout.csv per fact. The training half has period_index < cutoff; the holdout half has period_index >= cutoff.

Load-time gates: target must resolve to an int/float column on some fact table; periods >= 1; n_periods - periods >= min_training_periods (default 3). Only per_entity_per_period facts are split; dims, bridges, and event tables are excluded by design.

holdout and quality can run on the same config. The corrupted fact tables are sliced by date_key, so both <fact>_train.<ext> and <fact>_holdout.<ext> carry their proportional share of injected issues. The manifest's quality_injections.row_indices are keyed against the unsplit corrupted table, so a row index points at the pre-split row regardless of which file it ended up in.

Entity features¶

When you need a per-entity flat feature table — one row per entity with per-metric aggregates — set entity_features: true:

PythonYAML

config = create(
    about="Churn prediction features",
    unit="customer",
    window=("2024-01", "2024-12", "monthly"),
    metrics=[
        {"name": "engagement", "type": "score", "polarity": "positive"},
        {"name": "churn_risk", "type": "score", "polarity": "negative"},
    ],
    entity_features=True,
    segments=[{"name": "active", "count": 200, "archetype": "growth"}],
)

about: Churn prediction features
unit: customer
window: { start: "2024-01", end: "2024-12", every: monthly }

metrics:
  - { name: engagement, type: score, polarity: positive }
  - { name: churn_risk, type: score, polarity: negative }

entity_features: true

segments:
  - { name: active, count: 200, archetype: growth }

The output adds _entity_features.<ext> — one row per entity with per-metric _mean, _std, _slope, _first, _last, _peak_period aggregates. Optional archetype and final_trajectory_position ground-truth label columns are on by default.

The dict form narrows the metric set or strips labels:

PythonYAML

entity_features={
    "metrics":        ["engagement"],   # only this metric's aggregates
    "include_labels": False,            # no archetype label
}

entity_features:
  metrics: [engagement]
  include_labels: false

Gates: manifest.include = true (labels are read from the manifest payload), quality.quality_issues == [], every named metric must resolve to a numeric fact column. When holdout.enabled is true, aggregation is restricted to training periods and target-metric columns are dropped to prevent label leakage.

entity_features is not supported with output.format: sql.

Multi-source overlap¶

When the dataset models the same entities recorded in multiple upstream systems (e.g. CRM and billing each have their own copy of "customer"), declare a sources block. The engine emits a drifted per-source copy of each per_entity dim alongside the canonical dim, with optional drift on names, attributes, and key formats.

PythonYAML

sources=[
    {
        "name":                  "crm",
        "key_scheme":            "uuid_short",
        "name_drift_rate":       0.10,
        "attribute_drift_rate":  0.05,
    },
    {
        "name":                  "billing",
        "key_scheme":            "numeric",
        "name_drift_rate":       0.05,
        "attribute_drift_rate":  0.15,
    },
]

sources:
  - name:                 crm
    key_scheme:           uuid_short
    name_drift_rate:      0.10
    attribute_drift_rate: 0.05
  - name:                 billing
    key_scheme:           numeric
    name_drift_rate:      0.05
    attribute_drift_rate: 0.15

Two to five sources required when set. Each SourceDeclaration produces a dim_<entity>_<source> table. Ground-truth mappings between canonical and per-source identifiers land in the manifest's source_entity_mappings list — the answer key for entity resolution exercises.

Key schemes: prefix_padded (PK with zero-padded suffix), numeric (bare integer), uuid_short (truncated UUID).

Going deeper¶

Connecting metrics — time-varying correlations let the relationship shift across phases without an explicit treatment.
Shaping metrics — segment baselines and archetypes let you stage population-level differences without an A/B split.
Reference: Manifest schema — treatment_cohorts, holdout, source_entity_mappings, archetype_assignments are the ground-truth sections.