Adding realism¶

How messy should your data be? Plotsim offers four shorthand presets for typical noise levels, plus explicit knobs for heteroscedastic scaling, heavy-tailed families, and six post-generation corruption types when you need to teach a pipeline how to fail.

Noise presets¶

The fastest path: name a preset.

Preset	gaussian_sigma	outlier_rate	mcar_rate
`clean` / `perfectly_clean`	0.00	0.00	0.000
`slightly_messy`	0.03	0.01	0.005
`realistic` / `messy`	0.05	0.02	0.010
`dirty` / `very_messy`	0.10	0.05	0.030

The three numbers are:

gaussian_sigma — standard deviation of additive jitter, as a fraction of each value's magnitude (so 0.05 is ±5 %).
outlier_rate — fraction of cells replaced by a draw from Uniform(3|v|, 10|v|) with sign preserved.
mcar_rate — fraction of cells replaced by None (missing completely at random).

PythonYAML

config = create(
    about="Realistic SaaS data",
    unit="company",
    window=("2024-01", "2024-12", "monthly"),
    noise="realistic",
    metrics=[{"name": "engagement", "type": "score", "polarity": "positive"}],
    segments=[{"name": "active", "count": 40, "archetype": "growth"}],
)

about: Realistic SaaS data
unit: company
window: { start: "2024-01", end: "2024-12", every: monthly }
noise: realistic

metrics:
  - { name: engagement, type: score, polarity: positive }

segments:
  - { name: active, count: 40, archetype: growth }

Explicit noise¶

Drop the preset and pass a dict for full control. The pipeline order is: additive jitter → outlier replacement → MCAR. Each branch skips its RNG call when its rate or sigma is zero — a zero-noise config consumes no randomness.

PythonYAML

noise={
    "gaussian_sigma": 0.04,
    "outlier_rate":   0.005,
    "mcar_rate":      0.0,
}

noise:
  gaussian_sigma: 0.04
  outlier_rate:   0.005
  mcar_rate:      0.0

Caps: gaussian_sigma ≤ 5.0; outlier_rate ≤ 1.0; mcar_rate ≤ 1.0.

Heteroscedastic noise¶

By default, gaussian jitter is multiplicative on cell magnitude — a larger value gets larger absolute noise. Sometimes you want the noise to scale with the entity's trajectory position instead. Set scale_with_trajectory: true:

PythonYAML

noise={
    "gaussian_sigma":         0.04,
    "outlier_rate":           0.005,
    "mcar_rate":              0.0,
    "scale_with_trajectory":  True,
}

noise:
  gaussian_sigma:        0.04
  outlier_rate:          0.005
  mcar_rate:             0.0
  scale_with_trajectory: true

Now the resolved scale becomes gaussian_sigma * trajectory_position — position-zero cells receive zero gaussian noise, position-one cells receive the full gaussian_sigma. The outlier and MCAR branches are unaffected by the flag.

Use case: large-account MRR shows more variation as an entity grows; small-account MRR doesn't. The SaaS template uses this.

The shorthand presets always leave scale_with_trajectory: false. Opt in via the explicit dict form.

Heavy-tailed families¶

The additive jitter distribution defaults to gaussian. Two heavy-tailed alternatives:

Family	RNG call	When to use
`gaussian`	`rng.normal(loc=0, scale=scale)`	Default; symmetric, light tails
`student_t`	`rng.standard_t(df) * scale`	Heavy tails; needs `degrees_of_freedom ≥ 1.0`
`laplace`	`rng.laplace(loc=0, scale=scale)`	Sharper peak, heavier tails than gaussian

PythonYAML

noise={
    "gaussian_sigma":      0.04,
    "outlier_rate":        0.005,
    "mcar_rate":           0.0,
    "noise_family":        "student_t",
    "degrees_of_freedom":  4.0,
}

noise:
  gaussian_sigma:     0.04
  outlier_rate:       0.005
  mcar_rate:          0.0
  noise_family:       student_t
  degrees_of_freedom: 4.0

student_t requires degrees_of_freedom; both are rejected on any non-student_t family. The shorthand presets always resolve to gaussian — opt in via the explicit dict.

The Laplace scale parameter b lands directly at scale (σ·|v| or σ·position); third-party KS-test references that match on variance will reject — b = scale/√2 for variance-matched Laplace.

Quality issues¶

Six post-generation corruption types you can inject deliberately to teach pipelines how to fail safely. Each issue draws from a dedicated RNG (seeded from the master seed + a per-issue offset), so reordering issues doesn't perturb earlier draws.

Issue	Effect	Column required?
`null_injection`	Sets `rate` of cells in the target column to null	yes
`duplicate_rows`	Inserts exact copies of `rate` of rows at random positions	no
`type_mismatch`	Converts `rate` of values to the wrong type (numerics → strings)	yes
`late_arrival`	For `rate` of rows, appends `_arrival_period` = original + 1..5	no
`schema_drift`	For `rate` of rows, copies cell to `<column>_v2`, sets original to null	yes
`volume_anomaly`	At named period(s), spike duplicates or drop removes rows	no — uses `period`/`periods` and `mode`

PythonYAML

quality=[
    {"table": "fct_engagement", "issue": "null_injection",  "rate": 0.03, "column": "engagement"},
    {"table": "evt_login",      "issue": "duplicate_rows",  "rate": 0.015},
    {"table": "fct_revenue",    "issue": "late_arrival",    "rate": 0.02},
    {"table": "fct_orders",     "issue": "schema_drift",    "rate": 0.04, "column": "order_total"},
    {
        "table":  "evt_login",
        "issue":  "volume_anomaly",
        "rate":   0.5,
        "mode":   "spike",
        "period": 11,                # Black Friday spike at period 11
    },
]

quality:
  - { table: fct_engagement, issue: null_injection,  rate: 0.03,  column: engagement }
  - { table: evt_login,      issue: duplicate_rows,  rate: 0.015 }
  - { table: fct_revenue,    issue: late_arrival,    rate: 0.02 }
  - { table: fct_orders,     issue: schema_drift,    rate: 0.04,  column: order_total }
  - table:  evt_login
    issue:  volume_anomaly
    rate:   0.5
    mode:   spike
    period: 11

volume_anomaly requires mode set to spike or drop, and exactly one of period (single int) or periods (list of ints). column is rejected on volume_anomaly (it's a row-level issue).

Up to 50 quality issues per config.

Ground-truth records¶

Every quality issue is recorded in the manifest's quality_injections list — one QualityInjection per (issue_index, table, column) with the original row indices and clean values. A downstream consumer can recover the pre-corruption cells exactly. This is the answer key for training a pipeline to detect what plotsim injected.

Cross-reference rules¶

FK, period, and date_key columns are protected — quality issues targeting them are rejected.
Dim tables and bridge tables are rejected as targets (only facts and events can be corrupted).
target_columns = ["*"] is the sentinel for "all eligible metric and attribute columns" (FK / period / date_key excluded automatically). The builder's single-column shorthand expands to ["*"] when you omit column on duplicate_rows / late_arrival / volume_anomaly.

Holdout × quality¶

holdout and quality can run on the same config. The engine slices the corrupted fact tables by date_key, so both <fact>_train.<ext> and <fact>_holdout.<ext> carry their proportional share of injected issues. The manifest's quality_injections.row_indices are keyed against the unsplit corrupted table, so a row index points at the pre-split row regardless of which file it ended up in.

Cell-budget gate¶

Quality issues that grow row counts (duplicate_rows and volume_anomaly mode=spike) feed into the cell-budget gate. The soft budget defaults to 2,000,000 cells; the hard ceiling is 50,000,000. When the post-quality cell count crosses the soft budget but the pre-quality count is below it, the validator raises explicitly, pointing at the rate fields and the opt-out knobs:

output.cell_budget: N in the config — override per-config.
PLOTSIM_CELL_BUDGET=N env var — override per-environment.
plotsim run --allow-large-dataset — opt into a single run above the cap.

The hard ceiling is non-configurable.

Going deeper¶

Shaping metrics — distributions before noise is the right place to control variance.
Output and scaling — the cell-budget gate interacts with output format choice.
Reference: Manifest schema — quality_injections, outlier_injections, noise_config sections.