Output and scaling¶

How do you get data out, and how big can it get? Plotsim writes to four formats, supports Parquet partitioning, optional denormalized wide tables, and a cell-budget gate that prevents accidental multi-billion-row runs.

Formats¶

Format	Extension	When to use
`csv` (default)	`.csv`	Most things; opens anywhere
`parquet`	`.parquet`	Larger datasets; preserves nested cells; pyarrow required
`jsonl`	`.jsonl`	Streaming ingestion, schema-on-read pipelines
`sql`	`.sql`	One file with DDL + INSERTs for a target database

CSV¶

The default. Conventions: utf-8 encoding, index not written, float format %.4f, NaN/None as empty string, csv.QUOTE_NONNUMERIC, integer columns coerced to nullable Int64 so poisson columns that hit MCAR don't grow .0 suffixes, line terminator pinned to \n for cross-platform byte-identity. Column order: PK → FK (config order) → other config columns → DataFrame extras (e.g. stage).

PythonYAML

from plotsim import create, generate_tables, write_tables

config = create(
    about="Subscription customers",
    unit="customer",
    window=("2024-01", "2024-12", "monthly"),
    output={"format": "csv", "directory": "./output"},
    metrics=[{"name": "engagement", "type": "score", "polarity": "positive"}],
    segments=[{"name": "active", "count": 50, "archetype": "growth"}],
)
write_tables(generate_tables(config), config)

about: Subscription customers
unit: customer
window: { start: "2024-01", end: "2024-12", every: monthly }

output:
  format: csv
  directory: ./output

metrics:
  - { name: engagement, type: score, polarity: positive }

segments:
  - { name: active, count: 50, archetype: growth }

Parquet¶

Snappy-compressed, schema explicit for nested columns, inferred for the rest. Requires the parquet extra (pip install "plotsim[parquet]").

PythonYAML

output={"format": "parquet", "directory": "./output"}

output:
  format: parquet
  directory: ./output

Short string shorthand for the four formats:

PythonYAML

output="parquet"   # → {"format": "parquet", "directory": "output"}

output: parquet

JSONL¶

One JSON object per row. Dates as ISO-8601 strings, NaN/None as JSON null, struct/array cells as native nested JSON, unicode verbatim. Designed for Kafka/Kinesis/SQS replay and schema-on-read pipelines.

PythonYAML

output={"format": "jsonl", "directory": "./stream"}

output:
  format: jsonl
  directory: ./stream

SQL¶

One data.sql file with dialect-aware DDL + batched INSERTs (100 rows per statement). Dimensions emitted first (no FK dependencies), then fact / event / bridge in declaration order. Primary keys and foreign keys are emitted as constraints when the actual data is unique; quality-injected duplicates remove the PK constraint on that table.

PythonYAML

output={"format": "sql", "directory": "./load", "sql_dialect": "postgresql"}

output:
  format: sql
  directory: ./load
  sql_dialect: postgresql

sql_dialect: postgresql (default), mysql, sqlite. Each maps plotsim dtypes to the dialect's native column types — MySQL maps string PKs/FKs to VARCHAR(255) (TEXT can't be a PK); PostgreSQL maps float to NUMERIC; SQLite maps boolean to INTEGER and float to REAL.

entity_features is rejected with format: sql.

Partitioned Parquet¶

Set partition_by to a column name. Tables that have the column are written as a directory of per-partition Parquet files (<output>/<table>/<col>=<value>/...); tables without the column fall back to single files.

PythonYAML

output={
    "format":       "parquet",
    "directory":    "./output",
    "partition_by": "year",       # dim_date.year is the partition key
}

output:
  format:       parquet
  directory:    ./output
  partition_by: year

The partition column must exist on at least one table (literal or FK target) and use a partition-eligible dtype. float, struct, array are rejected. Requires format: parquet.

When the vectorized generation mode resolves AND format is parquet AND partition_by is not set, fact tables stream per-archetype row groups via ParquetWriter to bound peak pyarrow buffer memory.

Denormalized wide tables¶

Set denormalized: true to emit a <fct>_wide.<ext> alongside each normalized fact. Every FK'd dim is left-joined onto the fact, dim columns prefixed with <dim>__, SCD2 dims filtered to current rows.

PythonYAML

output={"format": "csv", "denormalized": True}

output:
  format: csv
  denormalized: true

Under format: sql, the wide tables and holdout splits emit as trailing CREATE TABLE + INSERT blocks in data.sql (no FK constraints — multi-dim shape doesn't fit the FK model).

Cell-budget gate¶

Plotsim refuses to generate huge datasets by accident. The cell-budget gate runs at config-load time and counts pre-quality cells across every table.

Threshold	What happens
≤ 2,000,000 cells (soft budget)	Run proceeds
> 2,000,000 cells	Validator raises with knobs to override
> 50,000,000 cells (hard ceiling)	Validator raises — not overridable

Three knobs to opt past the soft budget:

output.cell_budget: N — override per-config.
PLOTSIM_CELL_BUDGET=N env var — override per-environment.
plotsim run --allow-large-dataset — one-shot CLI opt-in.

PythonYAML

output={
    "format":      "parquet",
    "cell_budget": 5_000_000,    # raise the cap for this config
}

output:
  format:      parquet
  cell_budget: 5000000

Quality issues that grow row counts (duplicate_rows, volume_anomaly mode=spike) are counted in the post-quality estimate — when the post-quality count crosses the soft budget but pre-quality doesn't, the gate raises explicitly pointing at the rate fields.

The hard ceiling of 50,000,000 is non-configurable.

Output path handling¶

plotsim run, the library plotsim.run, and write_tables all accept any caller-supplied output path — absolute paths and ..-segment paths are honored. The caller is trusted to pick a safe target. For multi-tenant or hosted deployments, gate the path choice in the surrounding application before calling plotsim.

Table and column names are still SQL-safe identifiers ([A-Za-z_][A-Za-z0-9_]{0,127}), rejected at config load. The write_single_table entry point validates that the resolved file path stays inside output_dir as defense-in-depth against crafted table names from programmatic callers.

Suppressing the manifest¶

manifest.json ships by default. To disable it — for microbenchmarks, sandboxed CI runs, or downstream consumers that don't need the ground-truth payload — set manifest: {include: false} on the builder config:

PythonYAML

cfg = plotsim.create(
    # ... metrics, segments, etc. unchanged ...
    manifest={"include": False},
)
plotsim.run(cfg, "./output")     # ./output/manifest.json is NOT written

manifest:
  include: false

The manifest.trajectory_sample_rate knob also lives on this block — default 1.0 records every entity's trajectory; lower values sub-sample the per-period positions in the manifest while leaving the fact tables untouched.

Window caps¶

Maximum window span per granularity:

Granularity	Max periods	Max span
`monthly`	360	30 years
`weekly`	1,560	30 years
`daily`	3,650	10 years

Maximum total entities across all segments: 100,000. The runtime envelope is governed by the cell-budget gate, not this per-row cap.

Log-file companion writer¶

Event tables can opt into a .log file written alongside the CSV / Parquet. Set log_format (a Python str.format template whose placeholders must match column names) and optionally log_filename:

PythonYAML

events=[
    {
        "name":         "evt_login",
        "trigger":      "proportional",
        "driver":       "engagement",
        "scale":        10.0,
        "log_format":   "{event_ts} [INFO] {company_id} login {event_id}",
        "log_filename": "logins.log",
        "columns": [
            {"name": "event_id",   "type": "id"},
            {"name": "date_key",   "type": "ref.dim_date"},
            {"name": "company_id", "type": "ref.dim_company"},
            {"name": "event_ts",   "type": "timestamp"},
        ],
    },
]

events:
  - name: evt_login
    trigger: proportional
    driver: engagement
    scale: 10.0
    log_format: "{event_ts} [INFO] {company_id} login {event_id}"
    log_filename: logins.log
    columns:
      - { name: event_id,   type: id }
      - { name: date_key,   type: ref.dim_date }
      - { name: company_id, type: ref.dim_company }
      - { name: event_ts,   type: timestamp }

Log writer is event-only — rejected on facts, dims, and bridges. log_filename without log_format is rejected.

Going deeper¶

Reference: CLI — every CLI subcommand and flag.
Reference: API — the public Python surface.
Adding realism — the cell-budget gate interacts with quality issues.