Generation, validation, and business-rules guide

All sources and output adapters share the source/output contract. Implement and test source/format additions through the same reviewed DatasetSpec and deterministic rule engine.

Read this guide for generation models, validators, foreign keys, formulas, scenarios, invalid cases, or cross-table behavior.

Represent business logic as structured YAML or JSON and give it executable semantics. The LLM may draft or infer a rule, but deterministic code must enforce and validate it.

Supported rule categories include:

  • field and row rules;
  • cross-table and foreign-key rules;
  • conditional and temporal-ordering rules;
  • formula and aggregate-formula rules; and
  • scenario-distribution rules.

The generator must:

  • produce records satisfying requested rules;
  • preserve referential integrity when requested;
  • generate invalid records only when explicitly requested;
  • label or report controlled invalid cases;
  • validate every rule after generation; and
  • produce a business-validation report.

Reviewed DatasetSpec 1.1 can generate exact decimal_range fields with precision at most 38 and explicit base-ten bounds. This source-free path uses integer units and exports Parquet decimal128, never binary FLOAT. Automatic profiling still does not infer exact source bounds. Ordinary DECIMAL formulas and aggregate mappings use bounded rational arithmetic, with one ROUND_HALF_UP rounding at the target scale and precision overflow rejection. Sensitive DECIMAL ranges remain disallowed. Row and aggregate formula constants, numeric bounds and condition range endpoints retain literal privacy checks before assignment or comparison, including schema-less rule-file calls. Sensitive names, semantics and flags veto typed literal targets even when optional validation reports are disabled. Only reviewed DECIMAL numeric_measure fields retain their existing scoped numeric-literal exception. Quoted field references in aggregate calls remain metadata rather than literal values. Decimal validation and CSV, JSON, Parquet and SQL serialization share a fixed-point text bound of 41 characters, covering sign, leading zero, decimal point and 38 fractional digits. Non-finite values and excessive expansion fail before formatting, including controlled-invalid Parquet fallback cells. This does not relax declared precision, scale or range checks. Recognizable phone/card-like DECIMAL bounds and final values fail closed even when optional validation reporting is disabled. Do not treat a declared numeric type as permission to bypass content checks. Explicit reviewed numeric_measure DECIMAL fields may use the bounded policy described in RC4 feedback disposition; sensitive names, flags, identifiers and privacy classification veto it. Profiles never assign it automatically.

After constraint solving, valid-mode generation also performs unconditional schema/type and recognizable-PII checks before returning rows to any export or publication adapter. These safety checks are not disabled by validation-report settings. Controlled negative generation may violate schema rules by design, but it must still pass the privacy check.

Important rules must be represented by typed models and executable validators, not only by free-form LLM reasoning. Keep generation deterministic with an explicit local seeded random source. Do not introduce source-row reuse, identity preservation, or implicit real-value dictionaries as a shortcut. Entity names reserved for generated control artifacts are rejected by the dataset specification and writer before any output file is created.

Review-first plans warn when date/time ranges lack endpoints: default generation uses 2020-01-01 for a missing minimum and 2025-01-01 for a missing maximum. These are fallback assumptions, not observed source bounds or measured fidelity. Set explicit bounds in the reviewed specification when a particular period is required. This warning does not change generation or certify source-period utility.

Repeated identifier fields inferred from CSV profiles use a fixed synthetic key pool: four distinct source keys yield four synthetic keys even when output grows. Only the count is carried forward, never source identifiers or row mappings. The typed synthetic_identifier distribution accepts positive integer pool_size; without it, identifiers retain their existing per-row generation behavior. Pools cycle deterministically and unrelated identifier domains remain disjoint. String identifiers retain the exact synthetic_ plus ASCII integer namespace, including when their field has email, phone or SSN semantics. Identifier generation takes precedence over semantic formatting. Privacy validation recognizes only that exact token on identifier fields, not arbitrary prefixed sensitive text; ordinary non-identifier sensitive fields still use their semantic synthetic formats. The optional distribution prefix is retained legacy metadata and does not change this namespace or insert source text into generated identifiers. Fewer output rows or nulls can leave some pool members unused. Source frequencies are not reproduced, and declared relationship constraints still take precedence. Repeated-key pools are not nominated as primary keys; an explicit primary key cannot have a pool smaller than its requested row count. Folder overflow and censored single-CSV counts cannot infer a fixed pool. Other aggregate profiles may supply approximate counts: these define the requested synthetic pool, not proof of exact source cardinality. Reprofile older artifacts to obtain pool metadata; no source-preservation permission is implied.

Batch transformation relationship validation distinguishes logical null from empty text: only None is null, and empty strings participate in scalar foreign key membership and one-to-one cardinality. The shared relationship validator keeps its legacy generated-CSV default unless a caller explicitly selects empty_string_is_null=False.

Ordinary formula evaluation enforces fixed per-operation resource caps independently of wall-clock checks: strings and bytes are limited to1,000,000elements; integers and Fraction components to16,384bits; Decimal coefficients, absolute exponents and arithmetic context precision to1,024. Sequence concatenation/repetition checks the prospective size before invoking the operator, including named operands and intermediate results. Built-in numeric arithmetic and bounded string/byte operations remain supported; list/tuple/bytearray and overloaded objects are not formula operands. Rejections contain no operand values. These caps complement dataset/output budgets rather than replacing them.

Common YAML loading rejects recursive aliases before construction. Expanded alias nodes share TEST_DATA_AGENT_MAX_INPUT_CELLS; expanded scalar bytes share TEST_DATA_AGENT_MAX_INPUT_FILE_BYTES. Ordinary bounded acyclic aliases remain supported. Physical alias/depth ceilings still apply, and errors contain no input values. These checks precede model construction and JSON serialization.

Every generate_dataset consumer performs the common output allocation estimate before Faker setup and row creation, including direct Python calls, PostgreSQL SQL export and transformation synthesis. Filesystem capacity checks remain in I/O workflows. String construction checks TEST_DATA_AGENT_MAX_INPUT_CELL_CHARS before random-character allocation. These conservative payload estimates and cooperative deadlines do not promise an exact peak-RSS ceiling.

Deterministic native constraints and business rules share an explicit GenerationBudget through generation, solving, business application and validation, including mandatory writer reconciliation before publication. Writers and the PostgreSQL SQL renderer accept the caller's keyword-only budget; standalone writer calls create a bounded allowance when none is supplied. An adapter must not restart the invocation allowance or deadline at the export boundary. Deadline admission and publication checks also apply without business rules or optional validation reports, including stdout, Parquet and PostgreSQL SQL export. Inner row, field, expression and relationship-graph loops check the cooperative deadline and charge cumulative work against TEST_DATA_AGENT_MAX_BUSINESS_RULE_EVALUATIONS. Direct Python entry points also create a bounded budget when none is supplied. Native rule amplification is estimated before row generation; business rule preflight uses actual supplied row counts. Mixed/negative FK and aggregate estimates conservatively include repeated parent/table scans for every possible selected row, even with a small invalid ratio. Budget failures escape formula error reporting and abort the workflow before trusted publication. Optional validation report settings do not disable mandatory post-solve rule checks. Explicit keyword-only budget support in business callbacks is optional; existing two/three-argument callbacks remain supported. These operation allowances are conservative cooperative controls, not CPU-instruction or peak-RSS guarantees.

Generation seeds are bounded before generator setup: persisted DatasetSpec seeds remain nonnegative and at most 2**63-1; direct Python generation also permits signed 64-bit negative seeds for compatibility. Oversized seeds fail closed without truncation or hashing, bounding synthetic identifier width.

Business validation failure details contain rule location and condition only; conditional allowed-values failures never embed the rejected cell value.

Temporal sampling must compute whole-second offsets with integer timedelta days and seconds. Float total_seconds can round long fractional spans upward and violate a reviewed maximum. Preserve the existing second-resolution policy; calendar grain is unknown unless a separate reviewed typed contract is active.

A partial temporal bound that reverses the interval after applying the existing 2020/2025 defaults must reject with a value-free explicit-bounds hint. Do not silently clamp it to a date outside the supplied interval.

FormulaRule accepts optional typed when using the existing Condition policy. False branches skip both assignment and expression validation, with explicit eligible/evaluated/skipped counters. Omitted predicates preserve old hashes and report shape. Predicate fields cannot be row-rule write targets; conditional assignments sharing a target require proven disjoint equality/membership sets. Schema-less calls enforce predicate literal privacy too. Keep the expression AST unchanged and predicate work on the same GenerationBudget. EAV examples remain independently authored models, never source-row pivots.

Explicit DatasetSpec relational_scenarios bind exact synthetic orphan quotas and optional generated-parent ordinal fan-out to effective counts. Preparation lives in rules/relational_scenarios.py; generation allocates a bounded key preview and validation independently reconciles final counts. Controlled invalid references remain invalid even when expectations match. Scenario specs require mandatory schema/privacy/native/business/scenario reconciliation at generation and writer boundaries, regardless of optional reporting settings. Omitted declarations serialize as absent. See the relational synthesis contract for conflicts/limits.