Profiles And Specs

English · Русский

Profiles and specifications have different roles.

Safe Profile

A profile describes source structure without becoming a copy of the source dataset. It can contain:

  • field names and inferred types;
  • null ratios and approximate distinct counts;
  • numeric ranges and percentiles;
  • date and timestamp ranges;
  • safe distributions for non-sensitive low-cardinality fields;
  • string length statistics and masked sensitive patterns;
  • inferred relationship and constraint candidates.

A profile must not contain source rows, raw PII, credentials, tokens, or rare free-text values. Exact values are disabled by default. An explicit field-scoped local allowlist may retain a reviewed bounded non-sensitive business enum or constant after content, cardinality, and value-length checks.

Create one from a CSV folder:

test-data-agent profile-example data/example_dataset \
  --output out/profile.json

For example, retain a reviewed local enum while replacing every other source literal:

test-data-agent profile-example data/example_dataset \
  --local-category orders.status \
  --output out/profile.json

This authorization remains local. External advisors receive deterministic field-scoped labels, and default MCP responses do not receive the exact enum.

SQL Query Source

This source type is available in stable 1.6.0.

One reviewed local query file can shape a PostgreSQL or Trino virtual entity:

test-data-agent profile-query query.sql \
  --adapter postgres \
  --source-id warehouse \
  --entity paid_orders \
  --output out/profile.json

SQL source policy 1.1 accepts exactly one fully qualified, single-table SELECT with explicit projections or an authorized qualified wildcard, bounded filters, and a small deterministic scalar-expression set. It rejects joins, CTEs, subqueries, set operations, windows, table functions, commands, volatile or unknown functions, multiple statements, and unauthorized references before derived aggregate work begins.

It also accepts aliased SUM, COUNT, MIN, MAX and AVG projections, optionally grouped by explicit authorized source columns. Each other selected column must be a grouping key. COUNT(*) counts rows; it is not a row wildcard. Aggregate arguments may use the existing scalar expressions, but nested or wrapped aggregates, DISTINCT aggregates, HAVING, grouping expressions/ordinals, ROLLUP and grouping sets remain unsupported. Grouped/aggregate queries referencing sensitive-name source columns fail closed even when renamed. Existing budgets and disclosure checks remain; small groups are not an anonymity guarantee. For example, SELECT status, COUNT(*) AS measured FROM public.orders GROUP BY status profiles a grouped relation without fetching its rows.

Query profiles also record has_unmodeled_expressions: a source-dependent non-column projection or any aggregate (including COUNT(*)) sets it to true. Profiling remains available, but infer-spec rejects such profiles rather than silently generating independent values that break SQL-expression dependencies. Direct column aliases remain eligible. Older SQL profiles missing the marker must be reprofiled before automatic inference. Non-query legacy profiles retain prior inference behavior. The optional marker defaults to null (unknown); unknown does not change legacy profile fingerprints, while explicit true/false is included in the fingerprint. This does not translate SQL into formulas or authorize raw query-result access.

The profile records source_fingerprint and source_policy_version, not SQL text or literals. The adapter performs a no-row schema probe followed by bounded aggregates. Query result rows are never returned, persisted, sent to a provider or MCP, or supplied to generation.

DatasetSpec

DatasetSpec is the reviewed, executable generation contract. It contains:

  • entities, fields, types, nullability, and row counts;
  • primary keys and relationships;
  • distributions and constraints;
  • privacy annotations;
  • structured business rules;
  • generation and validation settings and schema version.

Infer a spec:

test-data-agent infer-spec out/profile.json \
  --count 100 \
  --output out/dataset_spec.yaml

Review the spec before generation. Inferred relationships and constraints are candidates, not unquestionable facts.

Generation

Generation reads the specification, not source rows:

test-data-agent generate out/dataset_spec.yaml \
  --seed 12345 \
  --format csv \
  --output out/generated

The seed belongs to the generation request and is recorded in the effective spec and manifest.

Validation

test-data-agent validate out/dataset_spec.yaml out/generated \
  --output out/generated/validation_report.json

Validation is deterministic Python code. An AI client may plan or summarize the workflow, but it is not the only validator.

Versioning

Newly constructed DatasetSpec files use schema_version: "1.1"; version 1.0 remains supported, and serialized files without a version load as 1.0 for compatibility. Exact DECIMAL fields require 1.1. Keep the schema version, package version, spec_sha256, business_validation.rules_sha256, and seed with any dataset that must be reproduced later.

The complete field reference is in DatasetSpec.

Date and time bounds

A reviewed date_range or datetime_range supplies sampling bounds. Missing endpoints use 2020-01-01 and 2025-01-01. Supply both ordered endpoints for an earlier or later period: a partial bound that reverses the effective interval now rejects with a recovery hint. Generation keeps previous output artifacts unchanged on this failure.

Datetime sampling retains its whole-second offset policy and the lower endpoint's microseconds. Very long fractional intervals use integer duration arithmetic to stay within the maximum. Range endpoints alone do not establish monthly or other calendar granularity; source-period fidelity remains unmeasured.