Dataset Profile And Spec Reference¶

This document describes the two main artifacts in the domain-agnostic pipeline:

• DatasetProfile JSON
• DatasetSpec YAML

DatasetProfile JSON¶

A profile is safe metadata and bounded evidence inferred from a source. It is not a complete description of the domain and must not imply that unobserved rules are known.

Evidence Envelope¶

The evidence-bounded contract records how each important observation was obtained. Relationships and constraints already expose confidence and status; the release contract extends that record with provenance and coverage rather than treating inference as fact.

Target shape for an observed rule or relationship:

origin: observed
status: candidate
confidence: medium
evidence:
  sample_size: 200
  sampling: bounded
  time_window:
    from: '2026-01-01'
    to: '2026-03-31'
  rows_checked: 200
  violations: 0
source_fingerprint: sha256:...
assumptions: []

high confidence means strong support from the recorded evidence, not a universal truth claim. DDL or ORM declarations are declared; facts measured from a sample or live profile are observed; deterministic or AI proposals are inferred until a human accepts them. AI receives this context as untrusted metadata and cannot promote a hypothesis into a generation rule.

If sample size, sampling method, coverage, or time window is unknown, that uncertainty must remain visible in the profile and review record.

Top-level shape:

{
  "source_type": "csv_folder",
  "entities": [],
  "relationships": [],
  "constraints": []
}

Entity Profile¶

Each CSV file becomes an entity:

{
  "name": "customers",
  "row_count": 3,
  "fields": [],
  "primary_key_candidates": ["customer_id"]
}

Field Profile¶

Field profiles contain schema and safe distribution metadata:

{
  "name": "email",
  "data_type": "string",
  "nullable": false,
  "null_ratio": 0.0,
  "unique_ratio": 1.0,
  "sensitive": true,
  "semantic_type": "email",
  "is_identifier": false,
  "distribution": {
    "kind": "masked_patterns",
    "patterns": [
      {"pattern": "email", "count": 3}
    ]
  }
}

Sensitive fields use masked patterns, not raw top values. By default, low-cardinality non-sensitive fields preserve category frequency ranks with synthetic labels such as category_1. Exact values appear only for a field in the typed local category allowlist after bounded non-sensitive content checks; that exception does not apply to source rows, provider payloads, or default MCP responses.

Common distribution kinds:

• synthetic_identifier
• masked_patterns
• numeric
• boolean
• date_range
• datetime_range
• categorical
• string_pattern

Large Input Behavior¶

CSV-folder profiles are built with a streaming schema/distribution pass. The profiler can read large local files without keeping all source rows in memory. Only bounded samples are kept for row-level relationship and constraint mining.

The optional profile cache stores the DatasetProfile wrapper only. It is keyed by CSV file names, sizes, and modification times and must remain metadata-only: no source rows, no real identifiers, and no raw PII.

Trino-derived profiles should follow the same artifact shape but should be computed with pushdown aggregate queries in Trino. The local agent should keep the compact profile JSON, not a downloaded copy of the source table.

SQL Query Source Profile¶

Stable 1.4.0 can profile one reviewed PostgreSQL or Trino query as a virtual entity. The resulting source_type is postgres_query or trino_query; the profile records a stable source_fingerprint and source_policy_version: "1.0" alongside ordinary safe entity metadata and aggregates.

The fingerprint identifies the validated query shape without serializing the SQL. Query text, literals, physical endpoints, backend messages, and query result rows are excluded. Generation reads the reviewed inferred DatasetSpec, never query rows.

Relationship Profile¶

Relationships are inferred with confidence:

{
  "parent_entity": "customers",
  "parent_field": "customer_id",
  "child_entity": "orders",
  "child_field": "customer_id",
  "relationship_type": "many_to_one",
  "confidence": 1.0,
  "status": "inferred"
}

Constraint Profile¶

Constraints are also inferred with confidence and status:

{
  "type": "formula",
  "entity": "orders",
  "fields": ["amount", "quantity", "unit_price"],
  "expression": "quantity * unit_price",
  "confidence": 1.0,
  "status": "inferred"
}

Supported constraint types:

• formula
• temporal
• conditional_required
• aggregate_mapping

DatasetSpec YAML¶

The spec is the editable generation contract. It has the same core concepts as the profile, but it is intended for generation.

Small example:

schema_version: '1.0'
entities:
- name: customers
  row_count: 1000
  fields:
  - name: customer_id
    data_type: string
    nullable: false
    null_ratio: 0.0
    sensitive: false
    semantic_type: null
    is_identifier: true
    distribution:
      kind: synthetic_identifier
  - name: email
    data_type: string
    nullable: false
    null_ratio: 0.0
    sensitive: true
    semantic_type: email
    is_identifier: false
    distribution:
      kind: masked_patterns
      patterns:
      - pattern: email
        count: 3
  primary_key: customer_id

relationships:
- parent_entity: customers
  parent_field: customer_id
  child_entity: orders
  child_field: customer_id
  relationship_type: many_to_one
  confidence: 1.0
  status: inferred

constraints:
- type: formula
  entity: orders
  fields: [amount, quantity, unit_price]
  expression: quantity * unit_price
  confidence: 1.0
  status: inferred

schema_version is the serialized contract version and is independent from the Python package version. Version 1.0 validates unique entity/field names and checks that relationships, constraints, primary keys, and scoped privacy rules refer to fields that exist. Older specs without the key load as 1.0 for backward compatibility.

What You Can Edit¶

Useful manual edits:

• Change row_count per entity.
• Remove an inferred relationship if it is wrong.
• Remove or edit a constraint if confidence is low.
• Change categorical counts to steer scenario frequency.
• Mark a field as sensitive: true if unsure.
• Set nullable and null_ratio to shape missingness.

Avoid putting real source examples into the spec. Keep it metadata-only.

Validation Report¶

Validation returns sections:

{
  "valid": true,
  "sections": [
    {"name": "schema", "passed": 1, "failed": 0, "errors": []},
    {"name": "relationships", "passed": 1, "failed": 0, "errors": []},
    {"name": "constraints", "passed": 1, "failed": 0, "errors": []}
  ]
}

If valid is false, inspect the section errors first. They tell you whether the problem is basic schema shape, relationship integrity, or constraint reconciliation.