CLI Command Index

English · Русский

This reference describes stable 1.6.0. For complete tasks start with the user guide or Transform Local Data. Historical anchors are retained so existing links keep working.

Generation retains supported intentionally invalid output and its validation report, but returns exit 1 / JSON status validation_failed when validation fails, including mixed and negative modes. Spec, profile and CSV inputs use the same contract. Valid reports return exit 0. Privacy failures still prevent publication; invalid mode never bypasses them.

The executable is test-data-agent. Built-in help is the authoritative option reference for the installed version:

test-data-agent
test-data-agent --help
test-data-agent COMMAND --help
test-data-agent --version

Running without a command prints the available commands and safe starting points, then exits successfully.

Use these focused references for task detail:

Generate And Validate

Command Purpose Primary output
demo Run the offline fictional example Synthetic CSV bundle
profile-csv Profile one CSV into safe metadata Profile JSON
profile-parquet Profile one Parquet file into safe metadata Profile JSON
profile-example Profile a folder with one CSV per entity Profile JSON
infer-spec Infer a reviewable DatasetSpec YAML or JSON spec
generate-from-csv Run the complete single-table workflow Data file and review artifacts
generate-from-example Run the complete related-table workflow Dataset bundle
generate Generate from a spec or safe profile Data file or dataset bundle
validate Validate generated data against a DatasetSpec Validation report

Selective Transformation (Review And Policy Decisions)

transform-review SOURCE POLICY.yaml reads a fixed local source snapshot and the policy's referenced local mapping files, then prints value-free field decisions and a snapshot digest. It does not approve, transform, or export source rows. Mapping paths in the policy are relative to the policy file's directory. Use --table NAME when the entity name differs from the source filename stem; --json wraps the same review in the standard CLI response. The policy's input_format selects CSV, Parquet or an already captured PostgreSQL/Trino query-result envelope. Configured query capture is a separate transform-batch query-execute operation; this review command neither imports SQL scripts as data nor connects to a database. Installed fictional acceptance covers all four inputs and three outputs using the same saved policy and review digest, including public command execution. Native Parquet/captured-result input supports nullable strings, signed integers, float64, BOOLEAN, date32 and declared decimal128. Native Arrow timestamps, nested types and unsupported numeric shapes fail closed; this is not a promise of arbitrary Parquet input support. Textual DATETIME output conversion is separate. For exact-text plans, a single-file policy may declare a top-level file_text_mapping and optional CSV mapping on each replace_text field. The review reports configured scopes but never shows mapping literals; matching field rules take priority over file-wide rules, without cascading. Duplicate keys within either table are rejected. Review itself never produces replacement output; execution consumes the separately reviewed snapshot.

CSV review uses literal empty cells: an empty cell is a non-null string. A behavior policy may set csv_nulls.input_token to a nonempty explicit null marker; the marker is matched exactly and excluded from type inference. The executor additionally uses csv_nulls.output_token for null output. Setting a marker is not approval or execution. Source/output markers are separate from a mapping file's null_token. All settings are bound to the exact policy snapshot, so changing them invalidates existing approvals. The schema fingerprint must match this transformation-specific profile; ordinary source-free profiling still treats empty cells as missing values. Single-source review requires an existing valid policy. The separate transform-batch create --create-csv-policies workflow can prepare common-profile CSV policy proposals; it does not approve their decisions. Review never silently repairs a stale fingerprint or approves the revised policy.

Add --trace to the same read-only command for up to 50 row/column/rule ordinal events plus match counts. It reads only the fixed snapshot (at most 10,000 replacement cells), returns no source or replacement literals or value hashes, and fails closed above the limit. Unmatched cells are counted, not copied. This is debugging metadata, not a preview of output or approval.

Review alone does not authorize source preservation. The separate local approval and execution entrances below enforce the exact reviewed snapshot.

Transformation Execution

The registered implementation uses the same saved policy and reviewed snapshot_sha256 for transform-execute SOURCE POLICY DESTINATION --snapshot-sha256 SHA. Output format comes from the policy; destination must be a new bundle, never an existing directory or an input file. Success reports digest, provenance counts and output-budget metadata, not rows. The consumer uses transformation_completed inside the existing JSON result envelope; errors retain the existing error envelope and exit code.

Preservation additionally requires a matching restricted local receipt. Explicit substitute and replace_text use one contract for all fields: mapped values may equal other source values, including sensitive permutations. Review retains a value-free sensitivity note. Such values stay only in the selected local mixed-origin output, never CLI summaries or logs. This does not authorize direct preservation or a copying fallback.

Only transform-approve SOURCE POLICY RECEIPT --snapshot-sha256 SHA in a controlling terminal may create one after displaying the value-free review and receiving fresh APPROVE. Piped input, flags and agent approval cannot replace this confirmation. --json keeps the review/prompt visible on the terminal. Execution consumes that receipt with --receipt RECEIPT; changing source, policy, classification or referenced mapping/generation-policy bytes requires review and approval again. Sensitive, unknown or conflicting fields remain ineligible for preservation.

Run caps are --max-total-input-bytes and --max-output-bytes. They may lower, not silently raise, effective session/profile limits. A request above the ceiling and actual exhaustion are distinct typed failures with safe counters and supported recovery keys. Cleanup-incomplete errors require local inspection of destination/staging before retry; they do not permit overwrite. Database capture and private derive are not enabled by this entrance.

These commands are available in stable 1.6.0. The finite fictional transformation/session campaign completed for stable; see the current evidence and limits. Command discovery, fictional tests and release acceptance do not authorize a dataset. Changed activation still requires its safety/spec and exact-source review gates; ordinary generation stays source-free.

The existing decision wizard feeds this same execution contract: edit an already reviewable policy with transform-review SOURCE POLICY --decide --edit-actions, save with SAVE, then review the saved bytes again and use that digest for execution. Inline and local CSV substitution mappings follow the same workflow; CSV paths remain relative to the policy. Saving decisions does not issue a preservation receipt. If the saved policy retains any eligible source field, separate local transform-approve remains mandatory before CLI or MCP execution. The wizard does not create or repair an invalid initial policy or turn a sensitivity answer into approval.

Common profile workflow

Stable 1.6.0 provides transform-batch in the CLI and common_transformation in the running generator MCP server. Their operations are:

Operation Required local arguments Effect
create root, draft profile, configuration destination Save policies/spec/references only; optional --decide invokes the local wizard.
review root, saved profile Return a bounded value-free plan and exact digest.
approve root, saved profile, new receipt, --snapshot-sha256 Require fresh controlling-TTY confirmation; pipe/flag approval is rejected.
validate root, saved profile, --snapshot-sha256 Validate temporary output without retained publication.
execute root, saved profile, --snapshot-sha256 Execute temporarily, or retain a new direct-child bundle with --destination.
query-execute root, saved profile, configured query references, new destination Capture once, review and complete the separate configured SQL workflow.

All operations require explicit --max-total-input-bytes and --max-review-bytes; review/validate/execute/query-execute also require --max-output-bytes. Saved-profile validate/execute need --receipt when preserving values; query-execute obtains any required receipt locally within its owning session. Creation from absent CSV policies requires --create-csv-policies --seed INTEGER; initial proposals are unknown/drop-only, not approved decisions. --edit-actions/--edit-formats require --decide, and CSV creation with interactive decisions requires explicit action editing. Saving configuration changes reference paths and requires a fresh reviewed digest. Existing files, unsafe references and output destinations are not overwritten. Registration is not authorization to process a dataset. The finite fictional stable campaign completed with recorded limits; it does not issue preservation authority or clear changed activation.

Stable 1.6.0 exposes these operations under transform-batch. Its machine-readable success and error responses use CLI schema version 1.0; success metadata is inside result, not flattened into the envelope. Runtime --json/--debug flags are accepted before the command, before the operation, or after operation arguments. Debug does not disclose rejected arguments. Controlling-TTY approval remains local-only; create and SAVE never issue a receipt. Isolated CLI/MCP checks alone do not approve changed public activation.

Save the profile and field decisions, review the exact saved inputs, then validate against the returned digest. Obtain local approval only when the selected actions preserve values, and execute using that same digest and receipt. Editing an input, policy, mapping or output setting requires a fresh review; a receipt is not a reusable blanket approval. Agents can consume a matching receipt but cannot issue one.

For these workflows install stable 1.6.0 or a compatible later release. Installing 1.5.0 does not expose transform-batch. Keep ordinary source-free generation workflows separate and retain their existing schemas. A transformed bundle may deliberately contain preserved or explicitly mapped source values; it is not an anonymity guarantee. Protect its destination as potentially sensitive and inspect the value-free action/provenance report before sharing. SQL output is a script artifact and is never executed by transformation.

Edit Field Decisions Locally

transform-review SOURCE.csv POLICY.yaml --decide edits an existing valid policy in place. For every column it displays the value-free system comment, observed sensitivity and current action, then requires an explicit sensitive, non_sensitive, or unknown answer. There is no accept-all or default answer. Actions, mapping references, unmatched behavior and operator comments are retained unchanged; serialization may normalize YAML formatting.

The wizard displays the revised review and requires SAVE before atomically replacing the owner-only policy file. Invalid answers, conflicts, timeouts, or source/policy/mapping changes observed before saving leave the policy untouched. Each answer has a 60-second timeout, within the command's overall work budget. Input and prompt output must be terminals; piped answers are rejected. With --json, prompts remain on terminal stderr and the final policy_saved result is emitted on stdout. Run --trace separately afterward.

This is policy editing, not data approval: no receipt is issued, no rows are transformed, and a non_sensitive answer cannot override conflicting evidence or enable preservation. A future approval must revalidate and bind all exact snapshots again. The wizard currently requires a reviewable policy; it does not repair an invalid input policy.

Add --edit-actions with --decide to choose each field's action as well: keep retains its existing configuration; drop, preserve, synthesize, substitute, replace_text, and derive request the corresponding versioned policy action. There is no implicit choice. Preservation asks for a reference and rationale, synthesis for a local generation-policy reference, derivation for an expression and a JSON array of dependency names. These declarations are not proof of executable formula/generator validity or permission to retain source data.

For derive, review parses the existing bounded arithmetic syntax (1024 characters, 128 AST nodes) without evaluation. Referenced column names must exactly match the declared dependencies; missing, extra, dropped or cyclic dependencies reject. Aggregate calls, attribute access and unsupported syntax are not row-local formulas and reject. Columns literally named sum or count still count as ordinary dependencies when used without a function call. This preflight does not establish result types, null handling or DECIMAL rounding.

Substitution/replacement asks for mapping JSON using the policy's existing schema. For example, a local exact-text CSV table uses:

{"kind":"csv","path":"status.csv","source_columns":["old"],"replacement_columns":["new"]}

substitute also accepts inline entries or a reference to an existing shared domain. For replace_text, JSON null selects only the policy's existing file-wide table; a CSV mapping adds column rules. Existing top-level domains and file-wide declarations are not silently removed or edited. The wizard then asks for unmatched behavior: reject, preserve, or synthesize.

Private configuration prompts disable terminal echo on POSIX terminals and accept a bounded single line (less than 4096 UTF-8 bytes, also subject to the terminal's line-length limit). Use local CSV tables for larger dictionaries. No private configuration is included in the displayed review or JSON result. Newly referenced files are read under the same restricted policy directory and byte budgets, validated, and checked again before saving. Changed or invalid references abort without replacing the policy. --edit-actions without --decide is rejected.

Temporal field settings (configuration only)

Use transform-review SOURCE POLICY --decide --edit-formats to edit each field's temporal_type metadata. At the hidden JSON prompt, empty input keeps the current setting and null clears it. For example:

{"type":"datetime","format":"%Y-%m-%d","output_format":"%Y-%m-%dT%H:%M:%S%z","source_timezone":"Europe/Samara","target_timezone":"UTC"}

date fields do not accept timezone settings. Explicit SAVE writes a validated policy atomically; it creates neither approval nor output data. This setting does not itself execute temporal conversion. Literal replace_text always ignores field format/timezone metadata and supplies its replacement unchanged. A separately selected typed Parquet/SQL output may explicitly parse that result using its output field's temporal_type, including declared source/target timezones. CSV replacement output remains literal; no implicit UTC is applied.

Database Sources And SQL

Command Purpose Primary output
profile-postgres Profile an allowlisted read-only PostgreSQL source Profile JSON
profile-query Profile one reviewed PostgreSQL or Trino query as an aggregate-only virtual source Profile JSON
export-postgres-sql Generate and export executable PostgreSQL DDL and INSERT statements .sql file

Database connection, allowlist, JDBC-style endpoint, qualified wildcard, and query-source requirements are documented in the PostgreSQL and Trino guides.

Review-First Agent Flow

Command Purpose
agent-plan Profile and prepare a spec, then stop for review
agent-review Inspect a metadata-only approval checklist
agent-advise Ask an installed provider for validated spec changes
agent-advisor-request Export safe metadata for an external advisor
agent-advisor-apply Validate and apply an external proposal
agent-status Inspect phase and next action without changing state
agent-approve Generate from an exactly approved workspace
agent-recover Revalidate and finish an interrupted approval

See CLI Workflows for the command sequence and CLI Automation And JSON for stable response contracts.

Utilities

Command Purpose
examples Show complete examples for common workflows
doctor Check installation and run a temporary smoke generation
completion Generate completion for bash, zsh, fish, or PowerShell
audit-verify Verify an HMAC-authenticated MCP audit log

Aliases:

  • profile-csv-folder is an alias for profile-example;
  • generate-from-csv-folder is an alias for generate-from-example.

Configured SQL Sessions

test-data-agent transform-batch query-execute completes a single owned session. It reads a saved batch profile and a bounded reference file, captures each configured query once and displays metadata review. When preservation requires it, the command requests fresh controlling-TTY confirmation, then issues and consumes an owned temporary receipt for that frozen session. It validates before retaining output in a new directory in the parent workspace. Captured inputs and session receipts expire on command completion or failure. This is not an installed 1.5.0 command or dataset approval. For example, with authorization to read the configured source and an authored batch profile:

test-data-agent transform-batch query-execute /path/to/workspace batch.yaml queries.yaml selected-output \
  --max-total-input-bytes 67108864 --max-review-bytes 4194304 \
  --max-output-bytes 67108864 --json

The saved reference file can contain this fictional entry; its key must match a batch input source and its entity must match that input policy:

schema_version: "0.1"
queries:
  captured-orders.query:
    adapter: trino
    source_id: trino
    entity: fictional_orders
    query_file: orders.sql
    max_rows: 1000
    max_bytes: 8388608
    max_seconds: 60.0

Success writes one schema-1.0 JSON envelope to stdout; bounded value-free review goes to stderr before execution. Preservation confirmation uses the controlling terminal, not stdin or an approval flag. SQL output remains an artifact file; this command never executes generated SQL.

The reference file has schema_version: "0.1" and a queries mapping keyed by batch input source reference. Each entry contains adapter (postgres or trino), source_id, entity, query_file, max_rows, max_bytes and max_seconds. Query files are bounded relative workspace references. PostgreSQL source IDs must match administrator configuration; the configured Trino source ID is trino. Endpoints, credentials, callbacks and driver factories are not accepted in this file. Connection authority comes from existing administrator environment settings and allowlists.

Capture-specific Trino row and cumulative decoded-result byte limits use the explicit authorized max_rows and max_bytes; ordinary profiling defaults remain unchanged. Metadata retains its existing row ceiling, and statement, scan, time and transport limits still apply. A caller cannot raise capture limits above effective transformation policy/session ceilings.

The registered runtime configured_query_session MCP tool retains one frozen capture across review, validation and execution requests. It returns an opaque handle, digest and metadata for local CLI approval; MCP cannot issue approval. Session expiry, active-count and lifetime byte admission are bounded. Execution consumes the existing exact local receipt and publishes to a new child of the owning workspace. Expiry, refusal and shutdown clean the owned capture; cleanup failures refuse further work. The server owns the sessions and closes them in its shutdown handler, including transport failures. The module-level SDK object does not create sessions; custom embedding must explicitly inject and close its own session owner. See MCP session operations and session settings.