DATTA Prep — Visual Data Preparation

Preview — feature under development. Behavior, screens and contracts may change without notice between releases.

Preparing data for analysis used to mean scattered SQL scripts, intermediate spreadsheets and a ticket to the engineering team for every new column. With DATTA Prep — the DATTA platform's Power Query — you combine data from several sources (Trino, Neo4j, OpenSearch, JDBC databases, files) into a single semantic Model, by clicking, without writing DATTAX. Every click becomes a traceable step, and the result comes out ready for DATTA BI dashboards.

Open it from the DATTA BI›DATTA Prep menu.

Concept

DATTA Prep adopts the Power Query mental model, in three panels:

• Queries (left): each "table" in your Model is a query. You add several queries, each from a different source.
• Preview (center): a 100-row sample of the active query.
• Applied steps (right): each transformation (filter, sort, rename, remove, change type, calculated column, group by, limit) becomes an automatically generated DATTAX |> pipe.

When you click Salvar como Modelo, DATTA Prep:

1. Persists each query as a DATTA BI Dataset.
2. Creates a Model (DataModel) grouping the queries.
3. Links each Dataset to the Model.
4. Registers each table in the Knowledge Catalog, with:
   • Per-column statistics (minimum, maximum, average, distinct count, null count).
   • Per-column semantics (semantic type, kind, tags, description and glossary term, when there is one).
   • A reference to the source connection (identifier and name).

In other words: what you prepared becomes a governed, reusable asset — not a loose file. The routes used in those four steps are in the API reference.

Step by step

1. Connect to a source. Click + Nova fonte in the left panel. The dialog lists every dataset already discovered by the automatic cataloging of your connections — Trino, Neo4j, OpenSearch, JDBC. Select one and DATTA Prep generates the DATTAX FROM automatically, runs the preview and displays the table.
2. Apply transformations. Use:
   • Column header (...): filter, sort, rename, remove and change type.
   • Top menu: GROUP BY, LIMIT and MUTATE (calculated column).
3. Semantics. Click each column's semantic chip (inferred automatically from the name and the type). Edit the type (dimension, measure, identifier, date, email, pii...), kind, tags, description and glossary term.
4. Save as a Model. Define name, description, workspace and domain. Choose Batch (snapshot, on-demand materialization) or Streaming (all origins streamed via Kafka — see the section below). For streaming, indicate the WATERMARK column (timestamp or cursor for incremental loads).

Load modes

Batch (default)

• Each FROM <KIND> stays as it is — Trino SQL, Cypher MATCH, an index query. Materialization happens when the Model is referenced by a DATTA BI dashboard or by a scheduled DATTAX job.
• Generated DATTAX: ``dattax EVALUATE FROM TRINO "memory.datta_demo" "SELECT * FROM memory.datta_demo.vendas" |> FILTER "valor_total" > 100 |> ORDER BY "data_pedido" DESC ``
• Ideal use: periodic analyses (daily, weekly), dashboards with manual refresh, free exploration.

Streaming via Kafka

• Every batch source is rewritten to the equivalent FROM STREAM <KIND>, regardless of whether the origin is naturally streaming. The Kafka backbone unifies everything.
• Automatic mappings:

• WATERMARK is mandatory for incremental loads: it is the timestamp or cursor column the platform uses to know "how far it has already consumed". Without WATERMARK, the stream delivers the initial snapshot but cannot advance.
• How it works under the hood:
  • Trino/JDBC: the Debezium connector reads the transaction log and publishes to one Kafka topic per table (datta.cdc.<db>.<table>).
  • Neo4j: the cdc.query procedure in PULL mode (Neo4j 5.13+) emits events to a Kafka topic maintained by a helper component — an on-premises alternative with no Confluent dependency.
  • OpenSearch: has no native CDC. OPENSEARCH_POLL queries at an interval (30s by default) with range > last_watermark, publishes the new documents to Kafka and advances the watermark.
• Materialization: the platform consumes the Kafka topic, applies the DATTAX pipes (|> FILTER, |> GROUP BY, etc.) and pushes to the destination (MATERIALIZE TO TABLE, MATERIALIZE TO INDEX, MATERIALIZE TO GRAPH) in 5s micro-batches or in configured tumbling windows.

When to use each mode

Demo: cross-source Model

The demo-ws-vendas workspace already ships with two pre-registered Models that combine all 3 sources simultaneously:

• Modelo Demo Cross-Source (Vendas + Logs + Conexoes) — batch.
• Modelo Demo Streaming (Vendas + Logs + Conexoes via Kafka) — streaming.

Each Model aggregates:

• Trino: memory.datta_demo.vendas (60 orders).
• Neo4j: datta.Connection (the platform's connection registry — 11 connections).
• OpenSearch: otel-v1-apm-span-000001 (~911k telemetry records).

Use these Models as a starting point for DATTA BI dashboards or to validate the streaming pipeline consuming from Kafka.

Permissions

• DATAPREP_VIEW — open DATTA Prep and list sources (analyst+).
• DATAPREP_EXECUTE — run a query preview (analyst+).
• DATAPREP_SAVE — save a Model (analyst, steward, admin).
• DATAPREP_STREAMING — choose Streaming mode (steward, admin — Kafka consumption has an infrastructure cost that requires approval).

What to avoid

• Combining dozens of queries into a single Model: prefer small per-domain Models (vendas, clientes, logs) and relate them on the DATTA BI Models screen.
• Saving streaming without a WATERMARK: the stream delivers the initial snapshot but gets stuck on the next refresh.
• Editing the generated DATTAX by hand: DATTA Prep re-executes from the steps, and manual changes are overwritten. For free-form DATTAX, use the pipeline designer in DATTA Extract.