DATTAX — Language Reference

Writing the same analysis three times — once in SQL, once in Cypher, once in the search index's query language — is rework that creates no value. DATTAX is DATTA's declarative language for querying, transforming and materializing data across any registered source (PostgreSQL, Neo4j, Iceberg, Kafka and dozens more): you write a single syntax and the platform translates it into the destination's dialect. This page is the complete language reference; for a guided introduction, start with the language guide.

Version 1.4.0 (2026-05). Semantic versioning — a minor bump adds backward-compatible functions/features; a major bump may break old scripts.

Script structure

LET windowDays = 30;
DEFINE DATASET vendas
  FROM JDBC "postgres-vendas"
  SELECT cliente_id, valor, data
  WHERE data >= NOW() - DAYS(windowDays);

EVALUATE
  vendas
  | GROUP BY cliente_id
  | AGG total = SUM(valor), pedidos = COUNT(*);

Top-level statements:

• LET nome = expr; — immutable local variable.
• DEFINE DATASET nome FROM <source> ... — declares a reusable dataset.
• DATASET nome = <pipeline>; — names a whole pipeline (with its |> steps) so you can reuse it in EVALUATE or MATERIALIZE.
• EVALUATE expr — final query; returns the rows to the caller (chart, materialization).
• MATERIALIZE nome FROM <source> ... — persists the result to a destination (Iceberg, Neo4j, the platform cache).

Universal Semantic Layer (USL)

You can reference entities by the business name defined in the ontology (e.g. Contribuinte) instead of the physical table or index name. Before interpreting the script, the platform textually substitutes the conceptual tokens with their resolved physical references:

Token format: ${(entity|conn):(<ontologyId>.)?<Nome>}.

How resolution works. The platform queries the ontology catalog and gets the entity reference back — the physical dataset plus the matching connection (or name space). The result is kept in a two-tier cache (local memory + the platform's distributed cache) under the key datta:dattax:catalog:ontology:<ontologyId>:<entity>, with a 30-minute TTL (datta.dattax.cache.ontology-ttl-minutes) — scripts that repeat the same entities don't pay the resolution cost again.

Safe behavior:

• An unresolved token is preserved textually (with a warning in the execution log) — the script fails informatively instead of silently generating incorrect SQL/Cypher.
• With the ontology disabled, preprocessing does nothing: the script runs exactly as you wrote it.

Sources (FROM <source>)

Every source is referenced by the connection name registered under Sistema›Conexões — addresses and credentials never appear in the script.

Transforms

The pipe | chains transforms over the dataset:

There are 50+ additional transforms defined in the language grammar — the ones you use day to day are in the language guide.

JOIN types (1.2.0+)

EVALUATE clientes
  |> JOIN ANTI pedidos ON id = cliente_id ;
-- Retorna clientes que nunca compraram.

Filter operators (1.2.0+)

Besides =, !=, <, >, <=, >=, AND, OR and NOT, the language accepts the SQL-like operators below, with precedence between arithmetic and comparison (i.e. x + 1 IS NULL parses as (x + 1) IS NULL):

Example (§5.7 of the functional specification):

EVALUATE FROM JDBC "warehouse"
    "SELECT * FROM vendas"
  |> FILTER status IN ("concluido", "pago")
       AND valor BETWEEN 100 AND 5000
       AND cancelado_em IS NULL
       AND cliente_id IS NOT NULL
       AND NOT_CONTAINS(observacao, "teste") ;

UNION — schema validation

UNION emits a non-fatal warning (UNION_SCHEMA_MISMATCH) when the two sides have different column sets. Execution continues: the missing columns are filled with NULL in the target row. The run finishes with status "completed with warnings" — the warning counter of the final event comes back greater than zero.

Data Quality — |> DATA_QUALITY (1.3.0+)

Fulfills §5.14 of the functional specification. Runs declarative checks over the current dataset and emits non-fatal warnings for each violation. Each warning has a fixed code the interface uses for color and icon.

EVALUATE FROM JDBC "warehouse" "SELECT * FROM vendas"
  |> DATA_QUALITY
       CHECK NOT NULL cliente_id,
       CHECK UNIQUE pedido_id,
       CHECK RANGE valor BETWEEN 0 AND 1000000,
       CHECK PCT_NULL desconto < 10,
       CHECK EXPECTED ROWS BETWEEN 100 AND 5000000,
       CHECK CUSTOM "data futura" data_venda <= NOW()
  |> SELECT * ;

DQ is observational — it does not filter. To drop invalid rows, combine it with FILTER. The dataset passes untouched to the next stage.

Granular execution states — 1.3.0+

Fulfills §5.16 of the specification. Every run reports progress on screen in real time, stage by stage:

At the end, the completion event carries the number of accumulated warnings: if greater than zero, the run is shown as "completed with warnings" — you know it finished, and you know the warnings are worth a look.

Persistence modes — MATERIALIZE ... MODE ... PARTITION BY ... (1.3.0+)

Fulfills §5.15. Syntax:

MATERIALIZE vendas_consolidadas AS ICEBERG TABLE "gold.vendas"
  MODE INCREMENTAL
  PARTITION BY ano, mes
  WATERMARK data_venda ;

Supported modes (each destination maps to the engine's native semantics):

PARTITION BY is forwarded to destinations that support partitioning (Iceberg, Parquet/Delta). Destinations without support (plain OpenSearch index, Neo4j label) emit the MATERIALIZE_PARTITION_IGNORED warning and proceed with the default.

MATERIALIZE ... AS NEO4J — target database required

Persistence to Neo4j never guesses the destination database. Opening a write session tied to a silent "default" database is a recipe for data in the wrong place: the destination must be an explicit decision.

Explicit syntax

The DATABASE clause is mandatory, followed by the label and the merge key — all required:

materializeNeo4j : 'NEO4J' 'DATABASE' STRING 'LABEL' IDENT 'KEY' IDENT ;

In practice, the script provides the database as a string literal, the label as an identifier and the key property used in the MERGE:

MATERIALIZE grafo_partes AS NEO4J
  DATABASE "processotributario"
  LABEL Parte
  KEY documento
  MODE UPDATE ;

Omitting DATABASE is a syntax error — the script is rejected before any execution, because the clause is not optional in the grammar.

Runtime validation (empty database)

Even if the value arrives empty (null or blank) at runtime, the write is rejected before opening any graph session, with this exact message:

Banco de destino da materializacao Neo4j e obrigatorio — nunca adivinhado.

There is no fallback to "neo4j" or to any other name. The label (validated against the [A-Za-z_][A-Za-z0-9_]* pattern) and the merge key are checked in the same step, before the write.

MATERIALIZE ... AS AUTO — the platform picks the engine

When you write MATERIALIZE <ds> AS AUTO, the platform scores the available engines (Iceberg, Neo4j, OpenSearch) and picks the one that best fits the result's shape. If Neo4j wins, the result is deposited in the graph's default system database ("neo4j"). This is not a silent fallback: an AUTO materialization is an artifact derived by the platform itself, with no domain context chosen by you, so the default destination is a deliberate, documented choice — distinct from uploads, which always carry an explicit destination context. For a specific destination, use the explicit form with DATABASE.

Standard Library (F11+)

Functions callable in any expression — around 40 in version 1.1.0, organized into 4 categories. The DATTAX editor offers autocomplete for all of them.

Math (category: math)

String (category: string)

Date (category: date)

Accepts an ISO-8601 string, an instant, a date or epoch millis. Default timezone America/Sao_Paulo.

Time Intelligence (category: time-intelligence, 1.1.0+)

Functions over time series. They take (rows, dateColumn, valueColumn, refDate?) and operate on a window determined by the function.

Measures such as YoY and Q vs Q-1 come ready, without you rewriting the query — the same comfort you expect from Power BI or Tableau. For the complete map of DAX equivalences, see DATTAX × DAX.

Hands-on example — full pipeline with quality checks and materialization

Scenario: consolidate the valid sales from the warehouse into an Iceberg gold table, with quality checks along the way.

1. Open the DATTAX editor.
2. Confirm the warehouse connection exists under Sistema›Conexões.
3. Paste and run the script — it declares the consolidated dataset and then materializes that dataset into the gold table:

DATASET vendas_mensais =
  FROM JDBC "warehouse" "SELECT * FROM vendas"
  |> FILTER status IN ("concluido", "pago") AND cancelado_em IS NULL
  |> DATA_QUALITY
       CHECK NOT NULL cliente_id,
       CHECK UNIQUE pedido_id,
       CHECK PCT_NULL desconto < 10
  |> MUTATE ano = YEAR(data_venda), mes = MONTH(data_venda)
  |> GROUP BY ano, mes SUMMARIZE total = SUM(valor), pedidos = COUNT(*),
       ultima_venda = MAX(data_venda) ;

MATERIALIZE vendas_mensais AS ICEBERG TABLE "gold.vendas_mensais"
  MODE INCREMENTAL
  PARTITION BY ano, mes
  WATERMARK ultima_venda ;

1. Follow the stages (connecting → extracting → transforming → materialize) in real time. If a quality check fails, the run ends as "completed with warnings" — the warnings are listed, but the table is written.

Versioning and compatibility

• The current standard library version is 1.1.0, and the platform loads only functions whose introduction version is less than or equal to it.
• Old scripts in production keep running after upgrades — new functions arrive in a minor bump, without breaking what already exists.
• Deprecated functions produce warnings at run time but remain available until the next major bump. You have time to migrate at your own pace.
• A new function is added by declaring an annotated static method in the standard library package — the catalog discovers it on startup, with no grammar edits (functions go through the generic funcCall that already exists).

Language evolution

• 1.3.0 — ANTI JOIN; SQL-like operators (IS NULL, BETWEEN, IN, NOT_CONTAINS, ENDS_WITH); UNION with schema validation; non-fatal warnings and "completed with warnings"; granular execution states (connecting/extracting/transforming/persisting); the DATA_QUALITY transform with 9 check types (§5.14); persistence modes (MODE, PARTITION BY, WATERMARK, §5.15); Excel (.xlsx) as a file source (§5.3); pause, retries and configurable timezone in the scheduler (§5.17); PII masking in the designer (§5.19); duplicate step (§5.6); canvas cards with visible schema + undo/redo.
• 1.4.0 (current) — editor with syntax highlight and autocomplete for the whole standard library and the main commands (EVALUATE / FROM JDBC / FILTER / JOIN / GROUP BY / DATA_QUALITY / MATERIALIZE), fed by the standard library catalog the platform publishes (see the API reference); correct per-dialect quoting when pushing queries down to the source (MySQL ` backticks , MSSQL [brackets], BigQuery backticks , Oracle/PostgreSQL "double quotes"); folder monitoring with glob patterns (/data/incoming/.csv, /data//.parquet); execution metrics (sources accessed, warnings, error stage, discarded records) persisted in Neo4j and visible in the pipeline monitor; the complete pipeline lifecycle (pause, resume, archive, activate — also available through the **API reference**); email/webhook notification templates with variables ({{pipelineName}}, {{status}}`); conversion of the visual pipeline into a DATTAX script, closing the loop between canvas and language.
• Still open for 1.4.0 — CALCULATE(expr, modifiers...) and formalized filter contexts (RowContext vs FilterContext, the DAX equivalent); query plan visualizer; visual JOIN/edge configuration dialog in the designer; automatic 1:1 / 1:N / N:1 / N:N cardinality.
• 2.0.0 (planned) — full filter pushdown (§5.7 — WHERE translated into the source's dialect); write modes fully implemented on every destination (not just in the grammar); production execution interface; new source families (GraphQL, SaaS, cloud storage, ODBC); optional static typing; query optimizer with common subexpression elimination.