DATTAX — Language Guide

DATTAX is DATTA's query and transformation language: a single syntax to read and join graph, table, index, file, warehouse and stream data. Before it, every answer meant mastering one dialect per engine — SQL in the warehouse, Cypher in the graph, a custom query in the search index, plus a framework for live events. With DATTAX you write one pipeline, with steps chained by the |> operator, and the platform translates each step to the right source — so you spend your time answering questions instead of switching tools.

This guide introduces the language from the script writer's point of view. It complements the full language reference, the graph algorithm catalog and the streaming guide.

1. Overview

DATTAX is inspired by Unix pipelines and by languages such as KQL/PRQL: each step receives a dataset and produces another, composed with the |> operator.

dattax

EVALUATE FROM JDBC "warehouse" "SELECT id, total, created_at FROM vendas"
  |> FILTER total > 0
  |> MUTATE year = YEAR(created_at)
  |> GROUP BY year AGGREGATE total_year = SUM(total)
  |> SELECT year, total_year
  ;

Statements end with ;. A script can contain multiple statements.

1.1 Top-level keywords

EVALUATE — executes and returns the dataset to the caller.
DATASET <nome> = <pipeline> ; — names a reusable dataset.
LET <ident> = <expr> ; — local constant.
DEFINE MEASURE <nome> = <expr> ; — reusable measure (sum, avg, etc.).
MATERIALIZE <pipeline> AS ICEBERG|NEO4J|OPENSEARCH [OPTIONS ...] ; — persists the result to the chosen destination.

1.2 Types

Scalars: INT, LONG, DOUBLE, STRING, BOOL, TIMESTAMP, DATE.
Graph column: NODE, REL, PATH.
Structured: LIST<T>, MAP<K,V>.
Vector: VECTOR<dim> (for embeddings / KNN).

Implicit conversions follow SQL rules. Explicit conversion via CAST(x AS INT).

2. Where the data comes from (`FROM ...`)

Syntax	Source
`FROM GRAPH "conn"`	Neo4j (Cypher + graph algorithms)
`FROM INDEX "conn"`	OpenSearch/Elasticsearch
`FROM JDBC "conn" "<sql>"`	Any cataloged JDBC database, with the vendor dialect applied
`FROM TRINO "conn" "<sql>"`	Trino (SQL federation)
`FROM ICEBERG "conn" "<table>"`	Apache Iceberg tables (REST catalog)
`FROM FILE "<path>"`	Files in the platform storage (CSV/Parquet/JSON/TSV)
`FROM STREAM <KIND> "conn"`	Kafka/Pulsar/Debezium/Neo4j CDC
`FROM CASSANDRA "conn" "<cql>"`	Cassandra/ScyllaDB (CQL)
`FROM MONGO "conn" { <pipeline> }`	MongoDB (aggregation pipeline)
`FROM BIGQUERY "conn" "<sql>"`	Google BigQuery (reads accelerated by the Storage API)

All connections are names, not URLs. You register the source once in SistemaConexões and reference it by name in any script: at run time the platform resolves the name and loads that user's credential from the credential vault. Addresses and passwords never appear in the code.

Examples:

dattax

EVALUATE FROM GRAPH "neo4j-main" "MATCH (n:Processo) RETURN n LIMIT 100" ;

EVALUATE FROM INDEX "opensearch-logs" INDEX "app-*"
  QUERY { "match": { "message": "error" } } LIMIT 1000 ;

EVALUATE FROM MONGO "mongo-prod" COLLECTION "orders" {
  { "$match": { "status": "paid" } },
  { "$group": { "_id": "$region", "total": { "$sum": "$amount" } } }
} ;

EVALUATE FROM BIGQUERY "bq-analytics"
  "SELECT country, COUNT(*) n FROM `ds.sessions` GROUP BY country" ;

3. How to transform the data

3.1 Tabular basics

Operator	Action
`FILTER <expr>`	Row-by-row filter (SQL-like)
`SELECT col1, col2 AS alias`	Projection / rename
`MUTATE col = <expr>`	New computed column (Power Query style)
`DROP col1, col2`	Column removal
`RENAME old TO new`	Explicit renaming
`GROUP BY ... AGGREGATE ...`	Aggregation
`ORDER BY col [ASC\	DESC]`
`TAKE <n>` / `SKIP <n>`	Pagination
`DISTINCT [col1, ...]`	Deduplication
`UNION <dataset>`	Vertical concatenation

3.2 Join

dattax

|> JOIN outroDs ON a.id = outroDs.id              -- inner
|> LEFT JOIN outroDs ON ...                       -- left outer
|> JOIN outroDs AS LOOKUP ON col = outroDs.key   -- enrichment (O(1))

AS LOOKUP is mandatory in streaming pipelines.

3.3 Window

dattax

|> WINDOW PARTITION BY region ORDER BY ts
     RANK() AS rank,
     LAG(valor, 1) AS valor_ant,
     SUM(valor) OVER (ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) AS media_7d

3.4 Pivot / Bucket

dattax

|> PIVOT categoria AGGREGATE total = SUM(valor)
|> BUCKET valor INTO 10                         -- quantile bucketing
|> BUCKET valor INTO [0, 100, 500, 1000]        -- edges explicitas

3.5 Match (Cypher subgraphs)

When the source is FROM GRAPH, MATCH accepts a full Cypher pattern:

dattax

EVALUATE FROM GRAPH "neo4j-main"
  |> MATCH (p:Processo)-[:TEM_PARTE]->(parte:Parte {tipo: "autor"})
  |> WHERE p.valor_causa > 10000
  |> RETURN p.numero AS processo, parte.nome AS autor, p.valor_causa
  ;

3.6 KNN (vector)

Two uses:

KNN on an OpenSearch/Elasticsearch index (document vector search, not a graph algorithm): ``dattax EVALUATE FROM INDEX "opensearch-docs" |> KNN FIELD "embedding" K 10 VECTOR [0.1, 0.2, ...] ``
KNN on a graph (similarity algorithm): ``dattax |> KNN ON myGraph K=10 ``

4. Graph algorithms (Neo4j Graph Data Science)

More than 30 algorithms from the Neo4j GDS open source catalog. All of them run in stream mode, read-only — a pipeline never changes the original graph. Leiden and SLLPA belong to the Enterprise edition and are blocked on the platform: trying to use them stops execution with a clear message in Portuguese.

4.1 Centrality

PAGERANK — global authority.
ARTICLE_RANK — variant that penalizes hubs.
EIGENVECTOR — principal eigenvector.
BETWEENNESS — betweenness.
CLOSENESS — closeness.
HARMONIC — harmonic closeness (robust on disconnected graphs).
DEGREE (IN/OUT/BOTH).
CELF — influence maximization.

4.2 Community

LOUVAIN (via COMMUNITY ALGO=LOUVAIN).
LABEL_PROPAGATION.
WCC (weakly connected components).
SCC (strongly connected components).
TRIANGLE_COUNT.
LOCAL_CLUSTERING.
KCORE.
KMEANS (over an existing embedding).
MODULARITY (modularity optimization).

4.3 Pathfinding

SHORTEST PATH (Dijkstra).
ASTAR (A* with lat/lon heuristic).
YENS FROM ... TO ... K N (top-K shortest).
ALL_SHORTEST_PATHS.
BFS FROM ... MAX_DEPTH=N.
DFS FROM ....
RANDOM_WALK WALK_LENGTH=..., WALKS_PER_NODE=....

4.4 Similarity

NODE_SIMILARITY SIMILARITY_CUTOFF=....
FILTERED_NODE_SIMILARITY.
KNN K=....
FILTERED_KNN K=....

4.5 Embeddings

FASTRP EMBEDDING_DIMENSION=..., ITERATIONS=....
HASHGNN ITERATIONS=..., EMBEDDING_DIMENSION=....
NODE2VEC WALK_LENGTH=..., WALKS_PER_NODE=..., EMBEDDING_DIMENSION=....
GRAPHSAGE MODEL "nome" (inference with a pre-trained model).

4.6 Link prediction (topological)

ADAMIC_ADAR, COMMON_NEIGHBORS, PREFERENTIAL_ATTACHMENT, RESOURCE_ALLOCATION, SAME_COMMUNITY, TOTAL_NEIGHBORS.

dattax

|> LINK_PREDICTION ADAMIC_ADAR BETWEEN "elem-a" AND "elem-b"

Syntax and options for each algorithm: graph algorithm catalog.

Each algorithm result is cached for 1 hour, identified by the combination of projection, algorithm and parameters — repeating the same run answers in seconds.

5. AI primitives

Primitives built into the pipeline: the intelligence configured on the platform processes free text row by row, without you leaving the script.

5.1 `LLM EXTRACT`

Extracts structured fields from free text.

dattax

EVALUATE FROM JDBC "warehouse" "SELECT id, body FROM tickets"
  |> LLM EXTRACT FROM body INTO {
       categoria: "categoria principal do ticket",
       prioridade: "baixa|media|alta",
       valor_mencionado: "numero em reais se houver, senao null"
     }

5.2 `LLM CLASSIFY`

Multiclass classification with confidence:

dattax

|> LLM CLASSIFY FROM texto LABELS ["spam", "relevante", "duplicado"] AS rotulo

5.3 `LLM EMBED`

Generates an embedding with the platform model (Qwen3-Embedding-0.6B by default) or another configured one:

dattax

|> LLM EMBED FROM descricao AS emb MODEL "Qwen/Qwen3-Embedding-0.6B"

There is a per-user quota and a rate limit: when exceeded, the script stops with a clear message in Portuguese. Repeated calls over the same text are answered from the platform cache, keyed by the text hash.

6. Streaming

Additive primitives — batch scripts are not affected.

dattax

-- Pedidos ao vivo enriquecidos com usuarios (batch dim):
DATASET users = FROM JDBC "warehouse" "SELECT id, nome, tier FROM users"
  |> CACHE AS LOOKUP REFRESH EVERY "30m" ;

EVALUATE FROM STREAM KAFKA "orders" TOPIC "orders" STARTING FROM LATEST
  |> PARSE TIMESTAMP event_ts
  |> WATERMARK ON event_ts DELAY "5s"
  |> JOIN users AS LOOKUP ON user_id = id
  |> FILTER tier = "premium"
  |> WINDOW TUMBLING "30s"
  |> STREAM OUTPUT ;

Primitives:

FROM STREAM KAFKA|PULSAR|CDC|NEO4J_CDC|OPENSEARCH_POLL "<conn>"
PARSE TIMESTAMP <col> — marks the event-time.
WATERMARK ON <col> DELAY "<dur>" — tolerates out-of-order events.
WINDOW TUMBLING "<dur>" / SLIDING "<dur>" SLIDE "<dur>" / SESSION "<gap>".
CACHE AS LOOKUP [REFRESH EVERY "<dur>"].
JOIN <ds> AS LOOKUP ON <eq>.
STREAM OUTPUT — terminal.

Details in the streaming guide.

7. Materialization

dattax

MATERIALIZE
  FROM GRAPH "neo4j-main"
  |> MATCH (p:Processo)-[:TEM_PARTE]->(parte)
  |> GROUP BY p.tribunal AGGREGATE qtd = COUNT(*)
AS ICEBERG TABLE "analytics.processos_por_tribunal"
OPTIONS (WRITE_MODE = "OVERWRITE", REFRESH_POLICY = "CRON \"0 0 2 * * *\"") ;

Destinations:

ICEBERG — Apache Iceberg tables on S3-compatible storage (MinIO).
NEO4J — reverse graph (creates / updates derived labels).
OPENSEARCH — analytical index.

The platform picks the write plan (MERGE, OVERWRITE, APPEND) and records the lineage in the Knowledge Catalog as (:Dataset)-[:DERIVADO_DE]->(:Dataset) — you always know where each dataset came from.

8. Measures (DEFINE MEASURE)

Measures are reusable named expressions:

dattax

DEFINE MEASURE total_vendas = SUM(valor) ;
DEFINE MEASURE ticket_medio = AVG(valor) ;

EVALUATE FROM JDBC "warehouse" "SELECT * FROM vendas"
  |> GROUP BY regiao
     AGGREGATE
       vendas = [total_vendas],
       ticket = [ticket_medio]

9. Practical examples

To try any of the examples below:

Open the platform's DATTAX editor.
Confirm that the connections cited in the script exist in SistemaConexões — swap the names for your own connections.
Paste the script, run it and follow the result as a table on the screen.

9.1 Analytical dashboard (warehouse + graph)

dattax

-- Processos por tribunal com centralidade de partes
DATASET centralidades =
  FROM GRAPH "neo4j-main"
  |> PROJECT GRAPH g NODES "..." RELS "..."
  |> PAGERANK ON g ITER=20 ;

EVALUATE FROM JDBC "warehouse" "SELECT id, tribunal FROM processos"
  |> JOIN centralidades AS LOOKUP ON id = nodeId
  |> GROUP BY tribunal
     AGGREGATE
       qtd = COUNT(*),
       score_medio = AVG(score)
  |> ORDER BY score_medio DESC
  ;

9.2 AI enrichment of free text

dattax

EVALUATE FROM JDBC "crm" "SELECT id, feedback FROM tickets WHERE data >= CURRENT_DATE - 30"
  |> LLM EXTRACT FROM feedback INTO {
       sentimento: "positivo|neutro|negativo",
       topico: "tema principal",
       nps_inferido: "0-10"
     }
  |> GROUP BY topico, sentimento
     AGGREGATE qtd = COUNT(*)
  ;

9.3 Real-time fraud detection

dattax

DATASET blacklist = FROM JDBC "compliance" "SELECT cpf FROM lista_negra"
  |> CACHE AS LOOKUP REFRESH EVERY "5m" ;

EVALUATE FROM STREAM KAFKA "transacoes" TOPIC "transacoes.v1"
  |> PARSE TIMESTAMP ts
  |> WATERMARK ON ts DELAY "3s"
  |> JOIN blacklist AS LOOKUP ON cpf_pagador = cpf
  |> FILTER blacklist.cpf IS NOT NULL
  |> WINDOW TUMBLING "1m"
  |> GROUP BY cpf_pagador AGGREGATE qtd = COUNT(*)
  |> FILTER qtd > 3
  |> STREAM OUTPUT ;

9.4 Semantic KNN in OpenSearch

dattax

LET query_emb = LLM EMBED_VALUE "contrato inadimplente valor alto" MODEL "Qwen/Qwen3-Embedding-0.6B" ;

EVALUATE FROM INDEX "opensearch-processos"
  |> KNN FIELD "embedding" K 20 VECTOR query_emb
  |> SELECT id, titulo, score
  ;

9.5 Batch embeddings + materialization

dattax

MATERIALIZE
  FROM JDBC "warehouse" "SELECT id, descricao FROM produtos"
  |> LLM EMBED FROM descricao AS emb MODEL "Qwen/Qwen3-Embedding-0.6B"
AS OPENSEARCH INDEX "produtos-emb"
OPTIONS (MAPPING = "knn_vector:emb:1024") ;

10. Limits and best practices

Non-materializable streams: FROM STREAM ... |> MATERIALIZE is rejected. Use STREAM OUTPUT + an external destination.
AI on large volumes: respect the quota and process in batches. LLM EXTRACT over 1M rows without a quota stops with a clear message in Portuguese.
Graph algorithms on giant graphs: use PROJECT GRAPH with a subset via Cypher MATCH ... WHERE .... Avoid global projections.
Automatic cache: every read goes through a cache (local memory + the platform's distributed cache); invalidation happens on its own when you materialize or change the source.
Large scripts: run EXPLAIN before EVALUATE when available.
Timeouts: each source has an explicit timeout (30s by default). Adjust it with OPTIONS (TIMEOUT = "60s").

11. Going further

Full language reference — grammar, standard library, versioning.
Graph algorithm catalog — syntax and options for each algorithm.
Streaming — windows, watermarks and real-time enrichment.
DATTAX × DAX — for those coming from Power BI.
API reference — to trigger the platform from your own systems.

DATTAX — Language Guide

1. Overview

1.1 Top-level keywords

1.2 Types

2. Where the data comes from (FROM ...)

3. How to transform the data

3.1 Tabular basics

3.2 Join

3.3 Window

3.4 Pivot / Bucket

3.5 Match (Cypher subgraphs)

3.6 KNN (vector)

4. Graph algorithms (Neo4j Graph Data Science)

4.1 Centrality

4.2 Community

4.3 Pathfinding

4.4 Similarity

4.5 Embeddings

4.6 Link prediction (topological)

5. AI primitives

5.1 LLM EXTRACT

5.2 LLM CLASSIFY

5.3 LLM EMBED

6. Streaming

7. Materialization

8. Measures (DEFINE MEASURE)

9. Practical examples

9.1 Analytical dashboard (warehouse + graph)

9.2 AI enrichment of free text

9.3 Real-time fraud detection

9.4 Semantic KNN in OpenSearch

9.5 Batch embeddings + materialization

10. Limits and best practices

11. Going further

2. Where the data comes from (`FROM ...`)

5.1 `LLM EXTRACT`

5.2 `LLM CLASSIFY`

5.3 `LLM EMBED`