PT EN
Back to site

DATTABI — Statistical Analysis with R/CRAN

Time-series forecasting, statistical models, hypothesis testing: some analyses only the R ecosystem does well — and they used to require exporting data out of the platform. In DATTABI you apply any CRAN function, procedure or package directly to the connected datasets, without DATTA having to rewrite or wrap each function in DATTAX: R is a native analysis engine, you write idiomatic R and the result is materialized like any other dataset on the platform.

Implementation status: the R execution engine already exists; parts of the interface described here (the "Análise R" tab and the package panel) are still roadmap. This guide describes the target architecture.

How execution works

The platform keeps a set of GNU R processes with Rserve (the binary that serves R sessions over a TCP socket) and talks to them through the Java client (REngine/RserveCli). The managed R environment takes care of:

  • Per-worker isolation: one R process per worker — a failure in R never brings down the others nor the platform.
  • Parallelism: each run spawns N R sessions in parallel (one per dataset shard) using Java 25 structured concurrency. Large datasets are partitioned by the column you define (hash, range or auto-shard).
  • Parallel R libraries already installed: parallel, foreach, doParallel, future, data.table, Rcpp.
  • Data transfer: Java ↔ R data frames by direct mapping, with no JSON serialization; large datasets travel in streaming chunks.
  • On-premises: datta/r-runtime:<versão> images, based on r-base + Rserve, run inside your installation — nothing goes to the cloud.

Pure R, no intermediate translation

You write real R, importing any enabled CRAN package:

r
library(forecast)
# 'dataset' é injetado pelo DATTABI como data.frame
modelo <- auto.arima(dataset$valor)
previsao <- forecast(modelo, h = 12)
result <- data.frame(
  mes = seq.Date(from = max(dataset$data) + 30, by = "month", length.out = 12),
  previsto = as.numeric(previsao$mean)
)
result

There is no intermediate DATTAX translation: the LANG R { ... } block is recognized and passed verbatim to the R engine, annotated with the input datasets and the expected output type. This unlocks the complete CRAN ecosystem (tidyverse, caret, xgboost, forecast, prophet, survival, igraph, sf, rstan, tm and every other enabled package).

The Análise R tab (interface — partially roadmap)

In the dataset/dashboard editor, the "Análise R" tab brings together:

  • An R code editor (Monaco, R mode) with auto-complete for dataset$<coluna>.
  • A panel of the installed CRAN packages, with the Solicitar pacote (request package) action, subject to administrator approval.
  • A multi-source selector of the input datasets.
  • Output type: data.frame → materialized dataset; ggplot2 chart as SVG (rendered headless); .rds model; scalar value → metric.
  • Executar (preview) — runs over a 1,000-row sample and shows the result right away.
  • Materializar — enqueues the run in the refresh scheduler.
  • Copilot R — the AI generates the script from a description in Portuguese, using the dataset schema and the available packages.

R as a step in a DATTAX pipeline

Inside a DATTAX pipeline, the LANG R { ... } block inserts an R step between transforms — and the result goes on to materialization like any dataset:

dattax
DATASET vendas_mensal =
  FROM JDBC "Oracle_Vendas"
  |> GROUP BY ano, mes
  |> AGG total = SUM(valor)
  |> LANG R {
       library(forecast)
       result <- data.frame(mes = dataset$mes, total = dataset$total,
                             tendencia = ma(dataset$total, order = 3))
       result
     }
  |> MATERIALIZE AS ICEBERG TABLE "vendas_com_tendencia";

The block is an opaque transform for the interpreter: it is not validated by the DATTAX grammar, just passed through to the R engine. For large datasets, execution can be partitioned with LANG R PARTITION BY {coluna} CONCURRENCY {N} { ... }, which splits the dataset into N slices processed in parallel, with an optional REDUCE { ... } block to consolidate (without it, results are stacked with rbind).

Materializing the results

The returned data.frame is treated like any DATTAX dataset: the platform picks the destination — Iceberg (tabular default), Neo4j (from_id / to_id columns become nodes and edges) or OpenSearch (text/full-text). To override the choice, use the explicit form MATERIALIZE AS NEO4J DATABASE "..." LABEL "..." KEY "...".

CRAN packages under control

  • Catalog: the administrator manages the enabled packages in the R environment console (/configurar/r-runtime).
  • Installation: through a local CRAN mirror (miniCRAN) inside the installation — works 100% offline. The administrator approves, the installation runs on every worker and the catalog is updated.
  • Versioning: every package has a pinned version (reproducible results); upgrades require approval.
  • Blocking: the security team maintains a blocklist of unsafe packages (arbitrary execution, unrestricted network, inline compilation).

Security, permissions and audit

  • Isolated per-worker environment: restrictive system-call profile, no network egress (except the internal CRAN mirror and the platform cache), a read-only filesystem outside the temporary area — wiped on every run — and CPU and memory limits. system(), external download.file() and library() of unapproved packages are blocked.
  • Permissions (Project Guidelines §13): DATTABI_R_EXECUTE (run), DATTABI_R_PACKAGE_REQUEST (request a package) and R_RUNTIME_ADMIN (approve installations and blocks).
  • Audit: each run emits DATTABI.R.EXECUTED with actor, datasetIds, packagesUsed, rowsIn/Out, durationMs, peakMemoryMb and outcome, redacting strings that match credential patterns.

Cache, observability and operations

  • Cache (Project Guidelines §1): results of deterministic runs (same script, same datasets) are kept in the platform's two-tier cache under datta:dattabi:r:result:{hash} with a TTL — repeating the analysis answers instantly, with no re-execution.
  • Observability (Project Guidelines §11): each run produces the r.execute span with r.script_hash, r.packages, r.rows_in/out, r.partitions and r.runtime_version; R logs reach OpenSearch (otel-logs-*) correlated by trace_id.
  • Console (/configurar/r-runtime, Project Guidelines §8): workers, packages, job queue, per-run logs and usage metrics per user.