PT EN
Back to site

DATTABI — Statistical Analysis with R/CRAN

Time-series forecasting, statistical models, hypothesis testing: some analyses only the R ecosystem does well — and they used to require exporting data out of the platform. In DATTABI you apply any CRAN function, procedure or package directly to the connected datasets, without DATTA having to rewrite or wrap each function in DATTAX: R is a native analysis engine, you write idiomatic R and the result is materialized like any other dataset on the platform.

Implementation status: the R execution engine already exists; parts of the interface described here (the "Análise R" tab and the package panel) are still roadmap. This guide describes the target architecture.

How execution works

The platform keeps a set of GNU R processes with Rserve (the binary that serves R sessions over a TCP socket) and talks to them through the Java client (REngine/RserveCli). The managed R environment takes care of:

  • Per-worker isolation: one R process per worker — a failure in R never brings down the others nor the platform.
  • Parallelism: each run spawns N R sessions in parallel (one per dataset shard) using Java 25 structured concurrency. Large datasets are partitioned by the column you define (hash, range or auto-shard).
  • Parallel R libraries already installed: parallel, foreach, doParallel, future, data.table, Rcpp.
  • Data transfer: Java ↔ R data frames by direct mapping, with no JSON serialization; large datasets travel in streaming chunks.
  • On-premises: datta/r-runtime:<versão> images, based on r-base + Rserve, run inside your installation — nothing goes to the cloud.

Pure R, no intermediate translation

You write real R, importing any enabled CRAN package:

r
library(forecast)
# 'dataset' é injetado pelo DATTABI como data.frame
modelo <- auto.arima(dataset$valor)
previsao <- forecast(modelo, h = 12)
result <- data.frame(
  mes = seq.Date(from = max(dataset$data) + 30, by = "month", length.out = 12),
  previsto = as.numeric(previsao$mean)
)
result

There is no intermediate DATTAX translation: the LANG R { ... } block is recognized and passed verbatim to the R engine, annotated with the input datasets and the expected output type. This unlocks the complete CRAN ecosystem (tidyverse, caret, xgboost, forecast, prophet, survival, igraph, sf, rstan, tm and every other enabled package).

The Análise R tab (interface — partially roadmap)

In the dataset/dashboard editor, the "Análise R" tab brings together:

  • An R code editor (Monaco, R mode) with auto-complete for dataset$<coluna>.
  • A panel of the installed CRAN packages, with the Solicitar pacote (request package) action, subject to administrator approval.
  • A multi-source selector of the input datasets.
  • Output type: data.frame → materialized dataset; ggplot2 chart as SVG (rendered headless); .rds model; scalar value → metric.
  • Executar (preview) — runs over a 1,000-row sample and shows the result right away.
  • Materializar — enqueues the run in the refresh scheduler.
  • Copilot R — the AI generates the script from a description in Portuguese, using the dataset schema and the available packages.

R as a step in a DATTAX pipeline

Inside a DATTAX pipeline, the LANG R { ... } block inserts an R step between transforms — and the result goes on to materialization like any dataset:

dattax
DATASET vendas_mensal =
  FROM JDBC "Oracle_Vendas"
  |> GROUP BY ano, mes
  |> AGG total = SUM(valor)
  |> LANG R {
       library(forecast)
       result <- data.frame(mes = dataset$mes, total = dataset$total,
                             tendencia = ma(dataset$total, order = 3))
       result
     }
  |> MATERIALIZE AS ICEBERG TABLE "vendas_com_tendencia";

The block is an opaque transform for the interpreter: it is not validated by the DATTAX grammar, just passed through to the R engine. For large datasets, execution can be partitioned with LANG R PARTITION BY {coluna} CONCURRENCY {N} { ... }, which splits the dataset into N slices processed in parallel, with an optional REDUCE { ... } block to consolidate (without it, results are stacked with rbind).

Materializing the results

The returned data.frame is treated like any DATTAX dataset: the platform picks the destination — Iceberg (tabular default), Neo4j (from_id / to_id columns become nodes and edges) or OpenSearch (text/full-text). To override the choice, use the explicit form MATERIALIZE AS NEO4J DATABASE "..." LABEL "..." KEY "...".

CRAN packages under control

  • Catalog: packages come installed in the R runtime image (the families are listed in "The R runtime in the installation", below). The searchable list is in the Pacotes tab of the R runtime console; a new package comes with a new image.
  • Installation: through a local CRAN mirror (miniCRAN) inside the installation — works 100% offline. The administrator approves, the installation runs on every worker and the catalog is updated.
  • Versioning: every package has a pinned version (reproducible results); upgrades require approval.
  • Blocking: the security team maintains a blocklist of unsafe packages (arbitrary execution, unrestricted network, inline compilation).

Security, permissions and audit

  • Isolated per-worker environment: restrictive system-call profile, no network egress (except the internal CRAN mirror and the platform cache), a read-only filesystem outside the temporary area — wiped on every run — and CPU and memory limits. system(), external download.file() and library() of unapproved packages are blocked.
  • Permissions (Project Guidelines §13): DATTABI_R_EXECUTE (run), DATTABI_R_PACKAGE_REQUEST (request a package) and R_RUNTIME_ADMIN (approve installations and blocks).
  • Audit: each run emits DATTABI.R.EXECUTED with actor, datasetIds, packagesUsed, rowsIn/Out, durationMs, peakMemoryMb and outcome, redacting strings that match credential patterns.

The R runtime in the installation

Since September 2026 the R runtime is part of the reference installation: two replicas that grow with the load on their own (up to six), each with four R processes ready, and the same image whether R is used through the chat assistant, a DATTAX pipeline or the DATTABI copilot. Packages come preinstalled by family — data (tidyverse, data.table, arrow), time series (forecast, prophet, fable), machine learning (xgboost, ranger, glmnet, tidymodels), clustering and anomaly (dbscan, mclust, isotree), graphs (igraph), statistics (lme4, survival, mgcv) and text (text2vec, tm). Every run has a maximum time (60 s by default, up to 600 s on request); once exceeded, the process is terminated and the result explains what happened. A script error comes back with R's own message so you or the assistant can fix it and try again. How to enable, size and diagnose it is in the R runtime administration guide.

Cache, observability and operations

  • Cache (Project Guidelines §1): results of deterministic runs (same script, same datasets) are kept in the platform's two-tier cache under datta:dattabi:r:result:{hash} with a TTL — repeating the analysis answers instantly, with no re-execution.
  • Observability (Project Guidelines §11): each run produces the r.execute span with r.script_hash, r.packages, r.rows_in/out, r.partitions and r.runtime_version; R logs reach OpenSearch (otel-logs-*) correlated by trace_id.
  • Console (Project Guidelines §8): under Sistema › Data Layer › Runtime R, the administrator sees replicas and workers, the latest runs (who, outcome, duration, rows and the error message — without the script text), usage per user, packages and logs, and can test the runtime, restart it gradually and adjust the autoscaling.