PT EN
Back to site

Datta Intelligent Query Layer — Discovery and Foundations

"Query intelligence" layers usually arrive with an uncomfortable demand: changing how people query. The Datta Intelligent Query Layer was designed for the opposite — it plugs into what the platform already has (Trino as the analytical engine, Neo4j as the graph, OpenSearch as the search and telemetry index, the ontology as the meaning layer) and improves queries by observation, asking nothing from whoever writes SQL. This page records the survey that gave rise to the layer: what existed before it, where it connects, the closed decisions and the mapped risks.


Name and navigation

  • Interface label: what used to appear as "SIQL" is now called "Intelligent Query Layer" (menu and trail; "Datta Intelligent Query Layer" remains the long-form name). The SIQL acronym remains the short name used internally — in the SIQL:ADMIN permission, in the dedicated graph database datta-siql-db, in the datta-siql.trino-events event topic and in this documentation.
  • Where it lives: the console is a feature of the Descobrir menu — DescobrirIntelligent Query Layer, alongside Knowledge Catalog and Explorar Dados. History: it was born in the Data Layer submenu (next to Neo4j, OpenSearch, the platform cache, Kafka, LLM/vLLM, Spark, Trino and MinIO), then became an item of the Sistema group under Configurar, and on 2026-08-07 it was promoted to a feature — it is a query-intelligence workbench (NL-to-SQL, learning, feature store), not just configuration.
  • Breadcrumb: DATTA | Descobrir | Intelligent Query Layer.
  • Effects of the change: opening the page highlights the Descobrir section of the side menu — no longer the Sistema group under Configurar; the corresponding card remains out of the Data Layer overview panel.

The changes were purely presentational: the page route (?view=settings-siql), the database, the permission and the internal names all stayed the same.


What the layer found already in place

The survey started from one principle: do not swap the analytical engine nor rewrite the platform — plug the intelligence into what already runs.

ComponentState found
Analytical engineTrino 480, with the iceberg catalog pointing to a Hive Metastore (embedded Derby) over HDFS, Parquet + Snappy format. There is no REST catalog nor Glue/Nessie. Auxiliary catalogs: hive (legacy Parquet tables) and memory
Query eventsNone. The engine did not publish execution events anywhere — neither to a topic nor over HTTP
GraphNeo4j in a 3-core causal cluster, with multi-database support — which made it possible to isolate the layer in its own database
Search and telemetry indexOpenSearch already in production, with index templates and lifecycle policies active; naming convention datta-<dominio>-* for indices with rollover
Event busKafka 3.8.0 (KRaft) with 3 brokers, available for asynchronous publishing
Semantic layerInvestigative ontology (entity types, relationships, semantic mappings, digital twins, OWL/SHACL import/export, entity resolution and materialization) and the Knowledge Catalog business glossary, both already in place
LLMLanguage models already integrated into the platform

Integration with the semantic layer was reserved for phase 5; in the MVP only the plug-in point was documented. LLM usage was out of scope for phase 1.


How the layer fits in

User SQL query
        |
        v
  Analytical engine (Trino 480)
        |
        |-- query event listener
        |        |-- publishes to the datta-siql.trino-events topic
        |        '-- (fallback) sends straight to the layer, over HTTP
        v
  Datta Intelligent Query Layer
        |- advisor (hot path, 20 ms budget)
        |- query event consumer
        '- snapshot refresher (daily; reads Iceberg metadata via Trino)
                 |
        +--------+-------------------+
        v                            v
  Metadata graph                Telemetry and analytics
  (datta-siql-db database)      (datta-siql-events-* indices)

The user keeps writing plain SQL — nothing changes in how they query. The layer receives query events and answers advisor requests through its own endpoints; see the API reference.

Coupling to the engine's optimizer (wrapping the connector metadata to consult the advisor during planning) was planned for phase 2 — in the MVP the layer only observes.


Closed decisions

DecisionValue
Official feature namedatta-intelligent-query-layer
Internal short nameSIQL
Service port8240
Dedicated graph databasedatta-siql-db — isolated from the others, less contention
Telemetry indexdatta-siql-events-<yyyy-MM-dd>, daily
Event topicdatta-siql.trino-events
Event publishing modeDual: the plugin tries the topic and, on failure, sends over HTTP straight to the layer
Default advisor modeSTATS_ONLY — returns only estimated rows and bytes, does not attempt read pruning. The RANKING and FULL modes arrive in later phases
PermissionSIQL:ADMIN
CacheA dedicated logical space in the platform cache, not shared with other modules
Configuration interfaceReuses the visual pattern of the platform's other configuration pages

Mapped risks and how they were handled

RiskHandling
The analytical engine had no query event listenerA dedicated plugin, compiled against the Trino 480 extension API, ships with the installation and starts publishing every execution event
The Iceberg catalog uses a Hive Metastore, with no REST catalogThe snapshot refresher reads metadata through the analytical engine itself, querying the $snapshots, $files, $manifests and $partitions system tables. It does not depend on an external REST catalog
The advisor must fit within 20 msTwo-tier cache (local memory + platform cache). The hot path is only a lookup by request hash; graph and index are touched only on a cache miss, in parallel and with 15 ms per source. If the budget overflows, the response falls back to a deterministic result
Additional load on the graph clusterThe layer writes only to its dedicated database, always in batches and in the background — never on the user's query path
Telemetry volume in the indexDaily index with a lifecycle of 7 days hot → 30 days warm → removal at 90 days. One shard and one replica in the MVP
Personal data in WHERE predicatesA masking stage runs before any persistence: literal values and the logged SQL go through masking
Dependency on the engine versionThe plugin is compiled against the stable version in production (480); an engine upgrade requires recompiling the plugin

Open questions

  • Iceberg REST catalog: the current installation uses the Hive Metastore. If it later migrates to Nessie or another REST catalog, the snapshot refresher will need to parameterize the client. It does not block the MVP.
  • Central connection registry: the platform requires every external source to be registered in the central registry. The advisor does not open a connection to any JDBC source — it only reads metadata through the analytical engine, the graph and the index, all internal — so there is no exception to the rule. If a later phase starts reading directly from an Iceberg REST catalog, that source must be registered normally.

Learn more

  • Layer specification — the value and scope in one page.
  • NL-to-SQL — questions in Portuguese become governed SQL.
  • API reference — the endpoints the platform exposes.