Datta Intelligent Query Layer — Discovery and Foundations
"Query intelligence" layers usually arrive with an uncomfortable demand: changing how people query. The Datta Intelligent Query Layer was designed for the opposite — it plugs into what the platform already has (Trino as the analytical engine, Neo4j as the graph, OpenSearch as the search and telemetry index, the ontology as the meaning layer) and improves queries by observation, asking nothing from whoever writes SQL. This page records the survey that gave rise to the layer: what existed before it, where it connects, the closed decisions and the mapped risks.
Name and navigation
- Interface label: what used to appear as "SIQL" is now called "Intelligent Query Layer" (menu and trail; "Datta Intelligent Query Layer" remains the long-form name). The SIQL acronym remains the short name used internally — in the
SIQL:ADMINpermission, in the dedicated graph databasedatta-siql-db, in thedatta-siql.trino-eventsevent topic and in this documentation. - Where it lives: the console is a feature of the Descobrir menu — , alongside Knowledge Catalog and Explorar Dados. History: it was born in the Data Layer submenu (next to Neo4j, OpenSearch, the platform cache, Kafka, LLM/vLLM, Spark, Trino and MinIO), then became an item of the Sistema group under Configurar, and on 2026-08-07 it was promoted to a feature — it is a query-intelligence workbench (NL-to-SQL, learning, feature store), not just configuration.
- Breadcrumb:
DATTA | Descobrir | Intelligent Query Layer. - Effects of the change: opening the page highlights the Descobrir section of the side menu — no longer the Sistema group under Configurar; the corresponding card remains out of the Data Layer overview panel.
The changes were purely presentational: the page route (?view=settings-siql), the database, the permission and the internal names all stayed the same.
What the layer found already in place
The survey started from one principle: do not swap the analytical engine nor rewrite the platform — plug the intelligence into what already runs.
| Component | State found |
|---|---|
| Analytical engine | Trino 480, with the iceberg catalog pointing to a Hive Metastore (embedded Derby) over HDFS, Parquet + Snappy format. There is no REST catalog nor Glue/Nessie. Auxiliary catalogs: hive (legacy Parquet tables) and memory |
| Query events | None. The engine did not publish execution events anywhere — neither to a topic nor over HTTP |
| Graph | Neo4j in a 3-core causal cluster, with multi-database support — which made it possible to isolate the layer in its own database |
| Search and telemetry index | OpenSearch already in production, with index templates and lifecycle policies active; naming convention datta-<dominio>-* for indices with rollover |
| Event bus | Kafka 3.8.0 (KRaft) with 3 brokers, available for asynchronous publishing |
| Semantic layer | Investigative ontology (entity types, relationships, semantic mappings, digital twins, OWL/SHACL import/export, entity resolution and materialization) and the Knowledge Catalog business glossary, both already in place |
| LLM | Language models already integrated into the platform |
Integration with the semantic layer was reserved for phase 5; in the MVP only the plug-in point was documented. LLM usage was out of scope for phase 1.
How the layer fits in
User SQL query
|
v
Analytical engine (Trino 480)
|
|-- query event listener
| |-- publishes to the datta-siql.trino-events topic
| '-- (fallback) sends straight to the layer, over HTTP
v
Datta Intelligent Query Layer
|- advisor (hot path, 20 ms budget)
|- query event consumer
'- snapshot refresher (daily; reads Iceberg metadata via Trino)
|
+--------+-------------------+
v v
Metadata graph Telemetry and analytics
(datta-siql-db database) (datta-siql-events-* indices)The user keeps writing plain SQL — nothing changes in how they query. The layer receives query events and answers advisor requests through its own endpoints; see the API reference.
Coupling to the engine's optimizer (wrapping the connector metadata to consult the advisor during planning) was planned for phase 2 — in the MVP the layer only observes.
Closed decisions
| Decision | Value |
|---|---|
| Official feature name | datta-intelligent-query-layer |
| Internal short name | SIQL |
| Service port | 8240 |
| Dedicated graph database | datta-siql-db — isolated from the others, less contention |
| Telemetry index | datta-siql-events-<yyyy-MM-dd>, daily |
| Event topic | datta-siql.trino-events |
| Event publishing mode | Dual: the plugin tries the topic and, on failure, sends over HTTP straight to the layer |
| Default advisor mode | STATS_ONLY — returns only estimated rows and bytes, does not attempt read pruning. The RANKING and FULL modes arrive in later phases |
| Permission | SIQL:ADMIN |
| Cache | A dedicated logical space in the platform cache, not shared with other modules |
| Configuration interface | Reuses the visual pattern of the platform's other configuration pages |
Mapped risks and how they were handled
| Risk | Handling |
|---|---|
| The analytical engine had no query event listener | A dedicated plugin, compiled against the Trino 480 extension API, ships with the installation and starts publishing every execution event |
| The Iceberg catalog uses a Hive Metastore, with no REST catalog | The snapshot refresher reads metadata through the analytical engine itself, querying the $snapshots, $files, $manifests and $partitions system tables. It does not depend on an external REST catalog |
| The advisor must fit within 20 ms | Two-tier cache (local memory + platform cache). The hot path is only a lookup by request hash; graph and index are touched only on a cache miss, in parallel and with 15 ms per source. If the budget overflows, the response falls back to a deterministic result |
| Additional load on the graph cluster | The layer writes only to its dedicated database, always in batches and in the background — never on the user's query path |
| Telemetry volume in the index | Daily index with a lifecycle of 7 days hot → 30 days warm → removal at 90 days. One shard and one replica in the MVP |
Personal data in WHERE predicates | A masking stage runs before any persistence: literal values and the logged SQL go through masking |
| Dependency on the engine version | The plugin is compiled against the stable version in production (480); an engine upgrade requires recompiling the plugin |
Open questions
- Iceberg REST catalog: the current installation uses the Hive Metastore. If it later migrates to Nessie or another REST catalog, the snapshot refresher will need to parameterize the client. It does not block the MVP.
- Central connection registry: the platform requires every external source to be registered in the central registry. The advisor does not open a connection to any JDBC source — it only reads metadata through the analytical engine, the graph and the index, all internal — so there is no exception to the rule. If a later phase starts reading directly from an Iceberg REST catalog, that source must be registered normally.
Learn more
- Layer specification — the value and scope in one page.
- NL-to-SQL — questions in Portuguese become governed SQL.
- API reference — the endpoints the platform exposes.