Platform AI models: where to choose them and how to check
Each type of AI model on the platform — the chat model, the DATTA Captain model, the entity extraction (NER) model, the embedding model and the rerank model — has one choice, which applies to the whole platform: every context, every service and every replica. It is made under (and, for the Captain, under ). No service has its own model in the code, no context keeps a copy, and what a request sends in the model field does not change the choice.
Required permission: CONFIG_EDIT to change; CONFIG_VIEW to view.
Where to change each model
| Model | Where to change | Who uses it | When the change applies |
|---|---|---|---|
| Chat (LLM) and Reasoning | → Modelo de Chat (LLM) and Modelo de Raciocínio (Thinking) | chat, conversation title, triage, rules, SIQL, MCP tools, knowledge generation, DATTA BI copilots, case-type classification | within 60 s |
| DATTA Captain | → Model card | the Captain (planning and vision) | within 60 s |
| Entity extraction (NER) | → Modelo de Extração de Entidades (NER) | NER for upload, Datta Extract and legislation; URL analysis and catalog classification on upload | within 60 s, from now on — the already extracted corpus does not change |
| Embedding | → Modelo de Embedding | every vectorized text and every semantic search | when the switch with backfill finishes (hours on a large corpus) |
| Rerank | → Modelo de Rerank (Cross-Encoder) | second stage of chat and triage search | about 2 minutes (the engine restarts; meanwhile search uses RRF fusion) |
Captain. The card's first option, "Usar o modelo do chat" (use the chat model), makes the Captain follow the chat model. Below the list, the card shows the model in use and where it comes from (the Captain's own choice or the chat model). Models that do not suit the Captain appear disabled, with the reason.
Embedding. Choosing another model opens a switch: the new model's field is created in the indices, the corpus is vectorized again and only then does the new model apply. Until then search stays entirely on the previous model. The step by step and the precautions are in Embeddings — one field per model.
What the save refuses
The tab does not save a choice the installation cannot serve. The refusal comes in Portuguese, and nothing is saved, when:
- the provider is not allowed in this installation (for example, Gemini on an installation that does not allow text to leave the cluster);
- the model's context window is known and smaller than the platform minimum (131,072 tokens);
- the installation's vLLM server does not serve the chosen model;
- for rerank, the engine does not confirm the change (remote reranker without a configured address, or reranker turned off).
With no model chosen
There is no "fallback" model. If the Chat Model is empty, the chat and the other features answer that the model is not configured under System → AI, instead of answering with a model nobody chose. An empty NER does the same for extractions.
On a new installation, the platform fills the tab once with the models the installer put on the server (in the offline bundle, the chat model served on CPU and the bundle's embedding and rerank engines). After that the choice belongs to the administrator: running the installer again or updating the platform does not change it.
macOS installation
On macOS, embedding and rerank are served by the computer itself, with a fixed model file. The tab shows "Servido pelo host: <model>" and is read-only for those two: changing them requires reinstalling with another model (and reindexing, for embedding).
Speed and request budget (lean mode)
A model served on CPU reads the request slowly — tens of tokens per second — and the chat's full request (context prompt, ~40 tools, search snippets and history) does not fit in the deadline of one call. So the chat computes, on every question, how much fits on the engine in use and, when the request exceeds it, switches to lean mode: only the tools the question needs, the search done by the model itself when it needs it (instead of snippets already pasted into the question) and, if it still does not fit, cuts announced at the end of the answer (least relevant snippets, tools beyond the essential ones, the oldest messages of the conversation). The question and the context prompt are never cut. None of this is configured: the formula decides, from the engine's measured speed. Gemini and Runpod publish no speed and never shrink the request.
Under , right below the Chat Model, the Velocidade e orçamento do pedido (speed and request budget) section shows (read-only, with CONFIG_VIEW):
- Chat request — the mode badge (Modo enxuto / lean mode, Pedido completo / full request, Decidido por pedido / decided per request, Sem modelo / no model), the reason with the numbers of the last question and the budget formula spelled out: how many input tokens fit, with the read and write speed, the deadline of one call, the expected rounds, the output per round, the prefix hit rate and the characters per token.
- Speed per engine — for each provider, the deadline of one call, the read (prefill) and write (decode) speed, the prefix hit rate and where the speed came from: measured on the engine, last stored measure (the engine restarted and has not finished a request yet) or chart seed (no measure yet). And the characters per token of each model, calibrated by real usage, or the provider default while there is not enough sample.
- Last cut per part — over the last 7 days, how many snippets, tools, history messages or tool results were cut, when and on which model.
- Prompt of each context — the size of each context's prompt on the engine in use, how long it takes to be read cold and whether it alone fits the budget. A prompt that does not fit does not block the question (it goes on with a warning that it may exceed the deadline), but it is what weighs most on a CPU engine: trimming it is up to whoever maintains the context. The table shows five contexts at a time; scroll it vertically to see the rest, with the header kept in place.
How to read it. Chart seed right after switching models or restarting the engine is expected: after the first finished question the origin becomes measured on the engine. Many tool or history cuts in one context point to a context prompt too large for the engine — check the prompts block. With the model on Gemini or Runpod the section shows Pedido completo and no speed, and there are no cuts. The numbers of each question are in the chat log (Modo enxuto and Pedido acima do orcamento do motor lines) for the operations team.
How to check what is in effect
- shows the saved model for each type. A rerank that came from a previous version shows the note "migrado de …" (migrated from).
- The Captain card shows the model in use and its origin.
- In the AI request History, each call records the provider and model that actually answered.
- The Velocidade e orçamento do pedido section shows whether the chat is in lean mode and why.
The full post-update check, feature by feature, is done by the operations team.
References
- Chat Reasoning Model (Thinking)
- Embeddings — one field per model
- DATTA Captain (user guide)
- AI engine scaling — how many replicas each engine keeps
- Long LLM flows (architecture) — the budget formula, the stable prefix and the cuts