AI engine scaling: how many replicas each engine keeps
The AI engines that run inside the platform — Embedding, Rerank, Chat on CPU and Chat on GPU — can have more than one replica. In , section Escala dos motores (engine scaling), the administrator sets, for each one, whether it is on and between how many replicas it may vary. Within that range the platform adds and removes replicas on its own, driven by the engine's request queue, and only when the new replica fits on a server. On an installation with more CPU and memory, the platform uses what it has, without anyone editing installation settings.
Required permission: CONFIG_EDIT to save the scaling; MODELS_MANAGE to see the engines' current state (replicas, queue, last change). Without MODELS_MANAGE, the section shows only the saved scaling and says which permission is missing.
What each card shows
Each engine has a card. On the left, what is happening now (refreshed on its own every 15 seconds); on the right, the saved scaling.
| On the card | What it means |
|---|---|
| Badge | estável (stable, nothing to do), escalando (just changed), réplica subindo (a new replica is starting), no máximo (at the maximum), sem espaço (no room), aguardando nó novo (the replica waits for the server the cloud creates), sem nó novo (the cloud did not create the server in time), fica como está (stays as is), erro ao escalar (scaling error), desligado (off) or não instalado (not installed) |
| Replicas and ready | How many replicas the engine has and how many already serve. "Alvo" (target) shows when the platform is moving to another number |
| Queue per replica | For each replica, how many requests are waiting and how many are being served. A replica still starting shows as "subindo (ainda não pronta)" |
| Running window | "Pedidos esperando há 40 s (sobe após 60 s…)" or "Ociosa há 6 min (desce após 15 min…)" — how long until the next change |
| Last scaling | From how many to how many replicas, why (queue, idle, minimum, maximum, on, off) and when |
| Reason | When the engine could not scale, the reason in Portuguese — see the table below |
| Replica size | Chat on CPU only: the applied size, the one computed for the server and the adjustment (own section below) |
The scaling fields
| Field | What it does | Default |
|---|---|---|
| Ligado (on) | Off = no replica, and it beats everything else | per installation |
| Mínimo de réplicas (minimum) | How many replicas stay up while the engine is on. Never below 1: there is no "scale to zero", because the first answer would wait for the engine to start | 1 |
| Máximo de réplicas (maximum) | The ceiling of queue-based scaling. Equal to the minimum = a fixed number | 1 |
| Janela de subida (s) (scale-up window) | How long in a row requests must be waiting to add one replica | 60 s |
| Janela de descida (min) (scale-down window) | How long in a row the engine must have no request (neither waiting nor being served) to remove one replica | 15 min |
Salvar escala saves the engine of that card. The choice applies to the whole platform, takes effect within a minute and survives platform updates: updating does not turn on, turn off or change the replica count of any engine.
How the platform decides
- By the queue, never by CPU. An AI engine fills the whole CPU with a single request; measuring CPU would add replicas with nobody waiting.
- One change at a time. After scaling up or down the count restarts; while a new replica starts, nothing is re-evaluated. Chat on CPU takes about 10 minutes to start: the default scale-down window (15 min) is longer than that on purpose.
- Only where it fits. Before scaling up, the platform checks whether some server has free CPU, memory (and GPU, for Chat on GPU) for one more replica. If not, it does not scale — no replica is left "waiting for room" — and the reason shows on the card.
- Each replica has its own copy of the model, and Chat on CPU replicas on the same server use separate cores.
What saving refuses
The refusal shows in red on the card, with the reason, and nothing is saved:
- minimum below 1 with the engine on (to go to zero, turn it off);
- maximum lower than the minimum;
- scale-up or scale-down window lower than 1;
- an empty field or a non-integer number (the screen asks for the number before sending).
When the engine does not scale
| Reason on the card | What to do |
|---|---|
| "fila acima de zero há N s, mas o motor já está no máximo" (queue above zero, already at the maximum) | Raise the maximum, if the server has room |
| "sem memória/CPU/GPU livre para outra réplica … faltam X no nó Y" (no free memory/CPU/GPU) | No server has room for one more replica. Free resources or accept the current ceiling |
| "sem núcleos próprios para outra réplica do vllm-cpu" (no own cores) | The server already has the Chat on CPU replicas that fit with separate cores |
| "nenhum nó elegível …" (no eligible node) | No server meets the engine's constraints (for example, GPU) |
| "o espaço livre nos nós não pôde ser medido agora" (free room could not be measured) | The platform could not measure the servers; no replica beyond the first starts without that measurement. Call the operations team |
| "a fila das réplicas não pôde ser lida agora" (queue could not be read) | Without reading the queue, the platform neither scales up nor down by it. Usually clears on its own |
| "aguardando a réplica nova ficar pronta" (waiting for the new replica) | Normal while a replica starts (minutes for Chat on CPU) |
| "aguardando nó novo do autoscaler: … sem nó há N min" (waiting for a new node) | Only on cloud installations with automatic server growth: the replica did not fit today's servers and waits for the new server the cloud creates (a few minutes). Nothing to do |
| "nenhum nó novo em N min: … desceu" (no new node in N min) | The cloud did not create the server in time (server group limit or provider quota): the replica went down, and for 1 hour the engine only scales up what fits today's servers. Call the operations team to review the limit or the quota |
Warning at the top of the section. If the service that applies the scaling is stopped or has not run a cycle for more than 3 minutes, the section says so. Meanwhile the replicas stay as they are — they never drop to zero on failure — and the saved scaling is applied when it comes back.
Replica size (Chat on CPU)
Chat on CPU has the size of each replica — threads, cores, memory for the context (KV), concurrent requests and reserved memory — computed for the server it runs on.
- Automatic (default): the card shows the applied size and the one computed for the largest available server. With the engine serving, a different computed size is not applied on its own (it would restart chat for minutes): the card marks difere do calculado (differs from computed), and the new size goes in on the next power-on or through Ajustar tamanho.
- Ajustar tamanho (adjust size): saves custom values; the replica restarts with them. Values outside the rule (for example, context memory larger than the memory limit) are refused with the reason.
- Voltar ao automático (back to automatic): clears the adjustment; the platform computes from the server again.
Embedding and Rerank cores
Embedding, Rerank and Chat on CPU running on the same server split its cores by a single rule, so that one does not contend for another's core. When the server cannot hold the three with cores of their own, the rule splits them, and the card warns who shares cores with whom.
- Applied and computed: the Embedding and Rerank cards show the cores in use, the ones computed for the server and the whole partition. With the engine serving, a different computed value is not applied on its own: the card marks difere do calculado (differs from computed), and it goes in on the engine's next power-on.
- Ajustar núcleos (adjust cores): saves threads, first core (-1 = vLLM chooses) and replicas per server; the engine restarts with them (1 to 3 minutes).
- Voltar à semente (back to the seed): goes back to the cores the engine used before the partition — the engine restarts with them.
- Voltar ao automático (back to automatic): clears the adjustment; with the engine serving, nothing restarts now.
Measure document upload before and after changing the Embedding cores.
Good practices
- Keep minimum = maximum = 1 where the server only holds one replica — that is the default.
- Raise the maximum only after checking, on the card itself, that the engine does not show "sem espaço" (no room).
- Do not shorten the scale-down window below the engine's start time: the engine would go down and come back up in a row.
- The full check after a platform update (building a queue, watching the scale-up and the scale-down) is done by the operations team.
References
- Platform AI models: where to choose them and how to check