MinT: Managed Infrastructure for Training and Serving Millions of LLMs

Source-first digest for monthly 2026_05 rank 25, rank_id p005.

• Routing status: success, route full_markdown
• PDF extraction: not used

Motivation / Background

MinT treats post-training as an infrastructure workload rather than as a sequence of isolated checkpoint conversions. The target setting is many trained LLM policies over a small number of expensive resident base-model deployments. Full fine-tuning makes every policy a full checkpoint; merge-based LoRA saves training memory but still tends to materialize a merged full checkpoint for inference. MinT instead makes the LoRA adapter revision the behavior-carrying object that crosses rollout, update, export, evaluation, serving, and rollback. This source claim is the starting point for the managed-policy evidence used in the support table.

The paper's contribution is best read as three separate scale claims. Scale Up keeps the adapter-revision path usable for dense and MoE frontier models, including distributed Megatron training and sparse-route handling. Scale Down removes full-checkpoint materialization from training-serving handoff and reuses idle time across concurrent GRPO policies. Scale Out separates durable policy addressability from CPU-cache, GPU-batch, and cold-load residency. Those distinctions matter because the paper does not claim one engine keeps 1,000,000 adapters active; it claims MinT can build, name, audit, and selectively serve revisions from a million-scale catalog through bounded working sets.

Claims And Evidence

Support scores are support-from-paper scores, not independent reproduction scores.

Core Technical Idea

MinT separates a policy record from an adapter revision. The adapter revision is a fixed exported LoRA payload in serving layout. The policy record is service-owned state: compatible base version, LoRA rank, target modules, latest training checkpoint, rollout records, optimizer state, exported revisions, and cache/residency state. That distinction lets the system answer four questions independently: which base can run the behavior, which checkpoint resumes training, which revision should be evaluated or served, and where the adapter bytes currently live.

The runtime has four linked mechanisms. First, the service plane creates pollable operation ids and commits outputs only after metadata is written, so retries do not select uncommitted attempt files. Second, training workers keep the base model resident and time-slice policy states. Third, export converts trainer-side LoRA tensors into PEFT/vLLM-compatible serving layout without copying the base checkpoint. Fourth, serving actors map a requested policy revision through GPU-batch, CPU-cache, and cold shared-storage states before generation.

The extracted equations.json contains no display equations for this paper, so Table 1 records the key inline symbols and source-side formulas that define the system semantics rather than claiming a formal optimization objective.

Method Details

The training side uses time-sliced multi-LoRA training. A resident trainer restores one policy's LoRA tensors and optimizer state, runs the update, checkpoints the result, then yields to another policy without reloading the base model. Workers advertise supported rank and target-module limits; smaller adapters can be padded/masked inside configured maximum slots. For Megatron MoE runs, dense LoRA tensors follow tensor-parallel shards, expert LoRA tensors are keyed by expert id, and export gathers or deduplicates tensors into the serving layout.

The serving side uses shared-base multi-LoRA sampling. The service maps a user-facing policy name to an exported adapter revision and sends that revision to a vLLM sampler that already holds the base model. If the adapter is active in the GPU batch, the request uses the hot path. If it is CPU-cached, the actor promotes it. If it is absent, MinT schedules a cold load from shared storage before generation. Cold loading is not hidden inside normal decoding latency; it is an explicit service stage with deduplication and backpressure.

MinT also records sparse-model provenance. For Qwen3-style MoE, rollout records can store selected expert ids so training can score tokens along the route that generated them; missing or unmappable routes are masked from the replayed policy-gradient term. For GLM-5-style dynamic sparse attention, the system stores backend/model path and correction policy, then uses IcePop-style rollout correction to zero unsafe importance weights outside a trusted probability-ratio band. This is a mitigation, not full DSA route replay.

Experiments And Results

Scale Down: Adapter Handoff And Utilization

The scale-down experiments isolate systems overhead. Adapter handoff compares a MinT adapter load against merge-and-load paths. Concurrent training compares three GRPO policies run sequentially against the same policies overlapped under one resident base allocation. The paper reports 18.3x and 2.85x handoff-step reductions in the abstract, and the table/figure evidence below shows the underlying file sizes, load/materialization times, cold first-sample times, wall-clock schedule times, and unchanged peak memory.

The handoff bars in Figure 1 show why moving adapter revisions is operationally different from moving full checkpoints.

The schedule timeline in Figure 2 shows the second part of the scale-down claim: concurrent GRPO policies fill idle resident-base windows rather than requiring another base copy.

Scale Up: Dense, MoE, UI, And AutoResearch Runs

The scale-up evidence is not one benchmark. It is a compatibility argument across training paradigms and model placements. Dense runs cover SFT, DPO, and GRPO under one adapter lifecycle. MoE runs add expert-route replay and distributed placement. GLM-5.1 A2UI adds a product-facing GRPO workload. AutoResearch adds proxy-to-full benchmark promotion logic around maintained adapter recipes.

Figure 3 shows dense-model traces for SFT, DPO, and GRPO, while Table 3 captures the endpoint evidence.

Figure 4 gives the sparse-model scale-up evidence, including Qwen3-30B/235B AIME24 and a Kimi K2 1.04T countdown-task curve.

Figure 5 covers the GLM-5.1 Generative UI GRPO case.

Figure 6 shows the AutoResearch LawBench trace.

Scale Out: Policy-Population Serving

The serving experiments split addressability from execution. A durable catalog can contain 1,000,000 adapter revisions, but one serving actor keeps only selected revisions in CPU cache and a smaller set in the GPU batch. The paper's strongest scale-out evidence is therefore a bundle: million-catalog build/audit, cache and same-batch bounds, warm selected-revision SLO, cold-load accounting, packed representation, and readiness control.

The mixed warm/cold control rows in Table 5 show why MinT distinguishes registered, prewarming, and ready states.

The packed-loader rows in Table 6 show that MoE LoRA cold load is object-bound, not only byte-bound.

The evidence boundary is explicit. The 1M result is a built-and-audited catalog plus selected-revision serving measurements. It is not a claim that every revision is resident, active, or warm. Warm selected-revision serving is measured for a particular Qwen3-30B rank-1 LoRA workload with prompt length 1024 and max output length 64. End-task quality evidence comes from named cookbook tasks and benchmarks, while the paper's main novelty is infrastructure: policy records, adapter revisions, export, cache state, cold-load scheduling, and readiness.

Practical Takeaways

MinT is most relevant when an organization expects many LoRA policies over a smaller set of expensive base models. The practical win is not only smaller files; it is making adapter revision the stable service unit that can be resumed, exported, evaluated, rolled back, cached, prewarmed, and served without turning every variant into another full-model deployment.

For training systems, the useful pattern is to keep the base resident and make policy state the object that moves. That requires versioned policy records, committed operation visibility, serving-compatible export, and exact attribution of rollout/evaluation scores to exported revisions. For serving systems, the useful pattern is to treat cold adapter loading as service work. A request-name catalog, CPU cache, GPU batch, and readiness state are different resources with different limits.

The main caution is that scale-out claims need careful wording. MinT's million-scale evidence is durable addressability over a real packed catalog, while runtime capacity depends on selected-revision locality, cold-load rate, batch adapter diversity, CPU cache, and prewarm policy. The paper is valuable because it makes those boundaries visible instead of compressing them into a single "serves millions of adapters" slogan.

Reference Coverage

This digest explicitly links all anchors used above: motivation, lifecycle mechanics, key equations and symbols, scale-down evidence, handoff figure, utilization figure, scale-down table, scale-up evidence, dense curves, MoE curves, GLM curves, AutoResearch trace, scale-up table, serving evidence, serving bounds, rollout control, packed loader, limitations, and reference coverage.