Triton Dynamic Batching Tuning: SLO and Rollback Runbook

Executive Summary

Triton dynamic batching tuning is an experiment in meeting a latency service-level objective (SLO) at higher useful throughput. Enable batching only for an eligible model, compare it with a verified baseline, and accept a candidate only when the client-observed latency distribution and error rate remain within the service's own limits. NVIDIA documents dynamic batching for stateless models, while Google’s SLO guidance recommends measuring both typical experience and the latency tail. ^[1] (Source: sre.google)

The recommended sequence is to inspect the loaded configuration, test batching without intentional delay, sweep max_queue_delay_microseconds, and investigate preferred_batch_size only when model-specific evidence justifies it. NVIDIA advises against preferred sizes for most models; a documented exception is TensorRT optimization profiles whose performance differs materially by batch size. A batch-formation delay is not an end-to-end latency guarantee: Perf Analyzer's server queue component also includes waiting for an available model instance. Use separate client and server observations to identify where time goes. ^{[1] [2] [3]} (Source: sre.google)

Use fixed concurrency to explore capacity and request-rate tests to evaluate an independently arriving workload. A closed workload slows its own arrivals when responses slow; an open workload separates arrivals from completion. Consequently, a throughput gain under concurrency alone cannot establish that production traffic will meet its SLO. Record input shapes, request batch size, arrival distribution, warmup, observation windows, errors, and p50, p95, and p99 latency. The experiment matrix below is a recording template, not a published GPU benchmark. ^{[4] [5]}

As of October 16, 2026, NVIDIA's live release page identifies 26.09 as available ^[6]; the corresponding official SDK distribution includes Performance Analyzer and Model Analyzer. ^[7] A planned 26.08 deployment should therefore be treated as a local upgrade milestone, with its own verification. This report recommends keeping that release change separate from batching changes. Stage the candidate alongside the baseline, judge equivalent traffic cohorts, and restore the complete known-good artifact set if acceptance fails. Kubernetes rollback restores the Pod template only, so model configuration stored elsewhere needs its own restoration mechanism. ^[8] No universal batch size, delay, throughput gain, or canary percentage is established here.

Introduction and Background

Dynamic batching combines compatible inference requests into work executed together. Triton exposes the scheduler through the model configuration, making batching a model-level serving choice. Eligibility matters: the documented dynamic batcher is intended for stateless models, rather than an interchangeable scheduler for every workload. This report addresses inference platform engineers operating owned GPUs who need to decide whether extra batching improves their particular service. ^[1]

The engineering decision begins with a measurement boundary. A server's execution measurement answers a different question from a user's elapsed request time. OpenTelemetry distinguishes remote procedure call (RPC) client and server duration metrics, providing a useful framework for correlating those observations without treating them as identical. The proposed acceptance boundary is the production client's logical operation, with backend timings retained for diagnosis. ^[3] (Source: sre.google)

Benchmark conditions also determine what a result means. MLCommons defines inference scenarios through both request patterns and the metric being measured. Its distinction supports an important discipline here: a capacity result belongs to the workload that produced it. A configuration that looks attractive under steady load still needs evaluation against the service's input mix and arrival pattern before it becomes a deployment candidate. ^[5]

From the perspective of an adjacent infrastructure engineering advisor such as GPU Smith, the useful deliverable is a reproducible decision record: which workload was tested, which constraint applied, which candidate passed, and how the previous configuration can be restored. The publisher is not a serving-runtime alternative in the comparison.

This runbook presents a tuning procedure, not measured benchmark results. Warmup, repeated runs, and an explicit inventory of artifacts therefore form part of the procedure. PyTorch's benchmarking guidance illustrates why: initialization can be lazy, and replicated measurements are needed to characterize noise. ^{[9] [10]}

Keep successful logical operations per second as the local useful-throughput measure. Report transport attempts separately when retries are enabled, since retry behavior can add streams and backoff time. That avoids treating repeated attempts as additional completed user work. ^[11] (Source: sre.google)

Key Changes: Make Batching a Controlled Variable

Verify the actual baseline

The baseline must describe the configuration Triton loaded. Merely deleting a handwritten batching block is insufficient: the model-configuration documentation states that backend auto-completion can enable dynamic batching. Inspect the generated configuration through the model configuration endpoint, and retain that response with the experiment record. ^[12]

Hypothetical Example: for a model named image_encoder, this command inspects its local endpoint:

curl --silent http://localhost:8000/v2/models/image_encoder/config

The documented endpoint returns the generated model configuration as JSON. Check its scheduling configuration, legal batch dimension, maximum batch size, input/output specification, and instance groups before labeling a run “no batching.” Preserve a fully specified known-good configuration for the baseline; disabling backend auto-completion is an available server control, but the loaded response remains the practical check. ^[12]

Keep the model artifact, runtime, protocol, request batch size, memory placement, and hardware allocation fixed during the first comparison. For example, ONNX Runtime documents that CPU-resident inputs can be copied to the device during execution, whereas I/O Binding can arrange transfers and allocation beforehand. Changing those conditions alongside batching would make attribution difficult. ^[13]

Separate the controls

Treat the experiment as a sequence of decisions:

• Baseline: prove that the loaded scheduler matches the intended control.
• Immediate batching: enable the batcher without adding intentional formation delay.
• Delay sweep: vary only the formation-delay setting within the local queue allocation.
• Preferred sizes: test these only after evidence identifies a useful size-specific execution profile.
• Instance sweep: vary execution instances after retaining a promising batching candidate.
• Revalidation: repeat the same traffic tests after any combined candidate is assembled.

These are proposed experiment stages, not NVIDIA performance thresholds. The vendor's documented default behavior forms available batches without intentional delay; when a configured delay expires, the scheduler sends the batch it has formed. ^[1]

A minimal fragment for the immediate-batching candidate is:

dynamic_batching { }

Retain the model's existing valid configuration around that fragment. Do not infer a legal maximum batch size from this example, and do not copy a preferred size from a different model.

Interpret delay as one contributor

Queue delay has two meanings that must remain distinct: intentional waiting to improve batch formation, and observed time waiting for service. Perf Analyzer's queue measurement includes waiting for a model instance. Accordingly, the acceptance test must observe total queue behavior rather than assume the configured formation delay bounds every wait. ^[2]

A useful operational check is to keep upstream failures visible too. For example, Envoy can reject requests under its overload controls or stop accepting new connections. A latency comparison that retains only completed requests could overlook those outcomes. Record admission, rejection, timeout, and completion counts at the service boundary. ^{[14] [3]}

Triton's model-control mode also determines how a configuration change takes effect. Its documentation distinguishes repository changes ignored in NONE mode from explicit model-control operations, and currently calls for unloading an existing model before loading its update. Document the real control path and its availability implications before relying on an in-place reload. ^[15]

Traffic and Measurement Design

Choose concurrency or request rate deliberately

Concurrency mode attempts to maintain a fixed number of outstanding requests. Use it to map capacity and compare configurations under controlled outstanding work. It answers how the server behaves when clients keep that amount of work in flight; it does not reproduce an independently specified arrival stream. ^[16]

For the SLO test, choose the arrival model that matches the service. Perf Analyzer supports request-rate and custom-interval modes; the latter consumes intervals expressed in microseconds. Its CLI also documents constant and Poisson request distributions. Retain the selected mode and distribution as experiment inputs rather than describing every run simply as “load testing.” ^{[16] [17]}

The distinction is consequential. Grafana's load-model documentation explains that a closed model waits for iteration completion before starting the next iteration, and can therefore lower its arrival rate as the system slows. An open-model test is needed when arrivals should remain independent of response completion. ^{[4] [5]}

Use representative inputs and timing

Production-like data must cover the model's relevant shapes and content classes. Perf Analyzer's default random data is generated once per input and reused, whereas JSON data can be supplied and iterated in round-robin order. Record the dataset identity and shape distribution explicitly. The CLI's -b controls the batch within a client request, distinct from the scheduler's combined batch; supply variable input shapes through --shape or JSON shape fields. ^[18]

Backend behavior provides further reasons to keep that distribution stable. TensorFlow documents shape-dependent function matching and expensive tracing. A different shape mix can therefore change the execution path being measured. Keep shape cohorts visible in the results rather than hiding them inside a blended average. ^{[19] [5]}

For backend microbenchmarks, GPU timing needs additional care. PyTorch documents asynchronous GPU operations and warns that measurements without synchronization are inaccurate. Keep such measurements separate from the ordinary service path: the purpose is to diagnose execution, not to introduce synchronization into every production request. ^[10]

Archive reproducible commands

Hypothetical Example: the following commands contain local experiment settings, not recommended production rates or sample sizes. They assume an existing compatible model named image_encoder, a model version named 1, representative inputs.json, and a local gRPC endpoint. All CLI options were checked against official Perf Analyzer documentation; no GPU benchmark was executed for this report. The CLI's --percentile option influences the latency used in stability measurement. ^[20]

perf_analyzer --help

perf_analyzer -m image_encoder -x 1 -i grpc -u localhost:8001 \
  --async -b 1 --input-data inputs.json \
  --concurrency-range 1:16:1 \
  --measurement-mode=count_windows --measurement-request-count=10000 \
  --warmup-request-count=1000 --percentile=99 \
  --verbose-csv -f baseline-concurrency.csv

perf_analyzer -m image_encoder -x 1 -i grpc -u localhost:8001 \
  --async -b 1 --input-data inputs.json \
  --request-rate-range 100:500:100 --request-distribution poisson \
  --measurement-mode=count_windows --measurement-request-count=10000 \
  --warmup-request-count=1000 --percentile=99 \
  --verbose-csv -f candidate-request-rate.csv

Here -x selects the model version; it is not a client binary version flag. Archive --help, the SDK image digest, and the package manifest to identify the actual toolchain. Customize the illustrated rates, counts, and concurrency range before running. ^[17]

Implementation Considerations and Process Changes

Freeze the experiment manifest

The manifest should identify enough state to reproduce the result without reconstructing a running machine. Record these items alongside every trial:

• Model identity: repository revision, model version, artifact checksum, backend, precision, and optimization profiles.
• Runtime identity: server and SDK image digests, backend packages, host driver, and deployment configuration.
• Input contract: dataset checksum, shape cohorts, request batch size, protocol, and preprocessing location.
• Traffic: arrival model, intended rate or concurrency, ramp profile, burst trace, and routing policy.
• Measurement: warmup rule, window rule, request count, repetitions, percentile method, and histogram boundaries.
• Resources: GPU allocation, memory conditions, host CPU allocation, client placement, and network path.

This manifest is the report's proposed audit record. Backend-specific controls belong in it because their effects extend beyond GPU batch execution. ONNX Runtime, for example, documents a thread-pool spinning tradeoff that consumes additional CPU cycles and power while potentially speeding inference. Keep that policy fixed during the batching sweep. ^[21]

Explicitly separate warmup from measurement. PyTorch's benchmark utility performs warmups because some components initialize lazily. Replicated measurements then help quantify noise. Adopt a local rule for reaching the intended warm state, retain the rule, and apply it to both baseline and candidate. A warm candidate should not be compared with a cold control. ^{[9] [19]}

Keep deadlines, retries, and errors in the record

Set and record the client deadline according to the service contract. gRPC does not set one by default, and its guidance recommends validating realistic values through load testing. A deadline should reflect the complete request path rather than just an execution-time observation. ^[22]

Retries require separate accounting. gRPC documents creation of another retry stream after backoff, so a logical operation may differ from a transport attempt. Compare user-visible completion time and successful logical operations, while retaining attempt counts to explain additional offered work. Record the retry policy for both control and candidate. ^{[11] [3]}

Use bounded error categories so failed operations remain analyzable alongside successful latency observations. OpenTelemetry's RPC conventions recommend low-cardinality error types. The scorecard should distinguish successful completions, deadline expiry, admission rejection, and other failures rather than presenting successful latency alone. ^[3] (Source: sre.google)

Observe service and backend boundaries

Maintain separate client and server duration series. OpenTelemetry explicitly splits these measurements for correlation; a gap is a diagnostic signal to investigate, not proof of a particular bottleneck. Correlate request identifiers or traces where available before assigning that difference to network, preprocessing, or queueing. ^[3]

Also verify that the load generator reaches the intended steady state. Locust distinguishes ramp-up rate from peak load, so a faster ramp does not by itself establish a larger sustained offered workload. Retain the achieved traffic trace through warmup, steady measurement, and bursts. ^[23]

Validate the generator as well as the server. Locust's guidance calls out generator CPU, network utilization, and operating-system metrics when increasing request rates. A saturated generator can prevent a test from delivering the traffic it was supposed to send. The run record should distinguish offered, started, completed, and successful operations. ^[23]

Bound the Search and Diagnose the Result

Model Analyzer can reduce manual trial management. Its configuration search supports custom model-configuration sweeps, including dynamic-batcher queue delay, and its configuration interface exposes search bounds and constraints. Use model-analyzer profile --help to inspect the command-line surface, then set explicit search bounds and constraints. Use the tool to enumerate legal candidates inside the service's resource envelope, not to choose the business objective implicitly. ^{[24] [25]}

A bounded search plan should make the following choices explicit:

• Batch range: include only sizes supported by the model and optimization profiles.
• Delay range: derive candidate waits from the queue allocation, beginning with no added wait.
• Instance range: constrain experiments to available GPU memory and deployment resources.
• Traffic range: cover the service's relevant low, normal, peak, and burst conditions.
• Objective: maximize useful completed work only after the latency and error constraints pass.
• Checkpoint identity: isolate measurements from experiments with different manifests.
• Readiness: require the generated configuration to load before measuring it.

Do not equate a syntactically generated configuration with a deployable configuration. Model Analyzer's documentation states that it cannot guarantee every generated configuration will load. Require an actual load and readiness check before counting a candidate as measured, and retain failed-load records rather than assigning them invented performance values. ^[24]

Use a distinct checkpoint directory for each materially different experiment. Model Analyzer loads its latest checkpoint at startup, making reused measurements helpful only when the underlying experiment identity remains valid. Changing the model, input distribution, GPU allocation, or toolchain should trigger a fresh experiment record. ^[26]

For toolchain selection, prefer a deliberately matched release distribution and then perform a local compatibility check. The official Model Analyzer installation instructions describe containers based on corresponding Triton server containers. That provides a reproducible starting point, but no retrieved source established a blanket guarantee for arbitrary cross-version combinations. ^[27]

Treat the recommended order as a way to preserve attribution. After delay tuning, investigate preferred sizes only when the model's execution profiles justify that experiment. Then test instance changes as a separate factor. If a combined candidate passes, retain the intermediate results so the team can explain which change produced the improvement.

For every load point, retain a row for each delay candidate, including failed trials. Filter first on client latency and errors. Among passing candidates, select the smallest delay that supplies the required capacity and reserve. If immediate batching already meets the requirement, require a measured benefit before adding wait. When testing preferred sizes, compare them at the same delay and instance setting, retaining only repeatable gains for the actual input cohorts. GPU utilization alone is not an acceptance condition.

Canary Rollout and Configuration Rollback

Apply a pass/fail scorecard

Table 1 defines the deployment decision. The service owner supplies its limits before testing. Google's event-based SLI formulation expresses good events relative to all eligible operations. (Source: sre.google)

Every scorecard condition must pass: throughput does not cancel a latency failure. Evaluate shape, tenant, or request-class cohorts where their contracts differ. Keep the same percentile method for both deployments, and use an aggregatable representation when reporting a fleet-wide distribution. ^[28]

Stage, compare, and restore

A practical rollout procedure is:

• Snapshot: preserve the known-good image digest, model artifacts, configuration, routing state, and experiment manifest.
• Stage: load the candidate separately and verify configuration identity and representative outputs.
• Warm: apply the same declared warmup policy before admitting measurement traffic.
• Route: send the predeclared canary share through a controlled traffic policy.
• Compare: evaluate matched cohorts against every scorecard condition.
• Stop: stop increasing exposure when a hard condition fails.
• Restore: return routing to the ready baseline and restore changed artifacts through the deployment's documented mechanism.
• Confirm: verify loaded configuration, output checks, traffic recovery, and SLO recovery before closing the change.

Kubernetes documents multiple Deployments as a canary approach. Readiness then determines whether a Pod receives traffic through Services, with readiness probes continuing throughout its lifecycle. Check the candidate and the restored baseline rather than treating process startup as sufficient evidence of serving readiness. ^{[8] [29] [3]}

Where Envoy Gateway manages weighted routing, proportions are calculated from each backend weight relative to the total. Multiple backends otherwise receive equal weighting by default. Retain the actual route and observed traffic split; merely adding a second backend should not be assumed to create the intended canary. ^{[30] [23]}

For a Kubernetes deployment, an illustrative rollback command is:

kubectl rollout undo deployment/triton-candidate --to-revision=2

Hypothetical Example: revision 2 represents a verified revision, not a universal recovery target. Kubernetes supports selecting a rollback revision, but only the Pod-template portion is restored. Separately managed model repositories, configuration, and routing must therefore be included in the recovery procedure. ^[8]

Track proxy and client outcomes during restoration. Envoy's overload action can return HTTP 503 ^[14], so a drop in successful throughput must be interpreted alongside rejection counts. gRPC cancellation also leaves the server application responsible for stopping work it started; a client deadline alone should not be treated as proof that all backend activity has ceased. ^[22]

Data Analysis and Evidence

Calculate a working queue allocation

Use an explicit planning equation:

Working queue allocation = end-to-end SLO budget − measured non-queue allocation − safety reserve.

The minus signs above express ordinary subtraction, not a percentile identity. The measurement boundary should follow the user-visible operation, consistent with Google's recommendation to move latency measurement closer to the user. The non-queue allocation should be informed by traces and measured path components, with a separately documented reserve. (Source: sre.google)

Hypothetical Example: an internally chosen 50 ms end-to-end budget, 35 ms non-queue allocation, and 5 ms reserve leave 10 ms of working queue allocation: 50 - 35 - 5 = 10. These are invented planning inputs, not Triton recommendations or observed results. A positive remainder permits an experiment; a nonpositive remainder calls for investigation before intentional waiting is added.

Do not subtract independently measured percentiles and treat the result as a guaranteed queue percentile. Nor should replica percentiles be averaged: Prometheus explicitly warns that averaging quantiles produces statistically meaningless values. Validate the resulting end-to-end distribution directly, with the chosen estimator and aggregation method recorded. ^[28]

Preserve the experiment matrix

Hypothetical Example: Table 2 is a reproducible trial template. B is the model's verified supported maximum batch setting, D is the local working queue allocation expressed in the scheduler's units, and N is the baseline instance configuration. Fractional-delay expressions are illustrative search points, not optimal settings. Every row carries the same manifest described above, including model/version, shapes, warmup, measurement window, and repetition identifier.

The matrix keeps changed controls visible. Convert units explicitly before producing a configuration, and treat D as an experimental boundary rather than a latency guarantee. Replay the same representative arrivals after each promising search result. MLCommons' documented scenario methodology likewise makes request pattern part of the result's definition. ^[5]

Separate averages, distributions, and useful work

For aggregate batch formation, compute average executed batch size = inference-count delta / execution-count delta, using matching intervals and labels. NVIDIA defines the underlying relationship between inference and execution counts. Handle resets and empty intervals explicitly. This reports an aggregate, not the distribution of batch sizes. ^[31]

For queue diagnosis, nv_inference_queue_duration_us records cumulative scheduler waiting time. Collect nv_inference_request_duration_us, nv_inference_compute_infer_duration_us, and nv_inference_pending_request_count alongside nv_gpu_utilization, nv_gpu_memory_used_bytes, and nv_gpu_power_usage. A matched interval calculation can estimate average queue time, but cumulative sums alone cannot reconstruct p99. Use client observations or a suitable histogram for the tail, and keep cache and request-accounting semantics consistent across the comparison. ^[31]

For cross-replica aggregation, Prometheus histograms support aggregation before quantile calculation; precomputed quantiles cannot simply be averaged. Histogram quantiles are estimates, so bucket resolution and sample size belong in the acceptance record, especially when a result lies close to the SLO boundary. ^[28]

Implications and Future Directions

The tuning result should expire when its assumptions change. Refresh the experiment after model or backend changes, a different shape distribution, altered routing, different client placement, or a material resource-allocation change. The point is to preserve evidence for the deployed workload, rather than keep a configuration indefinitely because it once won a benchmark.

Keep a release upgrade on a separate change record. The official 26.09 release describes the corresponding SDK tool distribution. A team's planned 26.08 rollout is therefore a local release choice. Pin the chosen artifacts, verify their behavior, and rerun the batching matrix after that upgrade lands. ^{[7] [27]}

There is likewise no basis here for turning a preferred batch size or canary fraction into a universal recommendation. Scenario-specific latency constraints are part of MLCommons' documented evaluation approach. The same principle is useful operationally: the service's contract determines acceptable performance, and the experiment establishes whether the configuration meets it. ^[32]

Three refresh checks deserve particular attention:

• Distribution changes: repeat relevant shape cohorts; TensorFlow's shape matching illustrates why inputs belong in the experiment identity. ^[19]
• Measurement changes: retain warmup and replicated measurements so changes in noise are visible. ^[9]
• Generator changes: recheck CPU and network headroom before comparing a new harness or traffic scale. ^[23]

For longer-term observability, keep client and server duration boundaries explicit. OpenTelemetry's split RPC metrics provide a useful naming model, while the deployed instrumentation and its semantic-convention version should remain part of the manifest. Preserve enough raw evidence to recompute the acceptance decision after a dashboard or aggregation method changes. ^{[3] [28]}

For a custom client harness, use an elapsed-time clock. Python documents that its monotonic clock is unaffected by system clock changes, and that its performance counter includes elapsed sleep time. That is appropriate for observing waits across a request, unlike measuring only the harness's CPU consumption. ^[33]

Do not describe GPU power telemetry as full-system energy. MLCommons' benchmark power boundary measures AC power at the wall. If the team evaluates energy alongside latency and throughput, identify whether the observation covers the GPU, host, or complete system. ^[5]

Conclusion

Triton dynamic batching should be enabled through an explicit, reversible performance decision. Begin with the configuration that actually loaded, retain a valid no-batching control, and test immediate batching before adding formation delay. The report's experiment matrix then changes delay, preferred sizes, and instances in a sequence that makes attribution possible.

The latency contract must remain attached to the user-visible operation. A good server execution average cannot substitute for an end-to-end distribution, and a closed load test cannot establish behavior under independently arriving traffic. Preserve the offered workload and achieved completions together; the distinction between open and closed models explains why both are necessary. ^{[4] [5]}

Acceptance should require the complete scorecard to pass: latency, useful throughput, errors, queue behavior, resources, output correctness, and recovery. Keep quantile aggregation valid, because averaging replica quantiles does not produce a meaningful fleet percentile. Treat results near the boundary with the precision of the estimator in view. ^[28]

Finally, retain a ready baseline and restore the complete artifact set when a candidate fails. The rollback boundary must be explicit: Kubernetes restores a Deployment's Pod template, while separately managed serving assets require separate recovery. A tuning result is operationally useful when another engineer can reproduce its workload, understand its acceptance decision, and restore its predecessor. ^[8]