Private RAG GPU Sizing: Embedding, Reranking, Latency

Executive Summary

Private retrieval-augmented generation (RAG) stacks run three unlike retrieval workloads next to the answer model: offline document embedding and re-indexing, online query embedding, and query-passage reranking. Each has its own rate unit, length distribution and failure mode, so a single "GPUs per user" figure cannot size them. This report, published by GPU Smith as an independent advisor (not a chip maker or cloud provider), treats published ratios as examples of structure and not as transferable capacity.

The sizing arithmetic is simple and should be done per stage. Ingest tokens per second equal total tokenized chunk tokens divided by the seconds left after setup and validation. Reranker pairs per second equal query arrivals per second times the observed mean pairs per query, counted after de-duplication across multi-query expansion. Replica count is the ceiling of required rate over measured, SLO-qualified per-replica capacity, plus an explicit failure scenario, because capacity planning "should be coupled with performance testing" (Source: sre.google). Hugging Face Text Embeddings Inference (TEI) states that it cannot infer the batch-token setting because "the actual memory overhead depends on the model implementation" ^[1], so every number must come from a sweep on the pinned model, runtime and GPU.

The published evidence shows why the stages scale differently. NVIDIA's reference guide gives its Scale 1X baseline as 2.75 GPUs, with a half-GPU embedder and a full-GPU reranker ^[2]. Its own pages state the embedding ratio as 1:40 in some places and 1:50 in others ^[3], which shows why the exact configuration and section must travel with any ratio. On NVIDIA's benchmark pages, a 1B reranker scores 40 passages of 512 tokens per request at 483.7 passages per second on one H100 at FP8, versus 181.4 on an L40S ^[4], so a top-K change or a GPU change moves capacity by multiples. Sentence-Transformers warns that an embedding model sharing a GPU with a generative model "can contend for memory and compute" ^[5], and NVIDIA's guide says to avoid co-locating its LLM service with other GPU workloads ^[6]. A cached vector hit or a small weight file proves nothing about the end-to-end 95th percentile (p95), because fan-out amplifies tails: with a one-second p99 per server and 100 servers, "63% of user requests will take more than one second" ^[7].

The recommended path is: separate the three workloads on paper, pin the model contract, sweep token batch, concurrency and length, compute replicas per stage, then run a co-location test with generation and ingestion on and off. Split pools only when that evidence requires it. Because "Switching models requires re-embedding all vectors in your collection" (Source: qdrant.tech), keep the old index and model contract until the new one passes quality and latency gates.

Introduction and Background

Enterprise teams that host their own RAG stack, often in air-gapped or regulated environments, usually buy GPUs for the answer model first. Embedding and reranking arrive later as "small models" that are expected to fit in the leftover memory. That assumption deserves a test, because the three retrieval workloads are not small in aggregate. A re-index touches every chunk in the corpus. A query embedder sits on the critical path of every request. A reranker multiplies each request into tens or hundreds of scored pairs.

The question is a recurring one. A GitHub issue opened on August 29, 2024 against the TEI repository has a user stating that they are trying to estimate how much GPU equipment is needed ^[8] for a reranker, and reports that their own load test observed about "5.4 TPS" on a partial A100 slice ^[8]. That is an individual user's question and measurement, not a benchmark, and the page shows no maintainer answer. It is useful as evidence of the shape of the problem: capacity is asked for in documents per query and requests per second, and the answer depends on lengths and hardware.

This report is a standard-analysis piece. It does not walk through generic RAG architecture and does not compare answer-model token prices. It concentrates on retrieval service capacity while keeping generation and vector-database latency visible, because the buyer's real constraint is the response time of the whole pipeline. As of November 2026, the report relies on vendor documentation, preprints and peer-reviewed work, each cited for its own configuration. Where a source gives a number, the configuration travels with it. Where no public benchmark exists for a given model, runtime and GPU combination, the report says so and specifies the test instead.

The decision the report supports is concrete. Should query embedding, corpus (re-)embedding and reranking share the generation GPUs, run on CPU capacity, or use separate replicas? And what capacity is needed to meet an ingestion deadline plus online latency and quality requirements? The answer is a worksheet and a gate, not a table of GPU counts.

Key Changes: From One GPU Pool to Three Retrieval Workloads

Inventory the stages and fix the targets first

The first change is to stop describing "the embedding model" as one thing. The same weights can serve three workloads whose demand is independent. Table 1 summarizes the stage ledger that the rest of this report fills in. A team should complete every cell with measured, dated values before choosing hardware.

Table 1 below lays out the stage ledger: the rate unit, the length distribution to record, the capacity measurement that qualifies a replica, and who owns the number.

The ledger is deliberately unit-specific. NVIDIA's own guide measures stage latency as a share of time to first token (TTFT), which NVIDIA's benchmarking documentation describes as including "request queuing time, prefill time, and network latency" ^[9]. For capacity planning, shares are a diagnostic. The planning quantities are rates measured per stage.

The ledger notes that each row's capacity must be SLO-qualified, meaning measured with the p95 target holding. A throughput number taken at saturation with queueing is a different quantity from one taken at the latency target. The Google SRE book makes the same point about overload, noting that "queued requests consume memory and increase latency" (Source: sre.google), which is what happens when a batch embedder is driven past its token budget.

Ingestion and re-index: a worklist, not a corpus size

Corpus re-embedding is a batch job with a deadline. The inputs are the exact chunk count and token-length distribution, the deduplicated changed content, the tokenizer and model revision, the retry policy and the processing window. The formula is:

• Required ingest tokens per second = total actual tokenized chunk tokens / available seconds after measured setup and validation.
• Setup and validation must be measured, not assumed, and include index build and a quality check on a held-out query set.
• Retries inflate the token total; count them in the worklist.

Vendor data shows how strongly length and precision move the answer. On NVIDIA's Embedding NIM performance page, a 1B embedding model on an H100 at FP8, with 300-token passages, batch 64 and concurrency 5, reaches 814.5 inputs per second. At FP16 with the same lengths and concurrency it is 608.3, and at FP8 with 512-token passages it falls to 504.3 ^[10]. Those are NVIDIA's measurements for one model, one GPU and one set of settings. They illustrate that a 1.7 times longer passage and a precision change each move re-index time by tens of percent, which is why the length distribution is the first input to record.

Batching internals matter as much as the model. Hugging Face documents that padding to "the longest sequence in the batch" is the default behavior ^[11], so one long input inflates the padded length of every item beside it. Sentence-Transformers documents that, where flash attention with variable-length support is present, it "skips padding for text-only inputs by concatenating them into a single sequence" ^[5], and that padding everything to a large fixed length "makes every call process the full length and is slower" ^[5]. Length-sorted or token-budgeted batches are therefore part of the ingestion design.

NVIDIA's ingestion guide adds a pipeline observation: in its wiki bulk-ingest test each embedding instance "saturates at approximately CR=8 concurrent threads" ^[12]. Its guide also recommends pre-computing embeddings off the serving path for its multimodal data. These are results for NVIDIA's stack, but the design lesson transfers: ingestion should be scheduled as its own pool or window, not as an unbounded neighbor of the online path.

Online query embedding: arrival traces, not corpus statistics

Query embedding is a latency-sensitive service with short inputs. Record these inputs before any sweep:

• Arrival trace, including bursts and the ratio of peak to mean.
• Language and instruction template, since both change token counts.
• Cache state, because a cached query vector never reaches the GPU.
• Percentile length mix, not only the mean.

The template is part of the model contract. The Qwen3-Embedding model card says queries use a task instruction while "No need to add instruction for retrieval documents" ^[13], and the technical report states that the instruction is concatenated for the query "while leaving the document unchanged before processing" ^[14]. Changing the query instruction changes only query vectors, but it changes retrieval behavior and must be treated as a versioned change. TEI exposes a related control that prepends a default prompt to every input, so those tokens count toward batch tokens.

On NVIDIA's performance page a 20-token query on one H100 at FP8, batch 1, reaches 340.0 inputs per second at concurrency 13 ^[10]. NVIDIA's reference guide also reports that latency overhead for reranking, embedding and the vector database "increases as concurrency increases" ^[2]. The planning consequence: measure the query embedder at the concurrency your gateway will actually present, with and without ingestion running, and take the p95 from that run.

Reranking fan-out: pairs, not queries

A cross-encoder or LLM reranker scores each query-passage pair separately. Sentence-Transformers describes the mechanism: "The query and a possible document is passed simultaneously to transformer network" ^[15], and warns that "Scoring thousands or millions of (query, document)-pairs would be rather slow" ^[15]. The original monoBERT work scores "each passage independently" ^[16] and caps query, passage and separator tokens at 512 ^[16], so cost grows linearly with candidate count and with pair length. A shallow cross-encoder study states the resulting bound directly: scoring K candidates needs K model inferences ^[17], so a latency window caps K.

The planning formula is:

• Reranker pairs per second = query arrivals per second x observed mean pairs per query.
• With multi-query expansion, count actual pairs after de-duplication, not queries x top-K.
• Token demand per pair is query tokens plus passage tokens plus any template tokens.

A production-style study of hybrid retrieval with query rewriting describes this fan-out: each fusion query's retrieved candidates are reranked by the same cross-encoder before rank fusion ^[18], and the study reports added latency from "query rewriting and larger candidate" sets without matching downstream gains ^[18]. The relevance-quality sweep belongs in the same test, because K and the number of rewritten queries are quality parameters as well as capacity parameters.

NVIDIA's reranker pages give configuration-specific reference points. For a 1B reranker at 512 tokens, 40 passages per request and concurrency 5, an H100 at FP8 scores 483.7 passages per second ^[4], at FP16 it scores 347.2, and an L40S at FP8 scores 181.4. A newer 8B reranker on the same kind of request (a query of about 20 tokens and 40 passages of about 512 tokens) is reported at 2933.4 ms average latency on an H100 ^[19]. The 8B figure is a latency for one request shape, while the 1B figures are throughput, so the two are not directly comparable. Model choice, not only top-K, changes rerank cost, and p95 for the chosen model must be measured.

Pin the model, runtime and hardware contract

Runtime support is a separate question from model-card availability. The Qwen3-Reranker-0.6B card loads the model as a causal language model, using AutoModelForCausalLM ^[20], and the technical report frames reranking as point-wise scoring from the likelihood of a yes or no next token ^[14]. TEI's reranker path is documented for sequence-classification models with a single class ^[21], and its supported-models page says it "currently supports CamemBERT, and XLM-RoBERTa Sequence Classification models" ^[22] (the page's example table lists other architectures, so the prose and table differ). A TEI issue opened in June 2025 asks for Qwen3 reranker support, stating that TEI "does not support Qwen3ForSequenceClassification" ^[23]. The status of that issue and the TEI release notes may have changed since it was filed, and the OpenSearch figures below come from the 2.9 documentation, so check the current versions before relying on either. Verify the exact version and image for your hardware instead of assuming support from a model card.

Table 2 below summarizes the contract fields to pin per model, with what the cited sources say.

The table's practical point is that a change in any embedding-side field changes the vectors, and the Qdrant migration guidance states plainly that "Switching models requires re-embedding all vectors in your collection" (Source: qdrant.tech). Reranker changes do not touch the index, which is one reason to version the two components independently. The contract also decides precision: Sentence-Transformers measured plain FP16 at a median 2.92 times the throughput of FP32 on its test models and hardware ^[5], but precision changes can shift quality and must be re-tested on the relevance set.

Vector storage and the vector database stay visible

Vector storage is a lower bound plus overheads (see also the vector index RAM sizing guide). Qdrant's documentation gives the dense size as "dense_size = base * dimensions * bytes_per_dim" (Source: qdrant.tech), where the base multiplies points by "replication_factor" (Source: qdrant.tech) and float32 is "4 bytes (default)". Graph and bookkeeping are separate lines: "hnsw_size = base * m * 2 * 4 bytes * 1.2" (Source: qdrant.tech) and "id_tracker_size = base * 52 bytes" (Source: qdrant.tech), with headroom added "on top of your final RAM and disk totals" (Source: qdrant.tech). Other engines use different formulas, for example OpenSearch's 2.9 documentation (an older release; current documentation may differ) gives "1.1 * (4 * dimension + 8 * M)" bytes per vector ^[26]; pgvector stores a float32 vector in 4 x dimensions + 8 bytes ^[27]; OpenSearch notes that "Having a replica doubles the total number of vectors" ^[26], and Elastic notes that "Each shard replica holds a full copy" ^[28]. Elastic also states that the HNSW graph "must be fully loaded in memory for efficient search" ^[28], and the Faiss wiki notes that "all Faiss indexes are stored in RAM" ^[29]. Quantization shrinks the footprint, with Elastic citing reductions of "4x, 8x, or as much as 32x" ^[28] and Qdrant describing 4x scalar compression with "minimal accuracy loss" (Source: qdrant.tech), but Qdrant also recommends binary quantization only "with rescoring enabled" (Source: qdrant.tech), so quality must be re-measured. Compare the lower bound with measured index memory on your engine, version and settings, and add import workspace separately.

Latency lives here too. Elastic warns that "Once the working set no longer fits in cache, queries start reading from disk and latency climbs sharply" ^[28]. A retrieval stage that benchmarks well on a warm cache can miss its budget after a re-index changes the working set.

Implementation Considerations and Process Changes

Sweep, then calculate replicas

Run sweeps on the pinned model and image, per stage. For embedding, vary the batch-token budget, concurrency and the length mix. TEI describes its maximum batch tokens as the critical control: "Overall this number should be the largest possible until the model is compute bound" ^[1], but because memory overhead is model-dependent the safe value is found by test, with peak memory (including scratch) observed. TEI's token-based dynamic batching ^[30] makes the token budget, not the request count, the unit that fills the GPU. Its concurrency limit is also a design lever, since a low limit refuses client requests, which is backpressure rather than silent queueing. Sentence-Transformers advises the same empirical approach: benchmark a few batch sizes on a small subset of your data ^[31].

For reranking, sweep K, pair length and concurrency, and record quality (for example a held-out relevance set) next to p95. The Pareto table to produce has columns top-K, queries expanded, mean pairs after de-duplication, tokens per pair, pairs per second at SLO, p95 for the stage and relevance score. Beyond a point, extra depth costs capacity without improving relevance; the hybrid-retrieval query-rewriting study above reports larger candidate sets and query rewriting adding latency without matching gains ^[18]; that is one study's data, not a general rule.

Then compute replicas from accepted throughput, not peak:

• Replica count = ceiling(required workload rate / measured SLO-qualified per-replica capacity).
• Failure scenario: add replicas so the stage still meets the SLO with the largest failure domain down. Google's SRE guidance describes N+2 as provisioning so peak traffic is handled "without making the user experience unacceptable" during a simultaneous planned and unplanned outage (Source: sre.google).
• Do not assume linear scaling. NVIDIA's guide reports that stage latency overhead rises with concurrency, and shared bottlenecks (CPU tokenizers, the vector database, the network) appear as replicas are added. TEI's tokenization workers default to the number of CPU cores on the machine ^[1], so host CPU is part of the capacity of a GPU embedder.

A worked deadline worksheet (Hypothetical Example)

The numbers below are invented inputs to show the arithmetic. They are not benchmarks.

(Hypothetical Example) A corpus has 12 million chunks, with a mean of 420 tokens per chunk after tokenization and prompt prefix, giving 5.04 billion tokens. The window is 72 hours, with 6 hours reserved for setup, index build and validation, leaving 66 hours or 237,600 seconds. Required ingest rate is 5.04 billion / 237,600, about 21,200 tokens per second. If a measured, memory-stable, SLO-qualified rate for one replica is 9,000 tokens per second, the ceiling of 21,200 / 9,000 is 3 replicas, but losing one leaves 18,000 tokens per second, below the requirement, so the plan needs 4 replicas or a longer window.

(Hypothetical Example) For reranking, assume a peak of 12 queries per second, 3 expanded queries with 25 candidates each (75 raw pairs) and 55 unique pairs after de-duplication, so the load is 660 pairs per second. At 560 tokens per pair that is 369,600 tokens per second. If measurement gives 400 SLO-qualified pairs per second per replica, the ceiling of 660 / 400 is 2, and a failure scenario raises it to 3.

(Hypothetical Example) For storage, 12 million chunks at 1024 dimensions in float32 is 12,000,000 x 1024 x 4 bytes = 49.15 GB before replicas; the same corpus at 2560 dimensions is 122.9 GB and at 4096 dimensions is 196.6 GB. With Qdrant's published graph formula at m = 16, the graph is about 1.84 GB and the identifier tracker, at 52 bytes per point, about 0.62 GB. Replication, payload, headroom and import workspace are added on top and then compared with measured resident memory.

The co-location gate

Co-location is decided by a measured gate, not by weight size. The gate has four parts.

• Isolation mechanism. Decide how memory and fault isolation will be provided. MPS gives each client fully isolated GPU address spaces, but NVIDIA's MIG guide says MPS "does not offer error isolation between" clients ^[32]. MIG is built "to provide a defined quality of service (QoS) with fault isolation for different clients" ^[33]. Time-slicing "trades the memory and fault-isolation that is provided by MIG" ^[34].
• Memory budget. vLLM's memory fraction is a per-instance limit that applies only to the current vLLM instance ^[35], and its documentation says to set it explicitly when other processes share the GPU, including ones that allocate after it starts ^[35]. Lowering it "takes memory away from the KV cache" ^[35], which shows how a co-located embedder reduces generation capacity. PyTorch's caching allocator keeps unused memory reserved, so it still appears used ^[36], and varying batch sizes can cause fragmentation ^[36].
• Test matrix. Run low, peak and burst demand with generation on, ingestion on and off, and include model reload and cold start. Record p95 for each stage and for the whole pipeline.
• Decision rule. Split pools only when the matrix shows the shared configuration missing a stage or end-to-end target, or when failure isolation requirements cannot be met.

A compatibility note matters here. NVIDIA's reference guide allocates the embedder 0.5 GPU "with Run:ai or MIG" ^[2], while the support matrix of its Embedding NIM 2.3 states that MIG mode clusters are "not supported" ^[37]. These are different documents for different product versions, and the point is only that a partitioning plan must be validated against the runtime's own support statement.

Index migration, readiness and rollback

Treat a new embedding model, instruction, tokenizer, dimension or normalization as a new index contract:

• Dual-run: keep the old index and model serving until the new one passes retrieval quality and online SLO checks.
• Cutover and rollback: switch traffic, then retain the old index for rollback for an agreed period.
• Readiness check: three conditions must hold.
  • Quality on the relevance set at or above the baseline.
  • Latency: p95 per stage and end-to-end within budget with ingestion off.
  • Headroom for incremental updates and a failed replica.

Rerankers can change independently, but their input length and pair count feed capacity. Document each contract version and the ingestion run that produced the index.

Data Analysis and Evidence

The published evidence is useful for structure and weak for transfer. Table 3 below collects configuration-specific reference points. Every row is a vendor-reported result for its own model, runtime, GPU and settings, and none is a general ratio.

The table is read as a pattern library. The same vendor reports 814.5 inputs per second for 300-token passages and 504.3 for 512-token passages on one GPU, a roughly 38% drop for a 1.7 times longer input. Precision moves the 1B reranker from 347.2 to 483.7 passages per second, about 39%, and the GPU class moves it from 483.7 to 181.4. None of this transfers to another model. A newer 8B model at 40 passages per request is reported at 2933.4 ms, a reminder that p95 for a rerank call must be tested for the chosen model.

Discrepancies inside one published guide

NVIDIA's retrieval guide is a good example of why section and configuration must be preserved. The scaling page lists the embedding ratio as "1:40 (Embed:RAG)" ^[2], while the performance-results page gives a 1:50 ratio, and the summary page contains a 1:40 narrative and a table row of 1:50. The accuracy page gives reranker ratios of 1:2 or 1:4, while the scaling page gives 1:8 for chat and 1:4 for summarization ^[38], and the embedding share of TTFT is given as 3% to 9% in one table and 3% to 12% in another. The reasonable reading is that each figure belongs to a specific table and workload definition. A reader should cite the section, not a single ratio.

What the published data does not show

• No universal ratio. NVIDIA's ratios are tied to its Nemotron stack, chat and summarization workloads, and named GPUs; the guide does not give TEI, Qwen or BGE ratios.
• Retrieval depth moves generation load. Raising vector top-K from 4 to 10 increases the context sent to the model, and NVIDIA reports that each extra 1,000 tokens of context reduce output throughput by up to 25% in its low-latency range ^[38]. Rerank depth is therefore a generation-capacity lever too.
• Older latency data is dated. BEIR reported dense retrievers "20-30x faster" than reranking models on 2021 hardware ^[39], and re-ranking models achieve the best zero-shot results "at high computational costs" ^[40]; the ColBERT paper makes the same cost argument for sending "each query-document pair through a massive neural network" ^[41].
• Latency budgets cap depth. In a 2024 shallow cross-encoder study, a full-scale BERT model at a 25 ms per-query limit reached NDCG@10 of only 0.431 on TREC DL 2019 ^[17]. That is one dataset and one model class, but it shows a latency window limits K.
• CPU is hardware-dependent. Sentence-Transformers reports that the best CPU backend differs by machine: OpenVINO INT8 led on a desktop CPU and llama.cpp on a cloud instance ^[5], and ONNX Runtime defaults its intra-op threads to physical cores ^[42]. Intel reports INT8 BGE accuracy loss within 1% relative on its MTEB STS test ^[43]. CPU is a valid candidate for query embedding where measured, not a default.
• Public speed tables are dated. Sentence-Transformers' cross-encoder speed table states that runtime was computed on a V100 GPU with Transformers v4 ^[44], so absolute figures predate current GPUs and runtimes.
• Generation can dominate multi-query pipelines. In the original RAG-Fusion study, the extra latency of multi-query retrieval was attributed mostly to the second large language model call ^[45], which is why retrieval fan-out must be traced next to generation.
• Benchmarks differ in purpose. The MTEB benchmark's speed figure was run on a single A100 80GB for STS15 ^[46], which is quality context and not production throughput.

Case Studies and Real-World Examples

A published single-GPU implementation

An EMNLP 2024 Industry Track paper on a power-plant maintenance RAG framework serves its embedding component with Hugging Face TEI, whose toolkit provides "functionalities such as a web server, load balancer," and a worker pool ^[47]. The reported experiments ran on "an NVIDIA RTX A6000 48GB" ^[47]. The paper is a neutral implementation reference. It shows a small private deployment using TEI for the embedding stage; it does not provide transferable throughput numbers for other corpora, lengths or models.

A user-reported sizing question

The TEI issue discussed earlier describes a reranker sized by documents per request and a target transaction rate, and the user's observed rate on their slice of an A100. It illustrates the missing inputs: chunk length distribution, request concurrency, batch settings and the latency target. This is why the sweep in this report records memory and utilization with every run.

Implications and Future Directions

Three implications follow. First, retrieval capacity planning should have its own owner and artifacts. The generation team's GPU budget will not reveal that a re-index consumed the embedder's memory headroom or that a top-K increase doubled rerank work. Second, end-to-end claims need per-request traces. vLLM exposes time to first token, inter-token latency and prefill as separate metrics, with time per output token computed as "(end-to-end latency - TTFT) / (number of output tokens - 1)" ^[48] and prefill as an interval ^[48]; retrieval stages need the same granularity, with the trace joined per request. Adding stage p95 values does not produce the end-to-end p95, because tails do not add. The Tail at Scale paper shows that for a service with a one-second p99 on each of 100 servers, "63% of user requests will take more than one second", and that in a measured service the p99 for all fan-out requests to finish was "140ms" against a much smaller single-request p99 ^[7]. Rerank fan-out and multi-query retrieval are fan-out in exactly this sense. The paper also notes that "Eliminating all sources of latency variability in large-scale systems is impractical" ^[7], and it reports that hedged requests cut a p99.9 from 1,800 ms to 74 ms in one measured case ^[7]; whether hedging is worth the extra load is a measured question for a GPU-bound reranker.

Third, queueing theory is a sanity check, not a substitute for measurement. Little's relationship between average concurrency, arrival rate and time in the system is credited in the MIT queueing text to J. D. C. Little as first proving it ^[49], and the SRE book applies it, noting that a frontend at 1,000 queries per second with 0.1 seconds of latency has about 100 worker threads busy (Source: sre.google). Capacity should be measured in resources consumed, not just request counts: the SRE book recommends to "measure capacity directly in available resources" (Source: sre.google). For retrieval that means tokens and pairs.

Operationally, burst handling and overload behavior matter more than average utilization. The SRE book calls overload "the most common cause of cascading failures" (Source: sre.google) and warns that "Depending upon this slack CPU as your safety net is dangerous" (Source: sre.google), which applies to borrowing idle generation GPU time for ingestion. Looking forward, runtime coverage will keep changing: TEI lists Qwen3 among supported embedding architectures ^[22] while reranker architectures are documented separately, so teams should re-check support with each version bump. A sound procurement sequence follows from this: set the targets, pin the contract, measure, then buy.

Frequently Asked Questions (FAQs)

How many GPUs do I need for RAG?

There is no universal count. NVIDIA's reference baseline is 2.75 GPUs for a named stack, but that includes an answer-model GPU and specific lengths. Size generation, query embedding, reranking and ingestion separately: compute required tokens per second or pairs per second, divide by measured SLO-qualified per-replica capacity, round up and add a failure scenario.

What are the on-premise RAG GPU requirements?

Requirements start with the runtime. TEI needs drivers compatible with CUDA 12.2 or higher ^[22]. Then memory: model weights, batch-token scratch and the runtime's own overhead, which TEI says depends on the model implementation. Add host CPU for tokenization and index memory for the vector database.

How do I size embedding model GPU throughput?

Measure tokens per second on the actual chunk length distribution with the batch-token and concurrency settings swept, then divide the ingest token total by the available window. Published figures such as 814.5 inputs per second for one model at 300 tokens are references for method, not targets.

What are reranker GPU requirements?

The unit is pairs per second. Each pair is a full forward pass, so requirements scale with top-K, queries expanded and pair length. TEI's reranker path is for sequence-classification models, so verify runtime support for a causal reranker.

How does top-K change reranker latency?

Cost is linear in candidates in the cross-encoder design: K candidates need K inferences. Sentence-Transformers uses 100 candidates as an example depth ^[15]. Measure p95 at each K together with relevance, and choose the knee of the curve.

What is a good end-to-end RAG latency budget?

Budget from the product requirement, then trace each stage per request. NVIDIA's guide proposes stage shares of TTFT for its stack: the LLM at 80% or more, reranking below 10%, embedding below 5% and vector search below 2% ^[2]. Those thresholds are for that stack and workload.

How do I size self-hosted RAG infrastructure for batch embedding?

Use the deadline worksheet: tokens over seconds, after measured setup and validation. Run ingestion in a separate window or pool, and use the co-location gate before sharing it with online serving. Retain the old index until the new one passes.

Conclusion

Private RAG GPU sizing is three calculations that share a model family but not a unit. Ingestion is a deadline problem measured in tokens per second. Query embedding is a latency problem measured at the concurrency the gateway presents. Reranking is a fan-out problem measured in pairs per second, where top-K, query expansion and pair length are both capacity and quality parameters. The vector database and the answer model sit in the same trace, and their latency must stay visible, because a fast embedder or a warm vector cache does not prove that the p95 of the whole pipeline is met.

The method is consistent: pin model, instruction, dimension, normalization, truncation and image; sweep token batch, concurrency and length; compute replicas from measured, SLO-qualified capacity with an explicit failure scenario; run the co-location test with generation and ingestion on and off at low, peak and burst demand; and migrate indexes with the old contract retained for rollback. Published vendor figures remain their own configurations, and no number in this report should replace a measurement on the target hardware.