Inference Queue Details
This benchmark measures whether a bounded FIFO improves aggregate API throughput around a single Nano-vLLM GPU worker without changing generated speech.
Controls
- The service must report exactly one inference worker.
- Sequential mode submits one request at a time and never builds a backlog.
- Queued mode releases four clients together, allowing one active request and three waiting requests.
- Both modes use the same English design prompt, voice direction, 10 inference steps, WAV output, and stable 32-bit seed for each paired call.
- Text normalization, loudness normalization, denoising, long-text splitting, and randomized seeds are disabled.
- Two warmup calls run before measurement. Round order alternates between sequential-first and queued-first to reduce thermal and ordering bias.
- Queue telemetry is sampled from
/tts/status. The run fails if more than one inference is active, no waiting request is observed, any HTTP request fails, or any paired WAV differs byte-for-byte.
For an implementation A/B, run the same image twice with
VOXCPM_NANO_QUEUE_SIZE=0 and VOXCPM_NANO_QUEUE_SIZE=8. Pass
--expect-queue disabled for the size-zero semaphore control and leave the
default expectation for the FIFO run. This separates the benefit of callers
submitting concurrently from any benefit attributable to the explicit FIFO.
Interpretation
Queued clients include their wait for the single GPU worker, so their median request latency is expected to rise. The useful comparison is total batch wall time, requests per second, and aggregate generated-audio speed. Any gain comes from overlapping request preparation and completed-audio handling around the serialized GPU stage; it is not concurrent model inference.
The queue-disabled A/B control is essential for attribution. Two semaphore control runs completed the concurrent workload in 41.32 s and 42.73 s (42.02 s mean); the FIFO completed it in 41.96 s, or +0.14% throughput versus the control mean. All 20 cross-mode WAV hashes matched. The explicit FIFO is therefore performance-neutral. Concurrent submission itself produced the large throughput improvement, and that behavior already existed around the original semaphore.
The benchmark is deliberately smaller than the multilingual release baseline.
Use task benchmark-baseline separately to detect broad speed, memory, or ASR
quality regressions against established releases.
Release Regression Check
The accepted queue candidate was also run through the complete 312-call,
24-language release baseline as run 20261022-085849, compared with accepted
Nano run 20261022-054748.
| Check | Result |
|---|---|
| WAV SHA-256 matches | 312/312 |
| Audio-duration matches | 312/312 |
| Normalized ASR transcript matches | 312/312 |
| ASR similarity-score matches | 312/312 |
| Total TTS time | 462.414s vs 461.444s (+0.21%) |
| Median RTF | 0.2717 vs 0.2707 |
| p95 RTF | 0.3276 vs 0.3294 |
| Peak VRAM delta | 7911 MiB vs 8330 MiB |
| Errors | 0 |
The reference-conditioning cache recorded one miss followed by 78 hits and held one 31,744-byte entry at the end of the run. An earlier implementation using FastAPI function middleware was rejected after it increased total TTS time by about 13%; pure ASGI request tracking restored baseline performance by forwarding streaming response bodies directly.
Raw run history is stored in the git-ignored
benchmark-artifacts/inference-queue/runs.json.