Inference Queue

FIFO Implementation A/B

The same image and concurrent four-client workload were run with the original semaphore path (VOXCPM_NANO_QUEUE_SIZE=0), then the bounded FIFO (8), then the semaphore control again. Each run used five rounds and 20 stable-seed calls.

Implementation Concurrent wall Throughput vs control mean WAV matches Peak active Peak queue
Semaphore control A 41.32s - 20/20 n/a 0
Bounded FIFO 41.96s +0.14% 20/20 1 3
Semaphore control B 42.73s - 20/20 n/a 0

The semaphore-control mean was 42.02s. A 0.14% FIFO throughput difference is measurement noise: the scheduler is performance-neutral. Its benefits are bounded admission, FIFO fairness, queue telemetry, request progress, and deterministic HTTP 429 overload handling rather than faster inference.

Backlog Behavior

Sequential submission leaves no request waiting; concurrent submission releases all clients together. Both modes use the same single GPU worker, payloads, stable seeds, and warmed model. This measures the value of keeping prepared work available, which applies to both the semaphore and FIFO implementations.

Run Candidate Clients Rounds Sequential wall Queued wall Throughput change WAV matches Peak queue
20261022-080908 Rejected candidate: Bounded FIFO scheduler validation at concurrency one 4 5 70.87s 50.54s +40.3% 20/20 3
20261022-083122 Rejected candidate: Optimized bounded FIFO scheduler at concurrency one 4 5 65.15s 47.35s +37.6% 20/20 3
20261022-085508 Accepted candidate: Pure-ASGI bounded FIFO scheduler at concurrency one 4 5 59.12s 42.55s +38.9% 20/20 3

Within the accepted FIFO service, concurrent submission changed aggregate request throughput by +38.9% relative to one-at-a-time HTTP submission. All 20 paired fixed-seed WAV files were byte-identical. This gain must not be attributed to the FIFO itself; the queue-disabled controls showed the same behavior.

Mode Calls Median batch Median latency p95 latency Requests/s Audio speed Peak active Peak queued
sequential 20 11.640s 2.932s 3.543s 0.338 3.48x 1 0
queued 20 8.319s 6.476s 8.528s 0.470 4.83x 1 3

Higher queued-request latency is expected because it includes time waiting for the single GPU worker. The relevant gain is aggregate batch wall time and throughput, while output hashes guard deterministic quality.

The accepted candidate also passed the complete release baseline as run 20261022-085849: all 312 WAV hashes and ASR results matched accepted run 20261022-054748, total TTS time changed by +0.21%, and no errors occurred.

See DETAILS.md for controls and interpretation.