Generation benchmarks
See the stage timing investigation for the short/long-song comparison and controlled duration/concurrency probes.
The resident-model suite measures warm speed, sampled
VRAM and ASR lyric fidelity separately, using 30-second songs in four languages.
Run task benchmark-plan, task benchmark-smoke or task benchmark-baseline.
It includes cold startup separately, verifies zero model loads in timed calls,
and supports matched before/after regression gates. See recorded results.
The earlier long-song test below remains a separate cold CLI baseline.
For five long English generations using the supplied cinematic synth-rock prompt,
run task benchmark-long. This resident-model profile targets 300 seconds and
makes no ASR requests. See long-music results
and warmup/count options.
task benchmark-variations is a separate two-minute English profile for acoustic
batching: three matched seeds with one/two variations, isolated warm workers,
sampled VRAM, per-track audio integrity and full-rendition ASR. See the
variation results and
workload details. It does not add multi-variation
requests to the short multilingual or five-minute no-ASR suites.
Long-song CLI baseline
task test1
Requires the built local image and models. It runs tests/benchmarks/test1.py against tests/fixtures/test1.txt in a temporary Docker container, independently of the UI server. Do not run competing generation jobs when comparing timings.
Fixed profile
- Full symbolic score generation.
- Upstream 24576-token context and normal eviction, attention and VAE tiling.
- 360-second duration ceiling, one track.
- 8 flow-matching steps.
- LM and acoustic seeds 831001.
- Q8_0 backbone, F32 VAE, MP3 128 kbps.
The 360-second limit is a ceiling; the model may stop earlier. An approximately five-minute song in approximately one minute is a comparison target, not a hard assertion. Fixed seeds do not guarantee byte-identical audio across GPU/toolchain changes.
Use task test1 LM_SEED=123 SEED=456 IMAGE=yue2:dev to select a different profile explicitly. task test1:prepare writes the request without starting inference. Rebuild the image after engine changes; the task does not silently rebuild it.
Artifacts and timing
Each ignored data/results/test1/
- Original prompt and exact request JSON.
- MP3, replay JSON, ABC score, semantic CSV and VAE latents.
- Native generation log and summary.json.
Summary records actual image ID/revision label, model SHA-256 hashes, GPU, seeds, pass/fail, generated duration, wall time and pipeline time. Model hashing precedes timing and warms the filesystem cache. Wall time includes Docker startup, model loading, inference and encoding. Native pipeline timing excludes final MP3 encoding and uses rounded log values.
Success checks native completion, nonempty audio, score/semantic replay data and the requested 8-step setting. Musical quality is not automatically evaluated. Failed generation returns a failing task exit code and retains evidence.
Initial measured baseline
2026-10-30, RTX 5070 Ti 16 GB, CUDA 12.9.1, fixed profile above:
| Measurement | Result |
|---|---|
| Generated audio | 270.8 s (4m 31s) |
| Pipeline | 85.1 s |
| Total wall | 90.5 s |
| End condition | Model end token; duration cap not reached |
This is one local result, not a performance guarantee or cross-hardware claim.
After the repository reorganization, the same fixed profile produced 270.8 s of audio in 79.3 s pipeline / 84.3 s wall time. The ABC score, semantic tokens, latents and MP3 were byte-identical to the initial baseline on this machine. The timing difference is not claimed as an engine speed improvement.
C++20 qualification
On 2026-11-01, the C++20 application linked against the existing GGML 0.28.0 artifact passed the same eight-step profile on the RTX 5070 Ti:
| Measurement | Result |
|---|---|
| Generated audio | 286.2 s (4m 46s) |
| Pipeline | 91.5 s |
| Total wall | 98.8 s |
The ABC score, semantic tokens, latents and MP3 were byte-identical to the C++17 application using the same dependencies. A native rebuild overlapped this qualification run, so these timings are not an isolated speed comparison.