Voxtral-Mini-4B-Realtime-2602-GGUF/README.md
| 1 | --- |
| 2 | library_name: audio.cpp |
| 3 | pipeline_tag: automatic-speech-recognition |
| 4 | tags: |
| 5 | - audio.cpp |
| 6 | - gguf |
| 7 | - voxtral |
| 8 | - asr |
| 9 | - streaming-asr |
| 10 | --- |
| 11 | |
| 12 | # Voxtral Mini 4B Realtime GGUF for audio.cpp |
| 13 | |
| 14 | This repository contains quantized standalone GGUF checkpoints for running Voxtral Mini 4B Realtime ASR with [audio.cpp](https://github.com/0xShug0/audio.cpp). The GGUF files embed the audio.cpp model spec and required sidecars, so the model can be used directly from the checkpoint path without a separate local model-spec directory. |
| 15 | |
| 16 | ## What audio.cpp does |
| 17 | |
| 18 | audio.cpp is a young C++/GGML audio inference framework focusing on CUDA performance. It runs speech and audio models locally with CLI and server interfaces, including ASR, TTS, voice conversion, source separation, diarization, VAD, and audio generation models. The project currently tracks 35+ model families and is growing quickly. For Voxtral Realtime, audio.cpp provides offline and streaming speech recognition from a single GGUF checkpoint. |
| 19 | |
| 20 | ## Files |
| 21 | |
| 22 | | File | Quantization | Size | Recommended use | |
| 23 | | --- | --- | ---: | --- | |
| 24 | | `voxtral-mini-4b-realtime-2602-q8_0.gguf` | Q8_0 | 4.8 GiB | Default balanced checkpoint for high-quality ASR with lower memory than BF16. | |
| 25 | | `voxtral-mini-4b-realtime-2602-q4_k.gguf` | Q4_K | 2.9 GiB | Lower-memory and faster checkpoint for CUDA testing and deployment. Validate output quality for your domain. | |
| 26 | |
| 27 | ## Performance |
| 28 | |
| 29 | Measurements below are from audio.cpp CUDA validation runs. Results vary by GPU, driver, backend, audio length, and decode settings. |
| 30 | |
| 31 | ### Q8_0 vs BF16 reference |
| 32 | |
| 33 | BF16 was used only as a local reference baseline for validation. This repository publishes quantized GGUF checkpoints only. |
| 34 | |
| 35 | | Mode | BF16 reference | Q8_0 | Q8_0 improvement | |
| 36 | | --- | ---: | ---: | ---: | |
| 37 | | Offline ASR speed | 11.1x-12.5x realtime | 14.7x-16.7x realtime | 1.31x-1.38x faster | |
| 38 | | Offline ASR peak VRAM | 10,909 MiB | 7,754 MiB | 3,155 MiB lower | |
| 39 | | Streaming server TTFT | 207.308 ms | 179.896 ms | 27.412 ms lower | |
| 40 | | Streaming client TTFT | 550.526 ms | 530.558 ms | 19.968 ms lower | |
| 41 | | Streaming speed | 4.7x realtime | 5.4x realtime | 1.15x faster | |
| 42 | | Streaming peak VRAM | 12,616 MiB | 8,972 MiB | 3,644 MiB lower | |
| 43 | |
| 44 | ### Q4_K quick check |
| 45 | |
| 46 | | Route | Q8_0 RTF | Q4_K RTF | Q4_K vs Q8_0 | |
| 47 | | --- | ---: | ---: | ---: | |
| 48 | | Offline short | 0.0862 | 0.0629 | 1.37x faster | |
| 49 | | Offline medium | 0.0643 | 0.0476 | 1.35x faster | |
| 50 | | Offline longer | 0.0576 | 0.0439 | 1.31x faster | |
| 51 | | Offline sampled | 0.0630 | 0.0500 | 1.26x faster | |
| 52 | | Streaming path | 0.1036 | 0.0904 | 1.15x faster | |
| 53 | |
| 54 | In the quick validation set, Q4_K transcripts matched Q8_0 except for one capitalization-only difference. |
| 55 | |
| 56 | ## Use with audio.cpp |
| 57 | |
| 58 | Build audio.cpp with the helper script for your platform, then use the generated `audiocpp_cli` binary. The project provides build paths for Linux, Windows, and macOS; see the [audio.cpp README](https://github.com/0xShug0/audio.cpp#build) for the current build matrix and detailed requirements. |
| 59 | |
| 60 | ```bash |
| 61 | git clone https://github.com/0xShug0/audio.cpp |
| 62 | cd audio.cpp |
| 63 | |
| 64 | # Linux: CUDA, Vulkan, or CPU |
| 65 | scripts/build_linux.sh --backend cuda --target audiocpp_cli |
| 66 | |
| 67 | # Windows: CUDA or CPU presets |
| 68 | powershell.exe -NoProfile -ExecutionPolicy Bypass -File .\scripts\build_windows.ps1 -Preset windows-cuda-release -Target audiocpp_cli |
| 69 | |
| 70 | # macOS: Metal |
| 71 | scripts/build_metal.sh --target audiocpp_cli |
| 72 | ``` |
| 73 | |
| 74 | The examples below assume `audiocpp_cli` is on your `PATH`. You can also replace it with the built binary path for your platform, such as `build/linux-cuda-release/bin/audiocpp_cli`. |
| 75 | |
| 76 | Run offline ASR with Q8_0: |
| 77 | |
| 78 | ```bash |
| 79 | MODEL=/path/to/Voxtral-Mini-4B-Realtime-2602-GGUF/voxtral-mini-4b-realtime-2602-q8_0.gguf |
| 80 | |
| 81 | audiocpp_cli \ |
| 82 | --task asr \ |
| 83 | --family voxtral_realtime \ |
| 84 | --model "$MODEL" \ |
| 85 | --backend cuda \ |
| 86 | --threads 8 \ |
| 87 | --audio input.wav \ |
| 88 | --text-out transcript.txt |
| 89 | ``` |
| 90 | |
| 91 | Run the Q4_K checkpoint by changing the model path: |
| 92 | |
| 93 | ```bash |
| 94 | MODEL=/path/to/Voxtral-Mini-4B-Realtime-2602-GGUF/voxtral-mini-4b-realtime-2602-q4_k.gguf |
| 95 | ``` |
| 96 | |
| 97 | ## Streaming ASR |
| 98 | |
| 99 | Streaming from an audio file: |
| 100 | |
| 101 | ```bash |
| 102 | audiocpp_cli \ |
| 103 | --task asr \ |
| 104 | --family voxtral_realtime \ |
| 105 | --model "$MODEL" \ |
| 106 | --backend cuda \ |
| 107 | --threads 8 \ |
| 108 | --mode streaming \ |
| 109 | --audio input.wav \ |
| 110 | --text-out transcript.txt |
| 111 | ``` |
| 112 | |
| 113 | Streaming raw 16 kHz mono PCM from `ffmpeg`: |
| 114 | |
| 115 | ```bash |
| 116 | ffmpeg -i input.mp3 -ar 16000 -ac 1 -f s16le - \ |
| 117 | | audiocpp_cli \ |
| 118 | --task asr \ |
| 119 | --family voxtral_realtime \ |
| 120 | --model "$MODEL" \ |
| 121 | --backend cuda \ |
| 122 | --threads 8 \ |
| 123 | --mode streaming \ |
| 124 | --audio - |
| 125 | ``` |
| 126 | |
| 127 | For better streaming throughput, batch a few decode steps: |
| 128 | |
| 129 | ```bash |
| 130 | audiocpp_cli \ |
| 131 | --task asr \ |
| 132 | --family voxtral_realtime \ |
| 133 | --model "$MODEL" \ |
| 134 | --backend cuda \ |
| 135 | --threads 8 \ |
| 136 | --mode streaming \ |
| 137 | --audio input.wav \ |
| 138 | --session-option voxtral_realtime.stream_batch_tokens=4 |
| 139 | ``` |
| 140 | |
| 141 | `stream_batch_tokens=4` improves throughput by amortizing encoder work, with up to roughly `4 * 80 ms` of additional buffering delay. |
| 142 | |
| 143 | ## Prompting and decoding |
| 144 | |
| 145 | audio.cpp exposes Voxtral Realtime as an ASR model. Pass audio to the CLI and audio.cpp builds the model's transcription prompt internally; no free-form text prompt is required for normal transcription. |
| 146 | |
| 147 | Decode options can be controlled from the request: |
| 148 | |
| 149 | ```bash |
| 150 | audiocpp_cli \ |
| 151 | --task asr \ |
| 152 | --family voxtral_realtime \ |
| 153 | --model "$MODEL" \ |
| 154 | --backend cuda \ |
| 155 | --threads 8 \ |
| 156 | --audio input.wav \ |
| 157 | --text-out transcript.txt \ |
| 158 | --request-option max_new_tokens=256 \ |
| 159 | --do-sample false \ |
| 160 | --temperature 1.0 \ |
| 161 | --top-p 1.0 \ |
| 162 | --top-k 50 \ |
| 163 | --seed 1234 |
| 164 | ``` |
| 165 | |
| 166 | ## Notes |
| 167 | |
| 168 | - Task: `asr` |
| 169 | - Family: `voxtral_realtime` |
| 170 | - Supported modes: offline and streaming |
| 171 | - Timestamp output is not currently exposed by audio.cpp for this model |
| 172 | - The GGUF package is intended to be standalone: model weights, sidecars, and audio.cpp model spec are embedded in the checkpoint |
| 173 | |