How do I configure CUDA for Whisper and FireRedASR?
Use provider="cuda:/path/to/cuda.config" to pass CUDA execution-provider
options to sherpa-onnx. For Whisper and FireRedASR AED, this also keeps the
encoder and decoder attention caches on the selected GPU, just as
provider="cuda" does. No additional cache option is needed.
This behavior requires a CUDA-enabled build containing sherpa-onnx PR #4160. Before that fix, using a CUDA configuration file disabled these models’ CUDA I/O binding, causing extra CPU/GPU cache copies during decoding. FireRedASR CTC is not affected by this cache-binding change.
v2.0.1 and earlier do not contain this fix. For a source build, use commit
aa273e8
or a later commit containing it.
See Install the Python Package for CUDA-enabled Python packages. The CUDA and cuDNN libraries must match the ONNX Runtime included in your build. Forwarding CUDA options from a file requires ONNX Runtime 1.12 or newer.
With older ONNX Runtime versions, the file is still read, but its CUDA
provider options are ignored. For the offline recognizers shown below,
device_id=1 in the file therefore still uses GPU 0. Upgrade ONNX Runtime
to use these options. If you only need to select a GPU on an older runtime,
use provider="cuda" and set CUDA_VISIBLE_DEVICES before starting the
application, as explained below.
Create a configuration file
Save the following as cuda.config:
# Select the first GPU visible to this process.
device_id=0
The file uses one key=value pair per line. Empty lines and lines starting
with # are ignored. Comments must be on separate lines: trailing inline
comments are not supported. For example, device_id=0 # first GPU is not
a valid device setting.
The path must point to a readable file; a relative path is resolved from the application’s working directory.
Warning
A missing or unreadable configuration file terminates the process. This does not trigger the CPU fallback described below. Check the file path and permissions before creating a recognizer.
You can add other CUDA execution-provider options, for example:
device_id=0
arena_extend_strategy=kSameAsRequested
See the ONNX Runtime CUDA execution-provider options for supported keys and values. Available options depend on your ONNX Runtime version; the second setting above is optional.
Use it from Python or the command line
For example, download tiny.en, then create a recognizer with the FP32 model files:
import sherpa_onnx
recognizer = sherpa_onnx.OfflineRecognizer.from_whisper(
encoder="./sherpa-onnx-whisper-tiny.en/tiny.en-encoder.onnx",
decoder="./sherpa-onnx-whisper-tiny.en/tiny.en-decoder.onnx",
tokens="./sherpa-onnx-whisper-tiny.en/tiny.en-tokens.txt",
provider="cuda:./cuda.config",
)
Accept audio and decode as usual. For FireRedASR AED, pass the same
provider="cuda:./cuda.config" argument to
sherpa_onnx.OfflineRecognizer.from_fire_red_asr() with your FireRedASR
model files.
The equivalent Whisper command-line example is:
./sherpa-onnx-offline \
--provider="cuda:./cuda.config" \
--whisper-encoder=./sherpa-onnx-whisper-tiny.en/tiny.en-encoder.onnx \
--whisper-decoder=./sherpa-onnx-whisper-tiny.en/tiny.en-decoder.onnx \
--tokens=./sherpa-onnx-whisper-tiny.en/tiny.en-tokens.txt \
./sherpa-onnx-whisper-tiny.en/test_wavs/0.wav
Select another GPU
Set device_id=1 to select the second GPU visible to the process. The ID
must be a non-negative integer identifying an available CUDA device.
With the offline recognizers above, omitting device_id defaults to 0.
CUDA_VISIBLE_DEVICES changes which devices are visible and renumbers them.
For example, with CUDA_VISIBLE_DEVICES=1, only one GPU is visible to the
process, so use device_id=0 in the file, not device_id=1. Set the
environment variable before starting the application.
If CUDA is unavailable or its provider options are rejected during provider registration, check the log: recognition can fall back to CPU, without CUDA cache binding. A successful transcript alone does not confirm GPU execution.