Transducer decoding is slow with many concurrent streams on GPU

The stateless decoder of icefall transducer models (zipformer and others with --context-size 2) contains one grouped Conv1d. When sherpa-onnx runs that decoder on the GPU for many concurrent streams, the CUDA Conv kernel of onnxruntime serializes every decoder run behind a lock.

icefall can export the decoder with that Conv replaced by an equivalent MatMul (k2-fsa/icefall#2120). The exported model has the same inputs, outputs and metadata, so sherpa-onnx loads it without any code change.

On an A10G GPU serving 200-250 concurrent streams, the Conv-free decoder cut the decoder’s time per decode call from 80-116 ms to 8-10 ms, p95 latency by 22-28% and server CPU by about one core. WER did not change.

Who should use it

  • You run a transducer model with --provider=cuda or --provider=trt. With --provider=trt only the encoder runs on TensorRT. The decoder and joiner run on the CUDA execution provider of onnxruntime (see GetSessionOptions(config, model_type) in sherpa-onnx/csrc/session.cc). The lock affects both providers.

  • Several threads decode at the same time with changing batch sizes. The streaming websocket server (sherpa-onnx-online-websocket-server) does this, and so does any application that calls DecodeStreams from several threads.

  • The decoder has a Conv node (context size > 1). Models exported with --context-size 1 have no decoder conv and are not affected.

CPU-only deployments are not affected: the CPU Conv of onnxruntime takes no lock. The Conv-free export has not been measured on CPU.

Why the Conv serializes

The CUDA Conv::ComputeInternal of onnxruntime (onnxruntime/core/providers/cuda/nn/conv.cc) holds a per-node std::mutex for the whole call. Inside the lock it compares the input shape with the single shape it cached from the previous run. If the shape changed, it rebuilds the cuDNN execution plan, which takes about 2 ms.

A batching server changes the batch size on almost every call, and “the previous run” is the previous run on any thread. So nearly every decoder run pays the rebuild, one thread at a time. A standalone test with only the decoder’s Conv (random weights, one shared session, onnxruntime 1.28.2) shows the effect (runs per second):

Input batch sizes

Conv, 1 thread

Conv, 32 threads

MatMul, 1 thread

MatMul, 32 threads

constant: 100

8,919

14,316

8,256

9,168

alternating: 100, 200

969

998

6,234

6,203

alternating: 40, 100, 200, 400

654

655

5,477

3,739

With a constant shape the Conv is fast. As soon as the shape varies, it is capped at about 650-1,000 runs/s, regardless of the number of threads.

The rewrite

At inference time the decoder conv sees exactly context_size frames and produces one frame. A grouped conv with one output frame is a block-diagonal linear map of the flattened input:

y[n, co, 0] = sum_{i,k} W[co, i, k] * x[n, g*cin_g + i, k],   g = co // cout_g
            = flatten(x)[n, :] @ W_dense[:, co]
W_dense[(g*cin_g + i)*kernel_size + k, co] = W[co, i, k]

So the export replaces

Conv(x [N, 512, 2], weight [512, 4, 2], group=128)  ->  [N, 512, 1]

with

Reshape(x, [N, 1024]) -> MatMul(., W_dense [1024, 512]) -> Unsqueeze(axis=2)  ->  [N, 512, 1]

Every other node is unchanged. For a 512-dim decoder the file grows by about 2 MB (for example from 3.17 MB to 5.25 MB), because the dense weight stores the zeros of the block-diagonal matrix. The extra multiply-adds are negligible on the GPU: the decoder run gets faster, not slower.

Export

Add --decoder-conv-as-matmul 1 to zipformer/export-onnx-streaming.py or zipformer/export-onnx.py in icefall. All other arguments are the same as for your normal export, for example:

cd icefall/egs/librispeech/ASR

./zipformer/export-onnx-streaming.py \
  --tokens $repo/data/lang_bpe_500/tokens.txt \
  --use-averaged-model 0 \
  --epoch 99 \
  --avg 1 \
  --exp-dir $repo/exp \
  --num-encoder-layers "2,2,3,4,3,2" \
  --downsampling-factor "1,2,4,8,4,2" \
  --feedforward-dim "512,768,1024,1536,1024,768" \
  --num-heads "4,4,4,8,4,4" \
  --encoder-dim "192,256,384,512,384,256" \
  --query-head-dim 32 \
  --value-head-dim 12 \
  --pos-head-dim 4 \
  --pos-dim 48 \
  --encoder-unmasked-dim "192,192,256,256,256,192" \
  --cnn-module-kernel "31,31,15,15,15,31" \
  --decoder-dim 512 \
  --joiner-dim 512 \
  --causal True \
  --chunk-size 16 \
  --left-context-frames 128 \
  --decoder-conv-as-matmul 1

The option is off by default. When it is on, the script also exports the original Conv decoder to a temporary file, runs both on the same random token ids with onnxruntime (GPU if available with TF32 off, else CPU) and fails if they differ by more than 1e-4. It logs the result, for example:

Checked the decoder with MatMul against the decoder with Conv (onnxruntime on cuda): max abs diff 9.54e-07

The encoder and joiner are exported exactly as before. Only the decoder file differs.

To confirm that an exported decoder has no Conv:

import onnx

m = onnx.load("decoder-epoch-99-avg-1-chunk-16-left-128.onnx")
print(sorted({n.op_type for n in m.graph.node}))
# Conv-free: no 'Conv' in the list; 'MatMul' and 'Reshape' are present.

Use it in sherpa-onnx

Pass the new file as the decoder. Nothing else changes, in any API or language binding:

./build/bin/sherpa-onnx-online-websocket-server \
  --tokens=./tokens.txt \
  --encoder=./encoder.onnx \
  --decoder=./decoder-conv-as-matmul.onnx \
  --joiner=./joiner.onnx \
  --provider=cuda \
  --decoding-method=modified_beam_search \
  --max-active-paths=10

To roll back, pass the original decoder.onnx again.

Results

Setup: 1x NVIDIA A10G, 8 vCPU; onnxruntime 1.28.2 with the CUDA provider, TensorRT 10.13, CUDA 12.9, cuDNN 9.10; a streaming zipformer2 English model at 8 kHz, FP32; modified_beam_search with 10 active paths. The Conv-free decoder was produced from the same weights by the same graph rewrite (Reshape + MatMul + Unsqueeze) and matched the original decoder to within 1e-6 for batch sizes 1, 7, 100 and 500.

Decoder alone

One shared decoder session and one shared joiner session (no sherpa-onnx). T threads repeat the server’s per-frame sequence (decoder run, then joiner run) with batch sizes drawn from {40, 100, 200, 400}, on the CUDA provider with heuristic cuDNN algorithm search, as in sherpa-onnx.

Threads

Original: runs/s

p50 ms/run

Conv-free: runs/s

p50 ms/run

Speed-up

1

511

2.16

2,491

0.16

4.9x

8

588

13.14

4,383

0.76

7.5x

32

594

53.20

4,269

0.98

7.2x

64

588

108.65

4,119

2.35

7.0x

The original decoder stays at about 590 runs/s from 8 to 64 threads, and its per-run latency grows with the number of waiting threads: one serial section. With 32 threads, 88% of all thread time was spent blocked on CUDA-provider mutexes. With the Conv-free decoder that wait almost disappears.

Streaming server

sherpa-onnx-online-websocket-server with the TensorRT encoder, the decoder and joiner on the CUDA provider, --loop-interval-ms=50, --max-batch-size=50 and 100 work / I/O / intra-op threads. The load was a replicated 500-utterance set at fixed concurrency. Latencies are per-chunk server latencies in ms (middle 80% of each run). Each number is from a single run, so treat differences under about 10% as noise.

Original, 200 streams

Conv-free, 200 streams

Original, 250 streams

Conv-free, 250 streams

p50 latency (ms)

244.1

164.5

316.3

200.7

p95 latency (ms)

328.9

258.0

456.9

330.3

p99 latency (ms)

394.2

313.8

631.6

381.2

Decoder time per call (ms)

80.4

8.2

115.5

10.4

Decoder CPU per call (ms)

50.1

6.4

53.9

7.4

DecodeStreams wall time (ms)

185.8

103.7

247.9

135.2

Server CPU (cores)

3.0

2.0

3.6

2.6

GPU utilization (%)

32.4

32.9

35.2

36.5

WER (%)

8.3

8.3

8.3

8.3

  • p95 latency is 22% lower at 200 streams and 28% lower at 250. p99 is 20% and 40% lower.

  • The decoder’s time per decode call falls by about 10x, and the server needs about one CPU core less.

  • GPU utilization and memory are unchanged. The GPU was not the bottleneck; the lock was.

  • WER is identical.

With the lock gone, the next costs in a decode call are the CPU log-softmax in modified_beam_search and the encoder run. Binding the decoder output to the GPU (I/O binding) gave a further 21-54% decoder throughput in the isolated test, but only once the Conv lock was gone (+1-5% with it).

Alternatives

  • Run the original decoder on TensorRT. In the isolated test the original decoder on the TensorRT provider scaled like the Conv-free one (4,094 vs 4,254 runs/s at 32 threads), but with a higher median latency per run (2.49 ms vs 1.03 ms). It also needs a TensorRT engine and shape profile for the decoder. sherpa-onnx does not run the decoder on TensorRT today, and this was not tested in the server.

  • Pad the decoder batch to a fixed size. With a constant shape the Conv does not rebuild its plan. Even two alternating sizes already cost about 9x, so this would need one fixed size for every call. Not tested.