Streaming ASR

Recognize speech in a WAV file using a streaming (online) Zipformer transducer model. The recognizer processes audio incrementally, producing partial results as more audio arrives.

Source file

nodejs-addon-examples/test_asr_streaming_transducer.js

Code

// Copyright (c)  2024  Xiaomi Corporation
//
// Streaming (online) automatic speech recognition with a Zipformer
// transducer model.
//
// Usage:
//   node streaming_asr.js
//
const sherpa_onnx = require('sherpa-onnx-node');

const config = {
  'featConfig': {
    'sampleRate': 16000,
    'featureDim': 80,
  },
  'modelConfig': {
    'transducer': {
      'encoder':
          './sherpa-onnx-streaming-zipformer-bilingual-zh-en-2023-02-20/encoder-epoch-99-avg-1.onnx',
      'decoder':
          './sherpa-onnx-streaming-zipformer-bilingual-zh-en-2023-02-20/decoder-epoch-99-avg-1.onnx',
      'joiner':
          './sherpa-onnx-streaming-zipformer-bilingual-zh-en-2023-02-20/joiner-epoch-99-avg-1.onnx',
    },
    'tokens':
        './sherpa-onnx-streaming-zipformer-bilingual-zh-en-2023-02-20/tokens.txt',
    'numThreads': 2,
    'provider': 'cpu',
    'debug': 1,
  }
};

const waveFilename =
    './sherpa-onnx-streaming-zipformer-bilingual-zh-en-2023-02-20/test_wavs/0.wav';

// Create the recognizer and a stream.
const recognizer = new sherpa_onnx.OnlineRecognizer(config);
const stream = recognizer.createStream();

// Read the wave file and feed it to the stream.
const wave = sherpa_onnx.readWave(waveFilename);
stream.acceptWaveform({sampleRate: wave.sampleRate, samples: wave.samples});

// Append tail padding so the model can process the last chunk.
const tailPadding = new Float32Array(wave.sampleRate * 0.4);
stream.acceptWaveform({samples: tailPadding, sampleRate: wave.sampleRate});

// Decode in a loop until all frames are consumed.
let start = Date.now();
while (recognizer.isReady(stream)) {
  recognizer.decode(stream);
}
const result = recognizer.getResult(stream);
let stop = Date.now();

const elapsed_seconds = (stop - start) / 1000;
const duration = wave.samples.length / wave.sampleRate;
const real_time_factor = elapsed_seconds / duration;
console.log('Wave duration', duration.toFixed(3), 'seconds');
console.log('Elapsed', elapsed_seconds.toFixed(3), 'seconds');
console.log(
    `RTF = ${elapsed_seconds.toFixed(3)}/${duration.toFixed(3)} =`,
    real_time_factor.toFixed(3));
console.log(waveFilename);
console.log('result\n', result);

How to run

1. Install the package:

   npm install sherpa-onnx-node
   

2. Download the model and test files:

   curl -LS -O https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-streaming-zipformer-bilingual-zh-en-2023-02-20.tar.bz2
   tar xvf sherpa-onnx-streaming-zipformer-bilingual-zh-en-2023-02-20.tar.bz2
   rm sherpa-onnx-streaming-zipformer-bilingual-zh-en-2023-02-20.tar.bz2
   

3. Set the library path and run:

   # macOS
   export DYLD_LIBRARY_PATH=$(npm root)/sherpa-onnx-node/lib:$DYLD_LIBRARY_PATH
   
   # Linux
   export LD_LIBRARY_PATH=$(npm root)/sherpa-onnx-node/lib:$LD_LIBRARY_PATH
   
   node streaming_asr.js
   

Expected output

Wave duration 6.625 seconds
Elapsed 0.234 seconds
RTF = 0.234/6.625 = 0.035
./sherpa-onnx-streaming-zipformer-bilingual-zh-en-2023-02-20/test_wavs/0.wav
result
 { text: ' 吃饭了吗', tokens: [...], timestamps: [...] }

Notes

• OnlineRecognizer is the streaming recognizer. Call createStream() to create a stream, then feed audio with acceptWaveform().

• Append 0.4 seconds of tail padding (zeros) after the main audio so the model can process the last chunk.

• Call isReady() and decode() in a loop until no more frames are available, then call getResult() for the final transcription.

• This model supports both Chinese and English.