Picovoice Cheetah Streaming Speech-to-Text Benchmark

Prerequisites

• Ubuntu 20.04 (x86_64)
• Git
• Python
• PIP
• Picovoice Console Account

Usage

Setup

1. Clone the repository:

git clone https://github.com/Picovoice/speech-to-text-benchmark.git

2. Install the dependencies:

pip3 install -r requirements.txt

3. Create a free Picovoice Console account and copy your AccessKey.

Word Accuracy Benchmark

1. Run the benchmark:

python3 benchmark.py \
--engine PICOVOICE_CHEETAH \
--dataset ${DATASET} \
--dataset-folder ${DATASET_FOLDER} \
--language ${LANGUAGE} \
--picovoice-access-key ${PICOVOICE_ACCESS_KEY}
--picovoice-model-path ${PICOVOICE_MODEL_PATH}

• --picovoice-model-path ${PICOVOICE_MODEL_PATH} is required for non-English languages and Cheetah Fast models. Model files can be downloaded from the Cheetah GitHub Repo.
• To measure the Punctuation Error Rate (PER) add --punctuation to the command.

Word Emission Latency Benchmark

1. Follow the instructions here to generate word timing information.

2. Run the benchmark:

python3 benchmark_latency.py \
--engine PICOVOICE_CHEETAH \
--dataset-folder ${DATASET_FOLDER} \
--language ${LANGUAGE} \
--picovoice-access-key ${PICOVOICE_ACCESS_KEY}
--picovoice-model-path ${PICOVOICE_MODEL_PATH}