Fantastic work!! On my RPi 4 with Raspberry Pi OS Lite (bullseye) installed, I had to run

sudo apt install libsdl2-dev

or the compiler would complain about a missing "SDL.h" header file. Now it works like a charm.

Thanks, worked pretty well on the pi for me! (twitter clip ) Yeah, I had to install libsdl2-dev as@eternitybt mentioned as well. (for the mic, I used a ReSpeaker Mic Array v2)

Made a video of the install here:https://youtu.be/caaKhWcfcCY

Hi, what is version Pi4 are you using? Is there a minimum memory requirement? I'm getting a 'Illegal instruction (core dumped)' when I try this on a 1GB Pi4.

can this be done on Raspberry? pi 3b+ model? I wanted to use is in speech recognition

Hello, I followed the build instructions on a Pi4 model B and am receiving this error: "fatal error: immintrin.h: No such file or directory" when attempting the make/build.

I typed the following in terminal: pi@raspberrypi:~ $ uname -a

Linux raspberrypi 6.1.29-v8+#1652 SMP PREEMPT Wed May 24 14:46:55 BST 2023 aarch64
GNU/Linux

Also typed:

pi@raspberrypi:~ $ cat /etc/os-release

PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Used the Raspberry Pi Imager v1.7.4 with Pi OS 64 bit, Debian Bullseye Desktop

I am running on Raspberry Pi 4b and I can record through ffmpeg, but Stream has no output

:

ffmpeg -f pulse -i alsa_input.usb-C-Media_Electronics_Inc._USB_PnP_Sound_Device-00.analog-mono-ar 16000 -ac 1 recording.wav

root@a0f34bc2c254:/whisper-cpp/whisper.cpp# ./stream -m ./models/ggml-tiny.bin -t 6 --step 0 --length 30000 -vth 0.6
init: found 1 capture devices:
init: - Capture device #0: 'USB PnP Sound Device 模拟单声道'
init: attempt to open default capture device ...
init: obtained spec for input device (SDL Id = 2):
init: - sample rate: 16000
init: - format: 33056 (required: 33056)
init: - channels: 1 (required: 1)
init: - samples per frame: 1024
whisper_init_from_file_no_state: loading model from './models/ggml-tiny.bin'
whisper_model_load: loading model
whisper_model_load: n_vocab = 51865
whisper_model_load: n_audio_ctx = 1500
whisper_model_load: n_audio_state = 384
whisper_model_load: n_audio_head = 6
whisper_model_load: n_audio_layer = 4
whisper_model_load: n_text_ctx = 448
whisper_model_load: n_text_state = 384
whisper_model_load: n_text_head = 6
whisper_model_load: n_text_layer = 4
whisper_model_load: n_mels = 80
whisper_model_load: ftype = 1
whisper_model_load: qntvr = 0
whisper_model_load: type = 1
whisper_model_load: mem required = 201.00 MB (+ 3.00 MB per decoder)

whisper_model_load: adding 1608 extra tokens
whisper_model_load: model ctx = 73.62 MB

whisper_model_load: model size = 73.54 MB
whisper_init_state: kv self size = 2.62 MB

whisper_init_state: kv cross size = 8.79 MB

main: processing 0 samples (step = 0.0 sec / len = 30.0 sec / keep = 0.0 sec), 6 threads, lang = en, task = transcribe, timestamps = 1 ...
main: using VAD, will transcribe on speech activity

[Start speaking]

Did you try the default example from above?

./stream -m models/ggml-tiny.en.bin --step 4000 --length 8000 -c 0 -t 4 -ac 512

I wasn't able to get your code to work either.

./stream -m ./models/ggml-tiny.bin -t 6 --step 0 --length 30000 -vth 0.6

Try taking the default example and add -vth 0.6 to the end for the voice activation detector (VAD) like below. Worked well for me.

./stream -m models/ggml-tiny.en.bin --step 4000 --length 8000 -c 0 -t 4 -ac 512 -vth 0.6

Also, the line below works with 6 threads which surprised me, because I thought the Raspberry Pi 4 could go up to 4 threads because it has 4 cores. In the task manager, the CPU usage would sometimes throttle up to near 100% when using either -t 4 or -t 6

./stream -m models/ggml-tiny.en.bin --step 4000 --length 8000 -c 0 -t 6 -ac 512 -vth 0.6

I turned the -step down to 0 like below and it worked once then stopped working.

./stream -m models/ggml-tiny.en.bin --step 0 --length 8000 -c 0 -t 6 -ac 512 -vth 0.6

From what I've seen, upping the --step to 2000 works better and 4000 even better.

The Raspberry Pi 4 is a bit slow, but some development boards equipped with the RK3588 chip have a 6 TOPS NPU. We should consider supporting these chips, as they could potentially enable "real" real-time transcription.@ggerganov

a bit slow

Well, it takes several tens of seconds for a 3 second long wav file...
That is more than just a bit slow. This is, in fact, so slow, that this is neither real-time nor usable...

Whisper is working on the Raspberry Pi 5, up to the small model. Video here:https://youtu.be/W39teHesXus

I managed to getmain working with wav files, but I am running out of memory with a large sound file.stream is looking like a promising alternative but I'm struggling with the implementation. I usedsnd-aloop which created two loopback devices. I'm assuming I can runstream with some parameters in one terminal andaplay with some parameters in another but I haven't been able to get it to work. Can someone advise me on what parameters I should be using to make this work, or if I need a different approach?

Output fromaplay -l:

**** List of PLAYBACK Hardware Devices ****card 0: Headphones [bcm2835 Headphones], device 0: bcm2835 Headphones [bcm2835 Headphones]  Subdevices: 8/8  Subdevice #0: subdevice #0  Subdevice #1: subdevice #1  Subdevice #2: subdevice #2  Subdevice #3: subdevice #3  Subdevice #4: subdevice #4  Subdevice #5: subdevice #5  Subdevice #6: subdevice #6  Subdevice #7: subdevice #7card 1: vc4hdmi [vc4-hdmi], device 0: MAI PCM i2s-hifi-0 [MAI PCM i2s-hifi-0]  Subdevices: 1/1  Subdevice #0: subdevice #0card 2: Loopback [Loopback], device 0: Loopback PCM [Loopback PCM]  Subdevices: 8/8  Subdevice #0: subdevice #0  Subdevice #1: subdevice #1  Subdevice #2: subdevice #2  Subdevice #3: subdevice #3  Subdevice #4: subdevice #4  Subdevice #5: subdevice #5  Subdevice #6: subdevice #6  Subdevice #7: subdevice #7card 2: Loopback [Loopback], device 1: Loopback PCM [Loopback PCM]  Subdevices: 8/8  Subdevice #0: subdevice #0  Subdevice #1: subdevice #1  Subdevice #2: subdevice #2  Subdevice #3: subdevice #3  Subdevice #4: subdevice #4  Subdevice #5: subdevice #5  Subdevice #6: subdevice #6  Subdevice #7: subdevice #7

Output fromarecord -l:

**** List of CAPTURE Hardware Devices ****card 2: Loopback [Loopback], device 0: Loopback PCM [Loopback PCM]  Subdevices: 8/8  Subdevice #0: subdevice #0  Subdevice #1: subdevice #1  Subdevice #2: subdevice #2  Subdevice #3: subdevice #3  Subdevice #4: subdevice #4  Subdevice #5: subdevice #5  Subdevice #6: subdevice #6  Subdevice #7: subdevice #7card 2: Loopback [Loopback], device 1: Loopback PCM [Loopback PCM]  Subdevices: 8/8  Subdevice #0: subdevice #0  Subdevice #1: subdevice #1  Subdevice #2: subdevice #2  Subdevice #3: subdevice #3  Subdevice #4: subdevice #4  Subdevice #5: subdevice #5  Subdevice #6: subdevice #6  Subdevice #7: subdevice #7

For example, if I useaplay -D hw:2,0,0 samples/jfk.wav, then what should I pass tostream to capture from that device?

I tried./whisper.cpp/stream -m whisper.cpp/models/ggml-tiny.en.bin -c 0 and-c 1 and both showed the following:

init: found 2 capture devices:init:    - Capture device #0: 'Loopback, Loopback PCM'init:    - Capture device #1: 'Loopback, Loopback PCM (2)'init: attempt to open capture device 0 : 'Loopback, Loopback PCM' ...                          <-- changes with -c 1init: obtained spec for input device (SDL Id = 2):                                             <-- doesn't change with -c 1init:     - sample rate:       16000init:     - format:            33056 (required: 33056)init:     - channels:          1 (required: 1)init:     - samples per frame: 1024whisper_init_from_file_with_params_no_state: loading model from 'whisper.cpp/models/ggml-tiny.en.bin'[whisper model info...]main: processing 48000 samples (step = 3.0 sec / len = 10.0 sec / keep = 0.2 sec), 4 threads, lang = en, task = transcribe, timestamps = 0 ...main: n_new_line = 2, no_context = 1[Start speaking]

But then no transcription, even using the sample jfk.wav.

Hi! Great job, really!

One question: is there any example/tutorials/guide/whatever on how to implements the same thing using whisper.cpp inside a python script?
Just for experimenting something more sophisticated like using the result as input to an LLM model and so on.

Thank you.

I'm getting an error when compiling usingmake -j stream on a Raspberry Pi 5 running Pi OS Bookworm 12.2.0(uname -a
Linux raspberrypi 6.1.0-rpi8-rpi-v8#1 SMP PREEMPT Debian 1:6.1.73-1+rpt1 (2024-01-25) aarch64 GNU/Linux)

following the build instructions I get an error from make which results in no ./stream folder

ggml-quants.c: In function 'ggml_vec_dot_iq1_s_q8_K':ggml-quants.c:9349:23: error: incompatible types when assigning to type 'int8x16x4_t' from type 'ggml_int8x16x4_t' 9349 |                 q8b = ggml_vld1q_s8_x4(q8); q8 += 64;      |                       ^~~~~~~~~~~~~~~~make: *** [Makefile:326: ggml-quants.o] Error 1

any ideas?

thanks for any help!

I'm getting an error when compiling usingmake -j stream on a Raspberry Pi 5 running Pi OS Bookworm 12.2.0(uname -a Linux raspberrypi 6.1.0-rpi8-rpi-v8#1 SMP PREEMPT Debian 1:6.1.73-1+rpt1 (2024-01-25) aarch64 GNU/Linux)

following the build instructions I get an error from make which results in no ./stream folder

ggml-quants.c: In function 'ggml_vec_dot_iq1_s_q8_K':ggml-quants.c:9349:23: error: incompatible types when assigning to type 'int8x16x4_t' from type 'ggml_int8x16x4_t' 9349 |                 q8b = ggml_vld1q_s8_x4(q8); q8 += 64;      |                       ^~~~~~~~~~~~~~~~make: *** [Makefile:326: ggml-quants.o] Error 1

any ideas?

thanks for any help!

can you show how to fix this build errer ?thanks!

Hi@ggerganov i was trying to run thestream with exactly same parameters except i use non-EnglishCantonese model, it seems whispercpp cannot make any processing for realtime, but it works very good on PC non-realtime (with wav-files). I also set thelanguage argument toauto and / oryue, still not make any processing, do you have the same effect on your RPI? Thank you so much!

Hi, I'm trying this on a Raspberry Pi 3, (tried on buster and bullseye) it compiles perfectly and the main example with the wav files works too. The problem is with the stream, it seems like it loses pieces or can't hear well, I'll start by saying that I tried both with a cheap USB microphone and with a Zoom H1 microphone at 44100hz 16bit, if you record with arecord the audio is perfect. I then tried to directly record what the stream program listens to by adding the-sa option
./stream -m models/ggml-tiny.bin --step 4000 --length 8000 -c 0 -t 4 -l it -sa -ac 512
and in fact the recorded audio is almost all blank except for some parts which are in fact transcribed, but at least 95% of the audio is lost, so there must be a problem with the way the sdl library captures the audio.

Some extra info..this is what happens when I run the program:

Jul 11 14:07:25 myraspy64 pulseaudio[704]: ALSA woke us up to read new data from the device, but there was actually nothing to read. Jul 11 14:07:25 myraspy64 pulseaudio[704]: Most likely this is a bug in the ALSA driver 'snd_usb_audio'. Please report this issue to the ALSA developers. Jul 11 14:07:25 myraspy64 pulseaudio[704]: We were woken up with POLLIN set -- however a subsequent snd_pcm_avail() returned 0 or another value < min_avail. Jul 11 14:07:25 myraspy64 rtkit-daemon[708]: Supervising 5 threads of 3 processes of 1 users. Jul 11 14:07:25 myraspy64 rtkit-daemon[708]: Supervising 5 threads of 3 processes of 1 users. Jul 11 14:07:25 myraspy64 rtkit-daemon[708]: Supervising 5 threads of 3 processes of 1 users. Jul 11 14:10:04 myraspy64 rtkit-daemon[708]: Supervising 5 threads of 3 processes of 1 users. Jul 11 14:10:04 myraspy64 rtkit-daemon[708]: Supervising 5 threads of 3 processes of 1 users. Jul 11 14:10:04 myraspy64 rtkit-daemon[708]: Supervising 5 threads of 3 processes of 1 users. Jul 11 14:11:47 myraspy64 kernel: Transfer to device 6 endpoint 0x1 frame 1460 failed - FIQ timed out. Data may have been lost. Jul 11 14:11:47 myraspy64 kernel: Transfer to device 6 endpoint 0x1 frame 1477 failed - FIQ reported NYET. Data may have been lost. Jul 11 14:11:47 myraspy64 kernel: Transfer to device 6 endpoint 0x1 frame 1488 failed - FIQ reported NYET. Data may have been lost. Jul 11 14:11:47 myraspy64 kernel: Transfer to device 6 endpoint 0x1 frame 1521 failed - FIQ reported NYET. Data may have been lost. Jul 11 14:11:47 myraspy64 kernel: Transfer to device 6 endpoint 0x1 frame 1534 failed - FIQ timed out. Data may have been lost. Jul 11 14:11:47 myraspy64 kernel: Transfer to device 6 endpoint 0x1 frame 1574 failed - FIQ reported NYET. Data may have been lost.

related:
More detailed explanation of --step and --length flags in stream
Problems running the stream example - [Start speaking] frozen

Here's a video on how to run Whisper.cpp with Adafruit I2S microphones on the Raspberry Pi 5 and CM5:https://www.youtube.com/watch?v=V6yoFzcKVJ0

Here are the terminal commands:

TO USE THE I2S MICROPHONES:

Navigate to /boot/firmware/config.txt:

cd /boot/firmware/config.txt

OPEN CONFIG.TXT:

Sudo nano config.txt

Add the following line to the bottom of the config.txt file then hit CNTRL+X to save and exit:

dtoverlay=googlevoicehat-soundcard

REBOOT THE RASPBERRY PI:

sudo reboot now

SHOW THE MIC CARD NUMBER (MOST LIKELY 0, 1, or 2):

arecord -l

TEST MIC VOLUME BY TYPING THIS LINE AND TALKING INTO THE MIC(NOTE, put the CARD NUMBER after plughw:, for example plughw:0):

arecord -D plughw:1 -c1 -r 48000 -f S32_LE -t wav -V mono -v file.wav

TO USE WHISPER.CPP WITH ADAFRUIT I2S microphones AFTER DECEMBER 15th, 2024 in Raspberry Pi 5 Bookworm 64 bit Desktop OS:

INSTALL CMAKE:

sudo apt install -y cmake

INSTAL SDL2 library:

sudo apt install libsdl2-dev

CHECK CMAKE VERSION (OPTIONAL)

cmake --version

GO TO YOUR DESIRED INSTALL DIRECTORY, FOR EXAMPLE Desktop

cd Desktop

CLONE THE WHISPER DIRECTORY:

git clonehttps://github.com/ggerganov/whisper.cpp.git

GO INTO THE NEW whisper.cpp DIRECTORY:

cd whisper

DOWNLOAD YOUR DESIRED MODEL (FOR example tiny.en, base.en, small.en will saturate the Pi 5 CPU's)

sh ./models/download-ggml-model.sh base.en

BUILD WITH CMAKE

cmake -B build -DWHISPER_SDL2=ON
cmake --build build --config Release

RUN WHISPER.CPP

./build/bin/stream

hey i was to run it on raspberry pi B+ but whenever i try to run the make -j stream command an error message keeps on popping up (make: *** No rule to make target 'stream'. Stop)
any help will be great

After some errors with make -j stream I managed to get it working on a raspberry pi 5 8Ram