A totally offline use of Whisper ASR and LLaMA-2 GPT Mannequin

These days, no one will probably be stunned by working a deep studying mannequin within the cloud. However the state of affairs may be way more sophisticated within the edge or client system world. There are a number of causes for that. First, the usage of cloud APIs requires units to all the time be on-line. This isn’t an issue for an online service however could be a dealbreaker for the system that must be purposeful with out Web entry. Second, cloud APIs value cash, and clients possible is not going to be comfortable to pay yet one more subscription payment. Final however not least, after a number of years, the challenge could also be completed, API endpoints will probably be shut down, and the costly {hardware} will flip right into a brick. Which is of course not pleasant for patrons, the ecosystem, and the surroundings. That’s why I’m satisfied that the end-user {hardware} ought to be absolutely purposeful offline, with out additional prices or utilizing the web APIs (nicely, it may be non-obligatory however not necessary).

On this article, I’ll present how one can run a LLaMA GPT mannequin and computerized speech recognition (ASR) on a Raspberry Pi. That may permit us to ask Raspberry Pi questions and get solutions. And as promised, all this can work absolutely offline.

Let’s get into it!

The code offered on this article is meant to work on the Raspberry Pi. However a lot of the strategies (besides the “show” half) may also work on a Home windows, OSX, or Linux laptop computer. So, these readers who don’t have a Raspberry Pi can simply check the code with none issues.

{Hardware}

For this challenge, I will probably be utilizing a Raspberry Pi 4. It’s a single-board laptop working Linux; it’s small and requires solely 5V DC energy with out followers and energetic cooling:

A more recent 2023 mannequin, the Raspberry Pi 5, ought to be even higher; in accordance with benchmarks, it’s nearly 2x sooner. However it’s also nearly 50% dearer, and for our check, the mannequin 4 is nice sufficient.