Augmenting LLM Apps with a Voice Modality

Many LLMs, significantly these which might be open-source, have usually been restricted to processing textual content or, sometimes, textual content with pictures (Massive Multimodal Fashions or LMMs). However what if you wish to talk along with your LLM utilizing your voice? Due to the development of highly effective speech-to-text open-source applied sciences in recent times, this turns into achievable.

We are going to go into the mixing of Llama 3 with a speech-to-text mannequin, all inside a user-friendly interface. This fusion allows (close to) real-time communication with an LLM by speech. Our exploration includes choosing Llama 3 8B because the LLM, utilizing the Whisper speech-to-text mannequin, and the capabilities of NiceGUI — a framework that makes use of FastAPI on the backend and Vue3 on the frontend, interconnected with socket.io.

After studying this submit, it is possible for you to to enhance an LLM with a brand new audio modality. It will help you construct a full end-to-end workflow and UI that lets you use your voice to command and immediate an LLM as an alternative of typing. This characteristic can show particularly useful for cell functions, the place typing on a keyboard will not be as user-friendly as on desktops. Moreover, integrating this performance can improve the accessibility of your LLM app, making it extra inclusive for people with disabilities.

Listed below are the instruments and applied sciences that this venture will enable you to get aware of:

• Llama 3 LLM
• Whisper STT
• NiceGUI
• (Some) Fundamental Javascript and Vue3
• The Replicate API

On this venture, we combine numerous parts to allow voice interplay with LLMs (Massive Language Fashions). Firstly, LLMs function the core of our system, processing inputs and producing outputs based mostly on in depth language data. Subsequent, Whisper, our chosen speech-to-text mannequin, converts spoken enter into textual content, enabling easy communication with the LLMs. Our frontend, based mostly on Vue3, incorporates customized parts inside the NiceGUI framework, offering an intuitive consumer interface for interplay. On the backend, customized code mixed with FastAPI varieties the bottom of the app’s performance. Lastly, Replicate.com offers the internet hosting infrastructure for the ML fashions, making certain dependable entry and scalability. Collectively, these parts converge to create a fundamental app for (close to) real-time voice interplay with LLMs.

NiceGUI doesn’t but have an audio recording part so I contributed one to their instance set: https://github.com/zauberzeug/nicegui/tree/most important/examples/audio_recorder that I’ll be reusing right here.

So as to create such part, we simply must outline a .vue file that defines what we wish:

<template>
<div>
<button class="record-button" @mousedown="startRecording" @mouseup="stopRecording">Maintain to talk</button>
</div>
</template>

Right here, mainly, we create a button factor the place when clicked will name a way startRecording and as quickly because the mouse is up will name stopRecording.

For this, we outline these most important strategies:

  strategies: {
async requestMicrophonePermission() {
strive {
this.stream = await navigator.mediaDevices.getUserMedia({ audio: true });
} catch (error) {
console.error('Error accessing microphone:', error);
}
},
async startRecording() {
strive {
if (!this.stream) {
await this.requestMicrophonePermission();
}
this.audioChunks = [];
this.mediaRecorder = new MediaRecorder(this.stream);
this.mediaRecorder.addEventListener('dataavailable', occasion => {
if (occasion.information.measurement > 0) {
this.audioChunks.push(occasion.information);
}
});
this.mediaRecorder.begin();
this.isRecording = true;
} catch (error) {
console.error('Error accessing microphone:', error);
}
},
stopRecording() {
if (this.isRecording) {
this.mediaRecorder.addEventListener('cease', () => {
this.isRecording = false;
this.saveBlob();
// this.playRecordedAudio();
});
this.mediaRecorder.cease();
}
}

This code defines three strategies: requestMicrophonePermission, startRecording, and stopRecording. The requestMicrophonePermission methodology asynchronously makes an attempt to entry the consumer’s microphone utilizing navigator.mediaDevices.getUserMedia, dealing with any errors which will happen. The startRecording methodology, additionally asynchronous, initializes recording by establishing a media recorder with the obtained microphone stream, whereas the stopRecording methodology stops the recording course of and saves the recorded audio.

As soon as the recording is finished, this code can even emit an occasion named 'audio_ready' together with a base64 encoded audio information. Inside the tactic, a brand new FileReader object is created. Upon loading the file, the onload occasion is triggered, extracting the base64 information from the loaded file consequence. Lastly, this base64 information is emitted as a part of the 'audio_ready' occasion utilizing $emit() operate with the important thing 'audioBlobBase64' containing the base64 information.

emitBlob() {
const reader = new FileReader();
reader.onload = () => {
const base64Data = reader.consequence.cut up(',')[1]; // Extracting base64 information from the consequence
this.$emit('audio_ready', { audioBlobBase64: base64Data });
};
}

This occasion can be obtained by the backend together with the base64 information.

The backend can be mainly the glue that ties the consumer’s enter with the ML fashions hosted in Replicate.

We can be using two main fashions for our venture:

1. openai/whisper: This Transformer sequence-to-sequence mannequin is devoted to speech-to-text duties, proficient in changing audio into textual content. Skilled throughout numerous speech processing duties, comparable to multilingual speech recognition, speech translation, spoken language identification, and voice exercise detection.
2. meta/meta-llama-3-8b-instruct: The Llama 3 household, together with this 8 billion-parameter variant, is an LLM household created by Meta. These pretrained and instruction-tuned generative textual content fashions are particularly optimized for dialogue use instances.

For the primary one, we outline a easy operate that takes as enter the base64 audio and calls the replicate api:

def transcribe_audio(base64_audio):
audio_bytes = base64.b64decode(base64_audio)
prediction = replicate.run(
f"{MODEL_STT}:{VERSION}", enter={"audio": io.BytesIO(audio_bytes), **ARGS}
)
textual content = "n".be a part of(phase["text"] for phase in prediction.get("segments", []))
return textual content

Which can be utilized simply as:

with open("audio.ogx", "rb") as f:
content material = f.learn()
_base64_audio = base64.b64encode(content material).decode("utf-8")
_prediction = transcribe_audio(_base64_audio)
pprint.pprint(_prediction)

Then, for the second part, we outline an identical operate:

def call_llm(immediate):
prediction = replicate.stream(MODEL_LLM, enter={"immediate": immediate, **ARGS})
output_text = ""
for occasion in prediction:
output_text += str(occasion)
return output_text

It will question the LLM and stream responses from it token by token into the output_text

Subsequent, we outline the total workflow within the following async methodology:

async def run_workflow(self, audio_data):
self.immediate = "Transcribing audio..."
self.response_html = ""
self.audio_byte64 = audio_data.args["audioBlobBase64"]
self.immediate = await run.io_bound(
callback=transcribe_audio, base64_audio=self.audio_byte64
)
self.response_html = "Calling LLM..."
self.response = await run.io_bound(callback=call_llm, immediate=self.immediate)
self.response_html = self.response.substitute("n", "</br>")
ui.notify("Outcome Prepared!")

As soon as the audio information is prepared, we first transcribe the audio, then as soon as that is executed, we name the LLM and show its response. The variables self.immediate and self.response_html are sure to different NiceGUI parts that get up to date routinely. If you wish to know extra about how that works, you’ll be able to look right into a earlier tutorial I wrote:

The total workflow consequence seems like this:

Fairly neat!

What takes essentially the most time right here is the audio transcription. The endpoint is at all times heat on replicate after I verify it, however this model is the large-v3 which isn’t the quickest one. Audio information are additionally so much heavier to maneuver round than plain textual content, so this contributes to the small latency.

Notes:

• You will have to set REPLICATE_API_TOKEN earlier than working this code. You will get this by signing up in replicate.com. I used to be in a position to do these experiments utilizing their free tier.
• Typically the transcription is delayed a bit bit and is returned after a brief “Queuing” interval.
• Code is at: https://github.com/CVxTz/LLM-Voice The entry level is most important.py.

In abstract, the mixing of open-source fashions like Whisper and Llama 3 has considerably simplified voice interplay with LLMs, making it extremely accessible and user-friendly. This mixture is especially handy for customers preferring to not kind, providing a easy expertise. Nevertheless, that is solely the primary a part of the venture; there can be extra enhancements to come back. The subsequent steps embody enabling two-way voice communication, offering the choice to make the most of native fashions for enhanced privateness, enhancing the general design for a extra polished interface, implementing multi-turn conversations for extra pure interactions, growing a desktop utility for wider accessibility, and optimizing latency for real-time speech-to-text processing. With these enhancements, the purpose is to enhance the expertise of voice interplay with LLMs, making it simpler to make use of for these, like me, that don’t like typing that a lot.
Let me know which enhancements do you suppose I ought to work on first.