You will discover the code on this GitHub repo:
https://github.com/amirarsalan90/personal_llm_assistant

The primary parts of the app embrace:

Llama-cpp-python is a python binding for the good llama.cpp , which implements many Massive Language Fashions in C/C++ . Due to its extensive adoption by open-source group, I made a decision to make use of it on this tutorial.

Notice: I’ve examined this app on a system with Nvidia RTX4090 gpu.

Very first thing first, lets create a brand new conda atmosphere:

conda create --name assistant python=3.10
conda activate assistant

Subsequent we have to set up llama-cpp-python. As talked about in llama-cpp-python descriptions, llama.cpp helps a variety of {hardware} acceleration backends to hurry up inference. As a way to leverage the GPU and run the LLM on GPU, we’ll construct this system with CUBLAS. I had some points with getting to dump the mannequin on GPU and I lastly discovered this put up on tips on how to correctly set up:

export CMAKE_ARGS="-DLLAMA_CUBLAS=on"
export FORCE_CMAKE=1
pip set up --upgrade --force-reinstall llama-cpp-python --no-cache-dir
pip set up llama-cpp-python[server]