Operating LLaMA and Gemma LLMs with C++ and Python

These days, “cellular AI” is a fast-growing pattern. Smartphones develop into extra highly effective, and enormous fashions develop into extra environment friendly. Some clients might wish to wait till new options are added by cellphone producers, however can we use the most recent AI fashions on our personal? Certainly, we will, and the outcomes are enjoyable. On this article, I’ll present find out how to run LLaMA and Gemma massive language fashions on an Android cellphone, and we are going to see the way it works. As normal in all my checks, all fashions will run domestically, and no cloud APIs or funds are wanted.

Let’s get into it!

Termux

The primary element of our take a look at is Termux, a full-fledged Linux terminal made as an Android utility. It’s free, and it doesn’t require root entry; all Linux parts are operating completely in a Termux folder. Termux might be downloaded from Google Play, however on the time of penning this textual content, that model was fairly previous, and the “pkg replace” command in Termux didn’t work anymore. A more recent model is out there as an APK on the F-Droid web site; it really works effectively, and I had no issues with it.

When Termux is put in on the cellphone, we will run it and see a typical Linux command-line interface:

In idea, we will enter all instructions instantly on the cellphone, however typing on the tiny keyboard is inconvenient. A significantly better means is to put in SSH; this may be achieved through the use of “pkg set up”:

pkg replace
pkg improve
pkg set up openssh

After that, we will begin the SSH daemon in Termux by operating the sshd command. We additionally must get the consumer title and set the SSH password:

sshd
whoami
#> u0_a461
passwd
#> Enter new password
...

Now, we will connect with a cellphone with any SSH shopper:

ssh -p 8022 u0_a461@192.168.100.101

Right here, 8022 is a default Termux SSH port, “u0_a461” is a consumer title that we get from a “whoami” command, and “192.168.100.101” is the IP…