There are a number of methods to find out the price of operating a LLM (electrical energy use, compute value, and many others.), nonetheless, if you happen to use a third-party LLM (a LLM-as-a-service) they usually cost you based mostly on the tokens you employ. Completely different distributors (OpenAI, Anthropic, Cohere, and many others.) have alternative ways of counting the tokens, however for the sake of simplicity, we’ll contemplate the associated fee to be based mostly on the variety of tokens processed by the LLM.

Crucial a part of this framework is the concept that completely different fashions value completely different quantities. The authors of the paper conveniently assembled the beneath desk highlighting the distinction in value, and the distinction between them is critical. For instance, AI21’s output tokens value an order of magnitude greater than GPT-4’s does on this desk!

As part of value optimization we all the time want to determine a method to optimize the reply high quality whereas minimizing the associated fee. Sometimes, greater value fashions are sometimes greater performing fashions, in a position to give greater high quality solutions than decrease value ones. The final relationship will be seen within the beneath graph, with Frugal GPT’s efficiency overlaid on prime in purple.

Utilizing the huge value distinction between fashions, the researchers’ FrugalGPT system depends on a cascade of LLMs to provide the person a solution. Put merely, the person question begins with the most affordable LLM, and if the reply is nice sufficient, then it’s returned. Nonetheless, if the reply is just not ok, then the question is handed alongside to the following most cost-effective LLM.

The researchers used the next logic: if a inexpensive mannequin solutions a query incorrectly, then it’s doubtless {that a} dearer mannequin will give the reply accurately. Thus, to reduce prices the chain is ordered from least costly to most costly, assuming that high quality goes up as you get dearer.

This setup depends on reliably figuring out when a solution is nice sufficient and when it isn’t. To resolve for this, the authors created a DistilBERT mannequin that may take the query and reply then assign a rating to the reply. Because the DistilBERT mannequin is exponentially smaller than the opposite fashions within the sequence, the associated fee to run it’s nearly negligible in comparison with the others.

One would possibly naturally ask, if high quality is most vital, why not simply question the perfect LLM and work on methods to cut back the price of operating the perfect LLM?

When this paper got here out GPT-4 was the perfect LLM they discovered, but GPT-4 didn’t all the time give a greater reply than the FrugalGPT system! (Eagle-eyed readers will see this as a part of the associated fee vs efficiency graph from earlier than) The authors speculate that simply as essentially the most succesful individual doesn’t all the time give the proper reply, essentially the most complicated mannequin gained’t both. Thus, by having the reply undergo a filtering course of with DistilBERT, you’re eradicating any solutions that aren’t as much as par and growing the percentages of a superb reply.

Consequently, this technique not solely reduces your prices however can even enhance high quality extra so than simply utilizing the perfect LLM!

The outcomes of this paper are fascinating to contemplate. For me, it raises questions on how we will go even additional with value financial savings with out having to spend money on additional mannequin optimization.

One such chance is to cache all mannequin solutions in a vector database after which do a similarity search to find out if the reply within the cache works earlier than beginning the LLM cascade. This could considerably cut back prices by changing a expensive LLM operation with a relatively inexpensive question and similarity operation.

Moreover, it makes you marvel if outdated fashions can nonetheless be price cost-optimizing, as if you happen to can cut back their value per token, they’ll nonetheless create worth on the LLM cascade. Equally, the important thing query right here is at what level do you get diminishing returns by including new LLMs onto the chain.